The quantification of left ventricular (LV) deformation from noninvasive image sequences is an important clinical problem. To date, feature information from either magnetic resonance (MR), computed tomographic (CT) or echocardiographic image data have been assembled with the help of different regularization models to estimate LV deformation. The currently available regularization models have tradeoffs related to accuracy, lattice density, physical plausibility and computation time. This paper introduces a new regularization model based on the boundary element method (BEM) which can overcome these tradeoffs. We then employ this new regularization model with the generalized robust point matching (GRPM) strategy to estimate the dense displacement fields and strains from 3D LV image sequences. The approach is evaluated on in vivo cardiac magnetic resonance image sequences. All results are compared to displacements found using implanted markers, taken to be a gold standard. The approach is also evaluated on the 4D real time echocardiographic image sequences and the results demonstrate that the approach is capable of tracking the LV deformation for echocardiography.