The face recognition ability varies across individuals. However, it remains elusive how brain anatomical structure is related to the face recognition ability in healthy subjects. In this study, we adopted voxel-based morphometry analysis and machine learning approach to investigate the neural basis of individual face recognition ability using anatomical magnetic resonance imaging. We demonstrated that the gray matter volume (GMV) of the right ventral anterior temporal lobe (vATL), an area sensitive to face identity, is significant positively correlated with the subject's face recognition ability which was measured by the Cambridge face memory test (CFMT) score. Furthermore, the predictive model established by the balanced cross-validation combined with linear regression method revealed that the right vATL GMV can predict subjects' face ability. However, the subjects' Cambridge face memory test scores cannot be predicted by the GMV of the face processing network core brain regions including the right occipital face area (OFA) and the right face fusion area (FFA). Our results suggest that the right vATL may play an important role in face recognition and might provide insight into the neural mechanisms underlying face recognition deficits in patients with pathophysiological conditions such as prosopagnosia.