Evidence supports the notion that the fusiform gyrus (FG), as an integral part of the ventral occipitotemporal junction, is involved widely in cognitive processes as perceiving faces, objects, places or words, and this region also might represent the visual form of an abacus in the abacus-based mental calculation process. The current study uses a combined voxel-based morphometry (VBM) and diffusion tensor imaging (DTI) analysis to test whether long-term abacus training could induce structural changes in the left FG and in the white matter (WM) tracts distribution connecting with this region in school children. We found that, abacus-trained children exhibited significant smaller gray matter (GM) volume than controls in the left FG. And the connectivity mapping identified left forceps major as a key pathway connecting left FG with other brain areas in the trained group, but not in the controls. Furthermore, mean fractional anisotropy (FA) values within left forceps major were significantly increased in the trained group. Interestingly, a significant negative correlation was found in the trained group between the GM volume in left FG and the mean FA value in left forceps major, suggesting an inverse effect of the reported GM and WM structural changes. In the control group, a positive correlation between left FG GM volume and tract FA was found as well. This analysis visualized the group level differences in GM volume, FA and fiber tract between the abacus-trained children and the controls, and provided the first evidence that GM volume change in the left FG is intimately linked with the micro-structural properties of the left forceps major tracts. The present results demonstrate the structural changes in the left FG from the intracortical GM to the subcortical WM regions and provide insights into the neural mechanism of structural plasticity induced by abacus training.