Speed of visual word recognition is an important variable affecting linguistic competence. Although speed of visual word recognition varies widely between individuals, the neural basis of reaction time (RT) differences is poorly understood. Recently, a magnetic resonance technique called diffusion tensor imaging (DTI) has been shown to provide information about white matter (WM) microstructure in vivo. Here, we used DTI to explore whether visual word recognition RT correlates with regional fractional anisotropy (FA) values in the WM of healthy young adults. Participants completed a speeded lexical decision task that involved visual input, linguistic processes, and a motor response output. Results indicated that lexical decision RT was correlated negatively with FA in WM of inferior parietal and frontal language regions rather than in WM of visual or motor regions. Voxels within the inferior parietal and frontal correlation clusters were composed primarily of DTI-based tracts oriented in the anterior-posterior orientation at or near the superior longitudinal fasciculus (SLF) and likely including other smaller association fibers. These results provide new microstructural evidence demonstrating that speed of lexical decision is associated with the degree to which portions of frontal and parietal WM are directionally oriented.