In a previous longitudinal diffusion tensor imaging (DTI) study, we observed cerebral white matter (WM) alterations (reduced fractional anisotropy (FA)) related to decreased cognitive performance 3–5 months after chemotherapy-treatment (t2) when compared to baseline (t1) (Deprez et al. in Journal of Clinical Oncology: Official Journal of the American Society of Clinical Oncology, 30(3), 274–281. doi:10.1200/JCO.2011.36.8571, 2012). The current study investigates the evolution and the nature of these previously observed microstructural changes. Twenty-five young women with early-stage breast cancer who received chemotherapy treatment (C+), 14 who did not receive chemotherapy (C-) and 15 healthy controls (HC) previously studied, underwent reassessment 3–4 years after treatment (t3). We assessed (1) longitudinal changes of cognitive performance and FA and (2) cross-sectional group differences in myelin-water-imaging and multishell diffusion MRI metrics at t3. MRI metrics were assessed on a voxel-by-voxel basis and in regions-of-interest (ROI) in which previous WM injury was detected. Longitudinal results: Mixed-effects modeling revealed significant group-time interactions for verbal memory and processing speed (p < 0.05) reflecting regained performance in the C+ group at t3. Furthermore, in chemotherapy-treated patients, FA returned to baseline levels at t3 in all ROIs (p < 0.002), whereas no FA changes were seen in controls. Additionally, FA increase from t2 to t3 correlated with time since treatment in two of the four regions (r = 0.40, p < 0.05). Cross-sectional results: Advanced diffusion MRI and myelin-water imaging metrics in the ROIs did not differ between groups. Similarly, no whole-brain voxelwise differences were detected. Initial WM alterations and reduced cognitive performance following chemotherapy-treatment were found to recover in a group of young breast cancer survivors three to four years after treatment.