Background Existing studies of the 1944-45 Dutch famine found little evidence of the association between early life malnutrition and midlife cognition. Methods Among 2446 rural participants born between 1958 and 1963 in the China Health and Retirement Longitudinal Study, we examined effects of exposure to China's 1959-61 Great Leap Forward famine during prenatal and early postnatal life, on four cognitive measures in 2011 (baseline) and changes in cognition between 2011 and 2013 (first follow-up). We obtained difference-in-differences (DID) estimates of the famine effects by exploiting temporal variation in the timing and duration of famine exposure across six birth cohorts born between 1958 and 1963, together with geographical variation in famine severity at the prefecture level. Results After adjusting for gender, marital status and provincial fixed effects, we found that the 1961 cohort who experienced full-term prenatal and partial-term postnatal exposures to famine had lower scores on the Telephone Interview of Cognitive Status (TICS), a test of drawing pentagons, and general cognition at age 50 years compared with the unexposed 1963 cohort. Adjusting for education, the famine effects on drawing pentagons and general cognition were fully attenuated, but the effect on TICS persisted. We also found a robust negative famine effect on the longitudinal change in general cognition during the 2-year follow-up in the 1959 cohort. Conclusions Severe nutritional deprivation during prenatal and postnatal periods has a lasting impact on cognitive performance in Chinese adults in their early 50s.