Few studies have examined how neighborhood contextual features may influence the food outlet mix. We evaluated the relationship between changes in neighborhood crime and changes in the food environment, namely the relative density of unhealthy (or intermediate) food outlets out of total food outlets, or food swamp score, in Baltimore City from 2000 to 2012, using neighborhood fixed-effects linear regression models. Comparing neighborhoods to themselves over time, each unit increase in crime rate was associated with an increase in the food swamp score (b = 0.13; 95% CI, −0.00017 to 0.25). The association with food swamp score was in the same direction for violent crime and in the inverse direction for arrests related to juvenile crimes (proxy of reduced crime), but did not reach statistical significance when examined separately. Unfavorable conditions, such as crime, may deter a critical consumer base, diminishing the capacity of a community to attract businesses that are perceived to be neighborhood enhancing. Addressing these more distal drivers may be important for policies and programs to improve these food environments.