Category selective regions in the ventral visual stream are considered to support higher-level representations of objects. The aim of this study was to determine the extent to which category selectivity in face and place regions can be explained by selectivity for low-level features of these complex objects. First, we compared the relative responses to intact and Fourier-scrambled images of faces and places. Next, we compared the magnitude of fMR adaptation to both intact and scrambled faces and places. The results revealed that global differences in the amplitude spectrum of face and place images can explain a small proportion of the category selectivity that is found in regions such as the fusiform face area (FFA) and parahippocampal place area (PPA). However, a whole-brain analysis revealed selectivity to scrambled images in more posterior regions of the ventral stream. Consistent with the pattern evident for intact images, more lateral regions responded selectively to scrambled faces, whereas more medial regions responded more strongly to scrambled places. These findings suggest that selectivity for object categories emerges from the differential processing of low-level features that are typical of different object categories in early visual areas.