A salient aspect of objects is their real-world size. Large objects tend to be fixed in the world and can act as navigational barriers and landmarks, whereas small objects tend to be moveable and manipulable. Previous work has identified regions of visual cortex that respond differentially to large versus small objects, but the role of size in organizing representations of object categories has not been fully explored. To address this issue, we scanned subjects while they viewed large and small objects drawn from 20 categories, with retinotopic extent equated across size classes. Univariate analyses replicated previous results showing a greater response to large than small objects in scene-responsive regions and the converse effect in the left occipitotemporal sulcus. Critically, multivariate analyses revealed organization-by-size both within and across functional regions, as evidenced by activation patterns that were more similar for object categories of the same size than for object categories of different size. This effect was observed in both scene- and object-responsive regions and across high-level visual cortex as a whole, but not in early visual cortex. We hypothesize that real-world size is an important dimension for object category organization because of the many ecologically significant differences between large and small objects.