We explore the interaction between local-global information processing in visual perception, leveraging a visual phenomenon known as crowding, whereby the perception of a target stimulus is impaired by the presence of nearby flankers. The majority of established models explain the crowding effect in terms of local interactions. However, recent experimental results indicate that a classical crowding effect, the deterioration in the discrimination of a vernier stimulus embedded in a square, is alleviated by the presence of additional flanker squares (“uncrowding”). Here, we propose that crowding and uncrowding arise from cortical inferences about hierarchically organized groups, and formalize this concept using a hierarchical Bayesian model. We show that the model reproduces both crowding and uncrowding for flanked vernier discrimination. More generally, the model provides a normative explanation of how visual information might simultaneously flow bottom-up, top-down, and laterally, to allow the visual system to interactively process local and global features in the visual scene.