Research suggests that perception and imagination engage neuronal representations in the same visual areas. However, the underlying mechanisms that differentiate sensory perception from imagination remain unclear. Here, we examine the directed coupling (effective connectivity) between fronto-parietal and visual areas during perception and imagery. We found an increase in bottom-up coupling during perception relative to baseline and an increase in top-down coupling during both perception and imagery, with a much stronger increase during imagery. Modulation of the coupling from frontal to early visual areas was common to both perception and imagery. Furthermore, we show that the experienced vividness during imagery was selectively associated with increases in top-down connectivity to early visual cortex. These results highlight the importance of top-down processing in internally as well as externally driven visual experience.