We present a new image-based approach to global illumination that is designed to explicitly make use of coherence in the radiance field of a scene. The approach builds on recently developed ideas from image-based rendering in order to reuse previously computed geometric and illumination information as far as possible. The algorithm consists of a hierarchy of three methods, each taking advantage of a different coherence pattern in the radiance field. At the top level, we sample primary layered-depth radiance maps of the environment, thereby abstracting from the exact geometry of the scene. In the intermediate level, coherence in ray-space is exploited in order to derive new radiance maps by reprojection from primary maps. At the lowest level, gradient information is used to spatially interpolate smooth irradiance contributions, handling non-smooth contributions separately. Missing or uncertain information in lower levels is handled by resampling the environment, and feeding these samples back to adjacent top-level maps. This adaptively improves the layered-depth images and thus the results of later reprojection operations. The essence of our algorithm is an image-based approach to quickly provide accurate radiance maps anywhere in a scene. In this paper, we use the radiance maps to compute irradiance in diffuse environments. However, the comprised information can be used for a much wider range of applications.