This paper presents a conceptual architecture for pervasive advertising on public displays. It can help researchers and practitioners to inform the design of future display networks. Due to falling hardware prices we see a strong proliferation of (public) places with displays and it is not only large outdoor advertisers anymore operating them. However, public displays currently fail to attract the attention of the user – a challenge that could be overcome by networking displays and deploying sensors that allow novel interaction techniques and engaging user experiences to be created. One major question is how to design an appropriate infrastructure that caters to the conflicting needs of the involved stakeholders. Users want interesting content and their privacy being respected, advertisers want to gather the user’s data, and display owners want to be in control of the content as they fund the infrastructure. We identify the core components and discuss how control can be appropriately distributed among stakeholders by presenting three different forms of the architecture (user-centered, advertiser-centered, trusted).