Allyl-PEG capped inorganic NPs, including magnetic iron oxide (IONPs), fluorescent CdSe/ZnS quantum dots (QDs), and metallic gold (AuNPs of 5 and 10 nm) both individually and in combination, were covalently attached to pH-responsive poly(2-vinylpyridine-co-divinylbenzene) nanogels via a facile and robust one-step surfactant-free emulsion polymerization procedure. Control of the NPs associated to the nanogels was achieved by the late injection of the NPs to the polymerization solution at a stage when just polymeric radicals were present. Remarkably, by varying the total amount of NPs injected, the swelling behavior could be affected. Furthermore, the magnetic response as well as the optical features of the nanogels containing either IONPs or QDs could be modified. In addition, a radical quenching in case of gold nanoparticles was observed, thus affecting the final nanogel geometry.