We explore the influence of an arbitrary external potential perturbation V on the spectral properties of a weakly disordered conductor. In the framework of a statistical field theory of a nonlinear sigma-model type we find, depending on the range and the profile of the external perturbation, two qualitatively different universal regimes of parametric spectral statistics (i.e., cross correlations between the spectra of Hamiltonians H and H+V). We identify the translational invariance of the correlations in the space of Hamiltonians as the key indicator of universality, and find the connection between the coordinate system in this space which makes the translational invariance manifest, and the physically measurable properties of the system. In particular, in the case of localized perturbations, the latter turn out to be the eigenphases of the scattering matrix for scattering off the perturbing potential V. They also have a purely statistical interpretation in terms of the moments of the level velocity distribution. Finally, on the basis of this analysis, a set of results obtained recently by the authors using random matrix theory methods is shown to be applicable to a much wider class of disordered and chaotic structures.