Based on an exact analytical approach to describe scattering fidelity experiments [Köber et al., Phys. Rev. E 82, 036207 (2010)], we obtain an expression for the fidelity amplitude decay of quantum chaotic or diffusive systems under arbitrary Hermitian perturbations. This allows us to rederive previous separately obtained results in a simpler and unified manner, as is shown explicitly for the case of a global perturbation. The general expression is also used to derive a so far unpublished exact analytical formula for the case of a moving S-wave scatterer. In the second part of the paper, we extend a relation between fidelity decay and parametric level correlations from the universal case of global perturbations to an arbitrary combination of global and local perturbations. Thereby, the relation becomes a versatile tool for the analysis of unknown perturbations.