The inverse elementary divisor problem for nonnegative matrices asks for necessary and sufficient conditions for the existence of a nonnegative matrix with prescribed elementary divisors. In this work a Brauer type perturbation result is introduced. This result allows the construction, from a given a list of real or complex numbers Λ = {λ1, . . . , λn}, of certain structured nonnegative matrices with spectrum Λ and with any legitimately prescribed elementary divisors.