Let G be a group and CayP (G) < Sym(G) be the subgroup of all permutations that induce graph automorphisms on every Cayley graph of G. The group G is graphically abelian if the map ν : g → g−1 belongs to CayP (G); these groups have been classified. Also G is irregular if there exists σ ∈ CayP (G) such that σ = 1G, σ(1) = 1 and σ = ν. We show G is irregular if and only if G = Dic(A, I); every non-abelian graphically abelian group is irregular; and if G is irregular but not graphically abelian, σ ∈ CayP (G) and σ(1) = 1, then σ ∈ Aut(G). No irregular group has a GRR. If an irregular group G is not graphically abelian then there is exactly one irregular map σ and CayP (G) ∼= G 〈σ〉, or otherwise CayP (G) ∼= (G Inn(G)) 〈ν〉.