This paper studies the transition of phyllotactic patterns by a group-theoretic approach. Typical phyllotactic patterns are represented here as dotted patterns on a cylinder, where the cylinder is regarded as the stem of a plant and the dots are points where leaves branch from the stem. We can then classify the symmetries of the alternate and opposite phyllotaxis into four types of groups, and clarify sequences of symmetry-breaking among these groups. The sequences turn out to correspond to transition paths of phyllotactic patterns found in the wild. This result shows the usefulness of classification of phyllotactic patterns based on their group symmetries. Moreover, the breaking of reflection symmetry is found to be an important rule for real phyllotactic transitions.