Much attention had been devoted to study the bit error rate (BER) of hierarchical modulation on point-to-point transmissions in recent years. But few mention the error performance of hierarchical modulation on cooperative communication systems. Thus, in this paper, we derive the explicit closed-form expressions of the exact bit error rate computation for cooperative communication systems with hierarchical modulation over additive white Gaussian noise (AWGN) channels and Rayleigh fading channels. To obtain the theoretical formulas of error rate performance, an analyzing model is proposed to clearly express all possibilities of each transmission. For error rate performance, the expression for BER is a function of distance parameters. Based on this relationship, a criterion is proposed to choose the optimal distance parameters for minimizing the BER of the refinement bits while guaranteeing the BER requirement of the base bits. Simulation results validate the correctness of the derived BERs of the base bits and the refinement bits. Also from simulation results, the performance with optimal distance parameters is greatly improved compared to the non-optimal distance parameters.