Coal rock stability evaluation has been proposed to predict the mine catastrophic hazards. Acoustic Emission (AE) as a useful nondestructive condition monitoring technique provides earlier fracture detection in coal rock. Due to various external and internal variables changings, AE events accurately characterized fracturing processes is still a problem. For this reasons, this paper proposes a new Waveform Fractal Dimension algorithm (WFD) for AE analysis to further recognize coal rock damage levels and predict prior to failure. This method is deduced based on BOX dimension, Shuffled Frog Leaping Algorithm (SLFA) for optimal parameters and Support Vector Machine (SVM) to recognition. The experimental investigations were carried out to summarize the exponent relationship between WFD-SFLA dimension and released energy, evaluate the recognition performance of the proposed method under BOX, GP, WFD and WFD-SFLA method. The results show that the proposed method can effectively recognize the samples in stable and critical instable at two different compression modes and lithology. It is a salient way to improve the practical coal rock stability prediction.