A novel technique is proposed for categorizing sports events in videos by tracking the positional and angular displacements of the centroid of the moving object in between successive frames. The various sporting events contained in videos are distinguished either by the speed of motion, for instance walking, jogging and running, or by the trajectory made by the human body while in motion, for instance diving, kicking, bending, lifting weights, jumping and playing golf. The speed of motion is measured by the randomness in the position vector magnitude, and the time-series of angular displacements represents the trajectory. Our method employs minimal training with only a single training video used as the reference for each sporting activity. Experimentation on the KTH, UCF and Weizmann datasets and comparisons with existing methods validate the efficiency of our approach which is simple and easy to implement. The accuracies are further improved by using the Hidden Markov Model to generate posterior state probability sequences from our extracted features which is then used for classification.