In this paper, we propose a novel algorithm for player detection and tracking in tennis games. The algorithm utilizes court knowledge as well as player color and edge information to extract deformable player figures. Several new techniques are presented in our algorithm: initially, the court lines are detected and reconstructed. Based on the court model, an adaptive search window is designed for locating the minimum region containing a player figure. After retrieving the region of interest, pixel data are processed by non-dominant color extraction and edge detection filters, respectively. Finally, the non-dominant color map and edge map are refined and combined, and a novel shadow removal method is then applied to isolate the player figure. The algorithm was tested on numerous videos with different courts and light condition. Experiments reveal promising results against various environmental factors.