A real-time gesture tracking and recognition system based on particle filtering and Ada-Boosting techniques is presented in this paper. The particle filter, which is a flexible simulation-based method and suitable for non-linear tracking problems, is adopted to achieve hand tracking robustly. In order to avoid the influence of the other exposed skin parts of a human body and skincolored objects in the background, our system further applies the motion information as a feature of the hand in addition to the skin color information. Compared with the conventional particle filters, our method leads to more efficient sampling and requires fewer particles. It results in lowering computational cost and saving much time for gesture recognition later. The gesture recognition uses the features derived from the wavelet transform, and employs an Ada-Boost algorithm which is excellent in facilitating the speed of convergence during the training. Hence, it is conducive to update new information and expand new gesture archives. The experimental results reveal our system is fast, accurate, and robust in hand tracking. Moreover, it has good performance in gesture recognition under complicated environments.