In order to improve the real-time of system and realize the harmonious human-machine interaction, we use the method which combines color segment and Adaboost to detect lips. Firstly, It segment the face region with skin model. Then scan the face region by Adaboost algorithm to detect lip. Through confirm the lips window's position in the entire image, to control the intelligent wheelchair's movement. The experiment show that this method could reduce the detection time greatly and improve the detection precision.