To recognize litchi image under natural condition correctly, and support three-dimension information to robot for complete automatic fruit picking. A method based on H component histogram threshold and level set is developed for automatic litchi image segmentation. In this paper, sixty-three images with natural scenes were analyzed using the color properties of target objects. Hue component of HSV color space is selected, and the rotation of Hue is used as the feature for image segmentation; And then, fuzzy rule are utilized to find the optimal threshold, and obtain the appropriate initial evolution curve contour automatically. Second, the sparse field level set method is adopted to extract the target region precisely. This algorithm, not only overcomes the influence of random noise, but also maintains the integrity of the segmentation area. The result shows that the proposed algorithm can keep the integrity of object area on litchi image segmentation, and the correct segmentation rate is up to 90.48%.