With the increase of terrorist and criminal activities, closed circuit television (CCTV) is widely used on many occasions. However, abuse of surveillance video may result in the leakage of personal privacy. To protect the privacy in the video of CCTV, an encryption scheme for region of interest (ROI) of H.264 video based on flexible macroblock ordering (FMO) and chaos is proposed in this paper, where human face regions are selected as an example of ROI. First, the human face regions in the video are detected and extracted. Then, they are mapped into slice groups by using FMO technology in H.264. After that, these regions are encrypted using selective video encryption based on chaos. Experimental results and analysis show that the proposed scheme can effectively protect the private information of H.264 video and, therefore, can strike a good balance among the security, encryption efficiency, and coding performance. It has great potential to be used in the privacy protection of the video of CCTV.