This paper presents a robust approach to detect multiple moving targets from aerial infrared (IR) image sequences. The proposed novel method is based on dynamic Gabor filter and dynamic Gaussian detector. First, the motion induced by the airborne platform is modeled by parametric affine transformation and the IR video is stabilized by eliminating the background motion. A set of feature points are extracted and they are categorized into inliers and outliers. The inliers are used to estimate affine transformation parameters, and the outliers are used to localize moving targets. Then, a dynamic Gabor filter is employed to enhance the difference images for more accurate detection and localization of moving targets. The Gabor filter’s orientation is dynamically changed according to the orientation of optical flows. Next, the specular highlights generated by the dynamic Gabor filter are detected. The outliers and specular highlights are fused to indentify the moving targets. If a specular highlight lies in an outlier cluster, it corresponds to a target; otherwise, the dynamic Gaussian detector is employed to determine whether the specular highlight corresponds to a target. The detection speed is approximate 2 frames per second, which meets the real-time requirement of many target tracking systems.