Although it is well known that salient nontargets can capture attention despite being task irrelevant, several studies have reported short fixation dwell times, suggesting the presence of an attentional mechanism to "rapidly reject" dissimilar distractors. Rapid rejection has been hypothesized to depend on the strong mismatch between distractor features and the target template, but it is unknown whether the presence of strong feature mismatch is sufficient, or if the presence of a target at a competing location is also necessary. Here, we investigated this question by first replicating the finding of rapid rejection for dissimilar distractors in the presence of a concurrent target (Experiment 1); manipulating the onset of the target stimulus relative to the distractor (Experiment 2); and using a saccade-contingent display to delay the target onset until after the first saccade was initiated. The results demonstrate that the speed of distractor rejection depends on the presence of target competition prior to the initiation of the first saccade, and not after the saccade. This suggests that stimulus competition for covert attention sets a "saccade priority map" that unfolds over time, resulting in faster corrective saccades to an anticipated object with higher top-down attentional priority.