We analyze sources of noise in kilohertz frequency pump/probe experiments and present a method for reducing experimental noise by identifying and filtering noisy shots. The power spectrum of instrumental noise shows high frequency, small amplitude modulations which cannot be averaged out. A histogram analysis shows that low frequency, large amplitude signals pose a serious obstacle to signal averaging for improved signal to noise. In kilohertz frequency pump/probe experiments, this low frequency noise typically arises from laser scatter due to bubbles, dust, and defects. We quantify the effectiveness in analyzing and rejecting these large amplitude signals which can produce a hindrance to the effectiveness of signal averaging.