Majority of the videos that have been captured by mobile cameras are suffering from low quality due to either low end manufacturing designs or complicated operating environments and untrained users. Thus videos taken by hand held mobile cameras tend to suffer from different undesired slow motions that cause annoying shaky motion and jitter. It is desirable to stabilize the video sequence by removing the undesired motion between the successive frames. Current methods are applicable to only specific camera motion models; hence having limitation to process gorse motion. In this paper an efficient video stabilization algorithm for hand held camera videos has been proposed. The proposed algorithm uses differential global motion estimation with Taylor series expansion to improve the estimation efficiency. Affine motion model has been assumed to define the inter-frame error between consecutive frames. Motion vectors have been estimated analytically by solving the derivatives of the inter-frame error. After motion estimation Gaussian kernel filtering has been used to smoothen out estimated motion parameters. Inverse rotation smoothening has been applied to remove the rotation effect from the smoothed transformation chain. This has led to reduced accumulation error and minimizes the missing image area significantly. The performance of the proposed algorithm has been tested on real time videos and compared with existing algorithm.