We are developing a prototype system for robotically assisted lung biopsy. For directing the robot in biopsy needle placement, we propose a non-invasive algorithm to track the 3D position of the target lesion using 2D CT fluoroscopy image sequences. A small region of the CT fluoroscopy image is registered to a corresponding region in a pre-operative CT volume to infer the position of the target lesion with respect to the imaging plane. The registration is implemented in a coarse to fine fashion. The local deformation between the two regions is modeled by an affine transformation. The sum-of-squared-differences (SSD) between the two regions is minimized using the Levenberg-Marquardt method. Multiresolution and multi-start strategies are used to avoid local minima. As a result, multiple candidate transformations between the two regions are obtained, from which the true transformation is selected by similarity voting. The true transformation of each frame of the CT fluoroscopy image is then incorporated into a Kalman filter to predict the lesion’s position for the next frame. Tests were completed to evaluate the performance of the algorithm using a respiratory motion simulator and a swine animal study.