CONTRIBUTION In order to track and estimate the pose of rigid objects with high accuracy in unconstrained environment, we propose a framework that combines 3D particle filter with algebraic pose optimization in a closed loop. The contributions include: 1. A new Particle Filter observation model based on line similarity in 3D space; 2. Coupled tracking and algebraic pose optimization in Particle Filter framework; 3. A dynamic ROI is developed which reduces the line detection and search space for real-time application.