A three-dimensional interactive “mouse” system for minimally invasive surgery(MIS) is proposed in this thesis. The system consists of a novel tracking framework, which uses the images of the existing surgical tools in the field of view to determine their corresponding spatial coordinates. This tracking framework combines the advantages of both Kalman filtering tracking and real-time feature-based tracking, offering a relatively robust system for the later 3D augmentation environment to be laid upon. The designed surgeon-computer interface based on this tracking framework allows the surgeon to access the medical information during surgery without introducing extra pieces of hardware. It also allows manual registration of pre-operative images on the live images. Finally, preliminary results on an optically instrumented surgical tool are presented, which can be used to scan inner cavity or image the inner structure of tissue in MIS.