Tracking of an object's full six degree-of-freedom (6-dof) position and orientation (pose) would allow a robotic system to autonomously perform a variety of complex tasks, such as docking from any preferred angle, surveillance of moving subjects, etc. Computer vision has been commonly advocated as an effective tool for 3D (i.e., 6-dof) tracking Objects of Interest (OIs). However, the vast majority of vision-based 6-dof pose trackers reported in the literature require a model of the OI to be provided a priori. Finding/selecting the OI to track is also essential to autonomous operation. A problem that has often been neglected.

This Thesis proposes a novel, real-time object-tracking system that solves all of the aforementioned problems. The tracking procedure begins with OI selection. Since what constitutes an OI is application dependent, selection is achieved via a customizable framework of Interest Filters (IFs) that highlight regions of interest within an image. The region of greatest interest becomes the selected OI. Next, an approximate visual 3D model of the selected OI is built on-line by a real-time modeller. Unlike previously proposed techniques, this modeller can build the model of the OI even in the presence of background clutter;​ an essential task for tracking one object amongst many. Once a model is built, a real-time 6-dof tracker (i.e., the third sub-component) performs the actual 6-dof object tracking via 3D model projection and optical flow.

Performing simultaneous modelling and tracking presents several challenges requiring novel solutions. For example, a novel data-reduction scheme based on colour-gradient redundancy is proposed herein that facilitates using colour input images whilst still maintaining real-time performance on current computer hardware. Likewise, a per-pixel occlusion-rejection scheme is proposed which enables tracking in the presence of partial occlusions. Various other techniques have also been developed within the framework of this Thesis in order to achieve real-time efficiency, robustness to lighting variations, ability to cope with high OI speeds, etc.

Extensive experiments with both synthetic and real-world motion sequences have demonstrated the ability of the proposed object-tracking system to track a priori unknown objects. The proposed algorithm has also been tested within two target applications:​ autonomous convoying, and dynamic camera reconfiguration.