This thesis focused on developing a framework for evaluating the 3D reconstruction created through simultaneous localization and mapping (SLAM). Common benchmarking techniques rely on having a fully realized ground truth, either as a trajectory or 3D structural data. The proposed system evaluates relations between known sets of landmarks and compares those to relations between the corresponding features within the reconstruction. This requires far less data for evaluation and is much more applicable outside of controlled environments.

First, CAD models of objects that are known to be within the environment are processed. The models are sampled into high resolution point clouds that are used in further processing. Features are extracted from the point clouds and compared to each other. For the experiments in this thesis, the features of interest were the centroids of each object with the comparison being the Euclidean distance within each other.

The point cloud representation of these objects are then separated from the reconstruction of the environment using a combination of a passthrough filter and Euclidean cluster extraction. The same procedure for feature extraction and comparison used for the ground truth models is applied to these objects. The metric used to evaluate the success of the SLAM algorithm is the error between the ground truth comparison of these objects and the relations between their reconstructed counterparts. Experiments were conducted using a ground vehicle robot in both simulation and a laboratory environment. The experiments were used to evaluate the performance of different SLAM algorithms, and to compare the evaluation against evaluation metrics used in literature.

The proposed system is designed to be extendable for use with any point cloud feature. It aims to help evaluate the suitability of algorithms within scenarios not properly covered within available data sets.