The aim of this paper is to introduce a structural dissimilarity measure which allows to detect outliers in automatically extracted landmark pairs in two images. In previous work, to extract landmarks automatically, candidate points have been defined using invariance criteria coming from differential geometry such as maximum curvature;​ or they are statistical entities such as gravity centers of confiners, where the confiners are defined as the connected components of the level sets. After a first estimation of the semi-rigid transformation (representing translation, rotation, and scaling) relating the candidate point sets, outliers are detected applying the euclidian distance between corresponding points. However, this approach does not allow to distinguish between real deformations and outliers coming from noise or additional features in one of the images. In this paper, we define a structural dissimilarity measure which we use to decide if two associated candidate points come from two corresponding confiners. We select landmarks pairs with a dissimilarity value smaller than a given threshold and we calculate the affine transformation relating at best all selected landmark pairs. We evaluate our technique on successive slices of a MRI image of the human brain and show that we obtain a significantly sharper error diminution using the new dissimilarity measure instead of the euclidian distance for outlier rejection.