The ‘bag of visual features’ image representation was applied to create generic microstructural signatures that can be used to automatically find relationships in large and diverse microstructural image data sets. Using this representation, a support vector machine (SVM) was trained to classify microstructures into one of seven groups with greater than 80% accuracy over 5-fold cross validation. In addition, the bag of visual features was implemented as the basis for a visual search engine that determines the best matches for a query image in a database of microstructures. These novel applications demonstrate the potential and the limitations of computer vision concepts in microstructural science.