With the fast-paced proliferation of robots in our daily lives, it is especially important that the human experience is prioritized for human-robot interactions and automated systems. We investigated how human eye movements can be leveraged in real-time by robotic systems to enhance the experience of humans that interact with and/or observe semi-autonomous robots. Specifically, we studied spatiotemporal relationships between human eye gaze, human intent, and human-automation trust for activities of daily living.

In Study #1, we identified features from 3D gaze behavior for use by machine learning classifiers for the purpose of action recognition. We investigated gaze behavior and gazeobject interactions as participants performed a bimanual activity of preparing a powdered drink. We generated 3D gaze saliency maps and used characteristic gaze object sequences to demonstrate an action recognition algorithm.

In Study #2, we introduced a classifier for recognizing action primitives, which we defined as triplets having a verb, “target object,” and “hand object.” Using novel 3D gaze-related features, a recurrent neural network was trained to recognize a verb and target object. The gaze object angle and its rate of change enabled accurate recognition and a reduction in the observational latency of the classifier. Using a non-specific approach for indexing objects, we demonstrated potential generalizability of the classifier across activities.

In Study #3, we evaluated subjective and objective measures of trust in human-automation interactions. We compared real-time physiological responses and trust levels (reported via joystick) to the state-of-the-art method of post-trial Likert surveys. Our results suggest that eye gaze features and heart rate are effective, nonintrusive metrics for real-time monitoring of human trust in automated systems.

In summary, we developed machine learning-based action recognition methods using novel 3D gaze-related features, and we related uncertainty in robot competence to realtime measures of trust variation. Our work establishes a foundation for enhancing humanrobot collaborative systems by leveraging eye tracking for intention recognition and trust calibration.