The limitations of both physics-based and data-driven models have led to efforts to integrate physics into data-driven methods. This integration seeks to combine the adaptability of data-driven methods with the accuracy, reliability, and generalization of physics-based models, developing more efficient, interpretable, and robust robotic systems capable of real-time autonomous operation. However, developing these physics-informed, data-driven models comes with several challenges:​

• Formulating knowledge of physics as priors for data-driven models is underexplored.
• Designing a learning strategy for a shared latent space that enables unified understanding and physics-informed decision-making is challenging.
• Transfer learning is vital in robotics due to limited training data and the need to adapt to diverse environments (transfer from source domain to action domain).

This thesis tackles the challenges of achieving full robotic autonomy by exploring advanced physics-based data-driven models for vision and dynamic control. By combining the strengths of physical principles with data-driven adaptability, these models enable more accurate predictions, better adaptation to uncertainties, and improved task performance in complex environments. The work examines common data-driven models in robotics, conventional physics-based approaches for dynamics and vision, and statistical techniques for integrating prior knowledge, offering a roadmap for implementing physics-informed data-driven models effectively.

A key concept in this thesis is the use of energy functions to bridge conventional physics and machine learning within a unified framework. It begins by applying Lagrangian energy frameworks as model priors to optimize data-driven models for manipulation control, showing how integrating physics improves generalization. The focus then shifts to developing machine-learning-based priors as energy functions to enhance robotic vision’s generalization in the diffusion process. This work lays the foundation for scaling physics-informed AI to tackle more complex tasks and sensory modalities, progressively building toward a unified framework for intelligent systems