AI – Mixed Reality Framework for the Enhancement of Human–Robot Interaction

The rapid convergence of collaborative robotics and artificial intelligence (AI) within industrial and academic sectors has created a critical need for intuitive, safe, and high-fidelity interaction platforms. This chapter details the design of a scalable, AI-augmented solution to advance safe human–robot collaboration across high-stakes industrial workflows and educational settings. By integrating unity-based simulation with the robot operating system, the architecture establishes a “Dual Twin” system: (i) virtual twin: utilizes AI for collision-free motion planning in Cartesian and joint spaces, alongside point-cloud-based grasp validation; (ii) real twin: mirrors physical robot states with high fidelity to ensure execution accuracy. A key feature of the framework is the implementation of voice-command-enabled interaction, allowing users to guide robotic tasks through natural language, which is processed and executed within a mixed reality environment. The system leverages connectivity to enable seamless trajectory transfer and predeployment validation of complex pick-and-place operations. Experimental results demonstrate the framework’s robustness, highlighting stable networking latency and precise control under voice-based operation.