Real-time Monocular 2D and 3D Perception of Endoluminal Scenes for Controlling Flexible Robotic Endoscopic Instruments

Endoluminal surgery offers a minimally invasive option for early-stage gastrointestinal and urinary tract, but is limited by basic surgical tools and a steep learning curve. Robotic systems, particularly continuum robots, provide flexible instruments that enable precise, intuitive tissue resection in confined spaces, potentially improving outcomes. This article presents an integrated visual perception platform for a continuum robotic system in endoluminal surgery. Our objective is to leverage monocular endoscopic image-based perception algorithms to accurately identify the position and orientation of flexible instruments and measure their distances from surrounding tissues. This thorough understanding of continuum robots and surgical scenes enhances the robustness of robotic procedures. We introduce 2-D and 3-D learning-based perception algorithms and develop a physically realistic simulator that models the dynamics of flexible instruments. This simulator features a pipeline for generating realistic endoluminal scenes, enabling control of flexible robots in a realistic environment and substantial data collection. Using a continuum robot prototype, we conducted extensive evaluations, including module assessments and system-level evaluation of the perception platform. Results demonstrate that our perception algorithms significantly improve control of flexible instruments, reducing manipulation time by over 70% for trajectory-following tasks and enhancing the understanding of complex surgical scenarios, leading to robust endoluminal surgeries.