EndoARSS: Adapting Spatially-Aware Foundation Model for Efficient Activity Recognition and Semantic Segmentation in Endoscopic Surgery

Endoscopic surgery is the gold standard for robotic‐assisted minimally invasive surgery. However, the complexity of surgical scenes, characterized by high variability in different surgical activity scenarios, presents challenges for surgical environment understanding. Deep learning models often struggle with cross‐activity interference, leading to suboptimal performance in each downstream task. To address this limitation, we explores multitask learning which utilizes the interrelated features between tasks to enhance overall task performance. In this article, we proposes EndoARSS, a novel multitask learning framework specifically designed for endoscopy surgery activity recognition and semantic segmentation. Built upon the DINOv2 foundation model, EndoARSS integrates low‐rank adaptation to facilitate efficient fine‐tuning while incorporating task efficient shared low‐rank adapters to mitigate gradient conflicts across diverse tasks. Additionally, we introduces the spatially‐aware multi‐scale attention that enhances feature representation discrimination by enabling cross‐spatial learning of global information within complex surgical environments. In order to evaluate the effectiveness of our framework, we presents three novel datasets, tailored for endoscopic surgery scenarios. Extensive experiments demonstrate that EndoARSS achieves remarkable performance across multiple benchmarks, improving both accuracy and robustness compared to existing models. These results underscore the potential of EndoARSS to advance AI‐driven surgical systems, offering valuable insights for enhancing surgical safety and efficiency.