Simulation-to-Real domain adaptation with teacher-student learning for endoscopic instrument segmentation

Purpose: Segmentation of surgical instruments in endoscopic videos is\nessential for automated surgical scene understanding and process modeling.\nHowever, relying on fully supervised deep learning for this task is challenging\nbecause manual annotation occupies valuable time of the clinical experts.\n Methods: We introduce a teacher-student learning approach that learns jointly\nfrom annotated simulation data and unlabeled real data to tackle the erroneous\nlearning problem of the current consistency-based unsupervised domain\nadaptation framework.\n Results: Empirical results on three datasets highlight the effectiveness of\nthe proposed framework over current approaches for the endoscopic instrument\nsegmentation task. Additionally, we provide analysis of major factors affecting\nthe performance on all datasets to highlight the strengths and failure modes of\nour approach.\n Conclusion: We show that our proposed approach can successfully exploit the\nunlabeled real endoscopic video frames and improve generalization performance\nover pure simulation-based training and the previous state-of-the-art. This\ntakes us one step closer to effective segmentation of surgical tools in the\nannotation scarce setting.\n