DC-SAM: In-Context Segment Anything in Images and Videos via Dual Consistency

In-context segmentation, also known as one-shot segmentation, aims to segment objects based on a single labeled example. While the Segment Anything Model (SAM) excels in interactive segmentation, it is not inherently designed for in-context tasks. To bridge this gap, we propose a new Dual Consistency SAM (DC-SAM), a prompt-tuning framework that adapts SAM and SAM2 for both image and video in-context segmentation. Instead of relying solely on pre-trained backbones, DC-SAM enhances the prompt encoder by generating high-quality visual prompts through feature fusion. Furthermore, we introduce a novel cycle-consistent cross-attention mechanism to enforce alignment between fused features and visual prompts, complemented by a dual-branch design incorporating discriminative positive and negative prompts. Additionally, we extend DC-SAM to the video domain via a novel mask-tube training strategy. To facilitate research, we curate the first In-Context Video Object Segmentation (IC-VOS) benchmark. Extensive experiments demonstrate that DC-SAM achieves state-of-the-art performance, yielding 55.5 mIoU (+1.4) on COCO-20<inline-formula><tex-math notation="LaTeX">$^{i}$</tex-math><alternatives><mml:math><mml:msup><mml:mrow/><mml:mi>i</mml:mi></mml:msup></mml:math><inline-graphic xlink:href="qi-ieq1-3646919.gif"/></alternatives></inline-formula>, 73.0 (+1.1) mIoU on PASCAL-5<inline-formula><tex-math notation="LaTeX">$^{i}$</tex-math><alternatives><mml:math><mml:msup><mml:mrow/><mml:mi>i</mml:mi></mml:msup></mml:math><inline-graphic xlink:href="qi-ieq2-3646919.gif"/></alternatives></inline-formula>, and a <inline-formula><tex-math notation="LaTeX">$\mathcal {J} { \& amp;} \mathcal{F}$</tex-math><alternatives><mml:math><mml:mrow><mml:mi mathvariant="script">J</mml:mi><mml:mo>&</mml:mo><mml:mi mathvariant="script">F</mml:mi></mml:mrow></mml:math><inline-graphic xlink:href="qi-ieq3-3646919.gif"/></alternatives></inline-formula> score of 71.52 on IC-VOS.