SeqSAM: Autoregressive Multiple Hypothesis Prediction for Medical Image Segmentation using SAM

Pre-trained segmentation models are a powerful and flexi-ble tool for segmenting images. Recently, this trend has ex-tended to medical imaging. Yet, often these methods only produce a single prediction for a given image, neglecting in-herent uncertainty in medical images, due to unclear object boundaries and errors caused by the annotation tool. Mul-tiple Choice Learning is a technique for generating multiple masks, through multiple learned prediction heads. However, this cannot readily be extended to producing more outputs than its initial pre-training hyperparameters, as the sparse, winner-takes-all loss function makes it easy for one prediction head to become overly dominant, thus not guaranteeing the clinical relevancy of each mask produced. We introduce SEQSAM, a sequential, RNN-inspired approach to generating multiple masks, which uses a bipartite matching loss for ensuring the clinical relevancy of each mask, and can pro-duce an arbitrary number of masks. We show notable im-provements in quality of each mask produced across two pub-licly available datasets. Our code is available at https://github.com/BenjaminTowle/SeqSAM.