Rebuttal by Authors
We deeply appreciate the reviewers for their insightful feedback!
**We would like to emphasize the importance of our contribution:**
**Significance:**
The removal of outlier matches is a critical step in Structure-from-Motion (SfM). Methods like ESFM (Moran et al., 2021) and GASFM (Brynte et al., 2023), which do not effectively handle outliers, have been applied only in limited scenarios—such as the outlier-free Olsson’s dataset (Olsson & Enqvist, 2011), which was captured with a single camera, small motion, and no light changes. When these methods are applied to more challenging datasets, such as those consisting of internet photos, they result in poor accuracy. Our work introduces an effective, deep network-based approach to outlier removal in a global SfM setting, addressing this critical gap.
**Novelty:**
We have introduced the novel task of multi-view outlier removal and will make our benchmark and datasets publicly available. We believe future research can build on our work to further improve performance. Similar to the advancements seen in classical global methods, which culminated in GLOMAP (Pan et al., 2024) being on par with the state-of-the-art incremental COLMAP, we hope our contribution will serve as a foundation for future progress.
**Approach:**
Our goal is to simultaneously recover both camera poses and 3D structure. To handle outliers in this context, we integrate an inlier/outlier classification module within a sets-of-sets permutation-equivariant architecture. This architecture efficiently manages point tracks while respecting their symmetries, thus eliminating the need for data augmentation. We further explain the details of this equivariant architecture below.
**Importance of Equivariance:**
Our network classifies individual track points as inliers or outliers by considering both their track and image memberships. The sets-of-sets-equivariant architecture allows the hidden layers for each track point to interact with other points both within and across cameras. This eliminates the need for exhaustive augmentation processes. Our comparison to a PointNet-based architecture in Table 7 demonstrates the importance of leveraging sets-of-sets equivariance in improving performance.
**Results:**
Our results demonstrate that our method can efficiently handle challenging datasets, including those with hundreds of internet photos, achieving great accuracy. In contrast, existing deep learning methods like ESFM and GASFM fail to handle such datasets due to their inability to remove outliers effectively. Other robust deep learning-based methods also struggle with large-scale datasets of internet photos. We believe this marks an important step in advancing deep learning as a leading technique for large-scale SfM.
**We have made the following revisions in the text:**
1) Thanks to the insightful question from reviewer oVEX, we identified a bug in the GLOMAP code. Specifically, in some reconstructions, unregistered cameras (cameras without any associated 3D points) were not filtered out. This oversight negatively impacted GLOMAP’s performance.
We have corrected this issue and updated the GLOMAP results, which affected the 1DSfM and MegaDepth datasets (Tables 1, 2, 9, and 10). While the updated results show an improvement for GLOMAP, the overall conclusions of our comparisons remain unchanged.
2) The best-performing method is now marked in **bold and underlined**, we hope this makes the tables clearer for readers.
3) At the request of the reviewers, we have added the mean errors to Tables 1, 2, 9, and 10 for the 1DSfM and Megadepth datasets to enhance clarity.