Consensus Learning with Deep Sets for Essential Matrix Estimation

Summary

This work proposes a simpler yet effective network architecture based on Deep Sets for the estimation of essential matrices. The method is based on three key properties: 1. A two-stage noise-aware training scheme first uses noise-free inlier matches for training, and then uses original matches and full loss for training; 2. A noise head predicts the displacement noise in inlier matches. 3. A classification head classifies point matches as either inliers or outliers. Evaluations show that the method outperforms previous methods in terms of accuracy and generalization.

Strengths

1. The proposed network architecture is simple yet effective. 2. NAC block represents denoising and inlier/outlier classification effects, well validated through a toy example, i.e., a line-fitting task in Fig. 2. 3. The key properties proposed are shown to be effective in the ablation study. 4. The paper is well-structured and is generally easy to follow.

Weaknesses

1. While the DeepSets-based architecture is shown to be effective, traditional fully convolutional networks, such as UNet, are also applicable. Is there any strong reason why a DeepSets-based network is preferred and explored first? 2. The authors remove half of the input matches to test the effect of outlier pruning. Is this approach reasonable? Is this value adjustable? It would be better to add an ablation study showing performance changes as the number of removed matches varies. Furthermore, this is an indirect method to validate the effectiveness of the proposed inlier/outlier classification head. Why was the ablation study not conducted with and without this head? 3. While the paper provides an ablation study on the proposed key properties, it lacks an ablation of the network architecture and hyperparameters, such as the number of set layers, the dimension of the Set Encoder, etc, but this would also be valuable to know — for example, how much of an impact do these parts of the network have on the final results? 4. The paper lacks details on applying the Optimal Triangulation Method to set inlier/outlier labels and obtain ground truth, noise-free inlier keypoints. Including these details would help readers better understand the training process. 5. What is the unit of the y-axis in Fig. 3? 6. To the best of my knowledge, the ground truth poses in the SUN3D dataset are inaccurate (see supplemental material in {1}). Why did the authors use this dataset solely to follow the evaluation protocol of previous methods? For indoor scenes, is it more suitable to use ScanNet dataset {2} for evaluation as [42]? 7. While the paper claims that the network is simpler than existing methods, the multiple NAC blocks and the noise regression module may introduce a certain level of complexity. It would be better to provide experiments and discussions on the computational efficiency of the proposed method compared to existing methods, including runtime and model parameter scale, which are critical for real-world applications. {1} Bellavia, Fabio. “Image Matching by Bare Homography.” IEEE Transactions on Image Processing 33 (2023): 696-708. {2} Dai, Angela, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas A. Funkhouser and Matthias Nießner. “ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes.” IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017): 2432-2443.

Questions

1. While the DeepSets-based architecture is shown to be effective, traditional fully convolutional networks, such as UNet, are also applicable. Is there any strong reason why a DeepSets-based network is preferred and explored first? 2. The authors remove half of the input matches to test the effect of outlier pruning. Is this approach reasonable? Is this value adjustable? It would be better to add an ablation study showing performance changes as the number of removed matches varies. Furthermore, this is an indirect method to validate the effectiveness of the proposed inlier/outlier classification head. Why was the ablation study not conducted with and without this head? 3. While the paper provides an ablation study on the proposed key properties, it lacks an ablation of the network architecture and hyperparameters, such as the number of set layers, the dimension of the Set Encoder, etc, but this would also be valuable to know — for example, how much of an impact do these parts of the network have on the final results? 4. The paper lacks details on applying the Optimal Triangulation Method to set inlier/outlier labels and obtain ground truth, noise-free inlier keypoints. Including these details would help readers better understand the training process. 5. What is the unit of the y-axis in Fig. 3? 6. To the best of my knowledge, the ground truth poses in the SUN3D dataset are inaccurate (see supplemental material in {1}). Why did the authors use this dataset solely to follow the evaluation protocol of previous methods? For indoor scenes, is it more suitable to use ScanNet dataset {2} for evaluation as [42]? 7. While the paper claims that the network is simpler than existing methods, the multiple NAC blocks and the noise regression module may introduce a certain level of complexity. It would be better to provide experiments and discussions on the computational efficiency of the proposed method compared to existing methods, including runtime and model parameter scale, which are critical for real-world applications. {1} Bellavia, Fabio. “Image Matching by Bare Homography.” IEEE Transactions on Image Processing 33 (2023): 696-708. {2} Dai, Angela, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas A. Funkhouser and Matthias Nießner. “ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes.” IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017): 2432-2443.

Rating

6

Confidence

5

Soundness

2

Presentation

3

Contribution

3

Limitations

NA

Summary

The paper proposes a deepsets based architecture for essential matrix estimation. The proposed architecture is required to overcome positional nosie, be capable of handling outlier matches which comprise a significant portion of the input data, generalize to unseen image pairs and be capable of handling arbitrary number of point matches. The propose archchitecture comes with the following: - a two-stage noise-aware training scheme - a noise head for predicting positional inlier nosie, and - a classification head for inlier/outlier discrimination. The architecture consists of - Nosie-aware Concensus Blocks (NAC) - These are stacks of set encoders (ie Deepsets) used to obtain latent representation of input sets and for classification of points as inliers or outliers. The ouput of these blocks are used to estimate the essential matrix.

Strengths

- A simple architecture is proposed that offers better empirical performance compared to current methods.

Weaknesses

- Ablation on the choice of set functions is missing.

Questions

- Can an ablation on the choice of set functions be provided? For instance, recent methods such as [1] and [2] have been shown to perform better than deepsets for modeling interactions over sets and these might provide further performance boost to the proposed architecture. - Also, how does the number of deepset layers affect performance? [3] shows that special techniques are required to ensure effective learning when multiple deepsets layers are stacked. Curious how this correlates with your experiments. ## References [1] Lee, Juho, et al. "Set transformer: A framework for attention-based permutation-invariant neural networks." International conference on machine learning. PMLR, 2019. [2] Bruno, Andreis, et al. "Mini-batch consistent slot set encoder for scalable set encoding." Advances in Neural Information Processing Systems 34 (2021): 21365-21374. [3] Zhang, Lily, et al. "Set norm and equivariant skip connections: Putting the deep in deep sets." International Conference on Machine Learning. PMLR, 2022.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

While no discussion of limitations is provided, i do not see any immediate negative societal impact of this work.

Summary

In this work, authors propose NACNet (Noise Aware Consensus network) for the robust essential matrix estimation. For this purpose authors apply DeepSets based architecture that predicts inlier / outlier class as well as inlier displacement error estimation. Authors also propose a two-stage training: (1st) train with noise-free matches by disabling noise-prediction head and (2nd) train on real data with noise. Extensive experiments with in-scene, cross-scene, cross-dataset, cross-feature generalization and extensive ablation study shows that propose simpler method is superior compared to other deep-learning based consensus methods with complicated architectures.

Strengths

Overall, paper is well written with sufficient mathematical rigor to prove authors claim that "Our network achieves accurate recovery that is superior to existing networks with significantly more complex architectures." (1) Predicting noise in key-points without feature-descriptor is well-motivated and proven to work through ablation study. (2) Architecture design is simple with DeepSet based 3 NAC block with Model Regression head on top. The network predicts confidence along with inlier-outlier classification (similar to [1]) that allows robust estimation of Essential Matrix via differentiable DLT. (3) Whole network is end-to-end differentiable. (4) Benchmarkin &, experiments across different datasets shows good generalization capability of the proposed method. This can have high impact in relative-pose estimation that is essential to multi-view geometry. References [1] Yi, Kwang Moo, et al. "Learning to find good correspondences." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

Weaknesses

Paper discusses predicting noise-free key-points as an output, however lacks sufficient discussion around difference between noisy and noise-free points. (1) How does keypoint noise changes feature points? It would have been nicer to see some visualization / empirical matrix of change in key-point locations. of noise-free key-points predicted. (2) Another interesting fact that is not tested: does the change in key-point position is to fit into the "Model" or is it actually becoming sub-pixel accurate as shown in [2] (3) Some more comparison when input has less "outlier" / more "outlier". Empirical evidence around effect of cardinality is also missing. References [2] Lindenberger, Philipp, et al. "Pixel-perfect structure-from-motion with featuremetric refinement." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

Questions

L97: "Our work is first one to apply keypoint position correction without using a geometric model" -- Last loss that defines distance between noise-free keypoints and prediction uses "geometry" (as in optimal triangulation) to get noise-free points. What does it mean here when authors say "noise-free"? What is the size of the model? How fast is the inference? It would be interesting to know how close are we compared to RANSAC in-terms of runtime?

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

Authors discuss limitation of their approach especially not-perfect generalization to other datasets / features, adding degeneracy test block inside the model. This work doesn't have societal impact, so authors ignore that.

Summary

The authors propose a method to tackle the traditional computer vision problem of estimating the essential matrix between two camera views of the same scene based on a set of point matches. The method distinguishes between inlier / outlier matches, and explicitly models the displacement noise in the inlier matches using a “Noise Aware Consensus Network”. The model is trained in two stages: first, the model sees only noise-less inlier matches with outlier matches, and secondly real world blends of noisy inlier and outlier matches. The authors show that this framework is able to compute accurate essential matrices with a variety of different image descriptors over different datasets.

Strengths

1. The paper presented convincing performance as compared to the baselines, especially in the cross-scene / cross dataset settings. This attribute is perhaps most important for practical applications, and shows that the explicit noise reasoning scheme appears to work well. 2. The ablation studies shed light on the contributions of the 2-stage training and keypoint denoising steps. These will help guide future research. 3. The architecture and training details are quite clear. Together with the authors’ promise of releasing the code, this work is a meaningful contribution to the research community. 4. The paper is easy to read and navigate; different parts of the design are motivated clearly.

Weaknesses

1. It would be important to understand how the capacity of the NACNet architecture compares with the baseline methods - while it indeed appears simpler and more streamlined than pipelines such as BCNet, it would be important to understand whether the complexity has simply been wrapped into higher capacity models. 2. It seems that correspondence pruning should have a large impact on the performance of the model as well, but not a lot of attention is given to this area (e.g. in ablation studies); it seems that only some qualitative results are shown in Fig 4 and in the appendix. However, what is the actual performance of the prediction of Y_i (inlier / outlier labeling) in the different settings (in-scene, cross-scene, cross-dataset, etc)? Does this have a large impact on the performance?

Questions

It would be great to read the authors’ responses to the issues raised in the weaknesses section.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors discussed the limitations of their work well with respect to the requirements in the Checklist.

Thanks for taking time to write rebuttal.

Reply W1: Thanks for providing the graph. What happens for image pairs with high pose error? (or what are some highest key-point movement you see between before and after denoising?) What I am trying to understand is what the noise-prediction block is learning? Is it learning through image gradient / feature gradients? Or is there something else going on here. Reply W2: Thanks, all clear. Reply W3: Make sense. This is a good insight, if possible I'll encourage authors to make space for it in the paper / appendix. Reply Q1: Thanks, this clarifies. Reply Q2: Amazing! I think this is very important. 11ms is highly desirable. Also, low memory usage shows that this method can also be used on smaller memory constrained devices. Overall, I am satisfied with authors response and the paper does show practical and novel algorithm for consensus learning. I will keep my rating to 8.

Thanks for the authors' feedback.

I read the rebuttal and all other reviews. This rebuttal has addressed most of my concerns, but some questions still remain. 1. For the reply to Q6, could you please provide the results of other baseline methods on ScanNet dataset? 2. For the reply to Q7, somewhat counterintuitively, the model achieves a faster runtime despite using more parameters.