ETO:Efficient Transformer-based Local Feature Matching by Organizing Multiple Homography Hypotheses

Summary

This paper proposes an efficient transformer-based local feature matching approach named ETO. ETO consists of three steps: hypothesis estimation, segmentation and refinement. The first and second steps can obtain coarse matching results whose resolution is similar to LoFTR’s coarse results. The third step can provide fine-level matching results. The hypothesis estimation performs the Transformer computation on the 1/32 resolution (rather than the 1/8 resolution in LoFTR), which is the core design that reduces the inference time. Experimental results demonstrate that ETO provides a good balance between accuracy and speed.

Strengths

1. The motivation of this paper is clear and valuable. Transformer-based approaches are prevalent in the area of feature matching. Nonetheless, it is still an open problem how to design an efficient and accurate transformer-based model. 2. The technical design of ETO is novel. The manner of predicting the local homography parameters on a very coarse resolution (and then refining them) differs from most of the existing coarse-to-fine transformer-based approaches. 3. The accuracy of ETO is acceptable, considering its superiority in inference speed.

Weaknesses

**1. This paper lacks some important technical details.** 1.1 What is the computation manner of the Transformer before the hypothesis estimation step? In line 237, the authors state that "then we perform transformer five times at M_1". Does the transformer contain some cross-attention processes? Or does it only involve the self-attention? 1.2 The computation process of the hypothesis estimation step is confusing. The authors seem to use the notations "M_1, M_2, M_3, f_i^1" to represent the features of both the source and target images. Such definitions make many statements in Sec. 3.2 hard to understand. For example, the meaning of the variable M_1 seems inconsistent in the statements "For each unit i on M_1" (line 144) and "within the neighborhood of the target units on M_1" (line 155). 1.3 The computation architecture of the segmentation step is unclear. The authors should provide the computation details on how to predict the classification confidence from the intermediate features. The statement in lines 177-180 is too brief to understand. 1.4 Why is the classification label in the segmentation step termed as the "pseudo" ground truth (line 181)? To my understanding, the classification label of every unit should be obtained from the ground truth camera pose and depth. It should be the "normal ground truth" rather than the "pseudo ground truth" if the above understanding is correct. The authors should clarify this detail. **2. Some details should be further clarified.** 2.1 The statement "while we only need to feed 300" (line 50-51) was not clarified in the subsequent text. Is it the unit number on the 1/32 resolution for a 640x480 input image? 2.2 In the segmentation step, what is the size of output segmentation results for a local input window whose size is 3x3? Is it 4x4? Or 12x12? 2.3 The statement in lines 304-305 should be further discussed. The authors just state that ETO is better on the more difficult YFCC100M without discussing the probable reason. 2.4 The title "Hypotheses Estimation" (line 137) should be "Hypothesis Estimation". **3. Some intermediate results should be visualized.** Some real intermediate results after the hypothesis estimation/segmentation steps should be visualized to show how these steps provide appropriate predictions. The virtual intermediate "results" in Figure 3 are helpful to understand, but the actual results are still necessary.

Questions

Please provide more discussions and experimental results to address the above weaknesses.

Rating

7

Confidence

4

Soundness

3

Presentation

2

Contribution

4

Limitations

The authors have discussed the limitations.

Summary

The authors propose a local feature matching method that leverages homography to accelerate the transformer-based feature matching pipeline. Additionally, they employ unidirectional cross-attention in the refinement stage to further reduce computational overhead. Experimental results demonstrate the efficiency and effectiveness of this approach.

Strengths

1. The paper is well-written and easy to understand. 2. Integrating homography as theoretical guidance into the transformer pipeline is a commendable approach, enhancing the transformer-based pipeline with theoretical support. 3. Experimental results demonstrate that the proposed method achieves much smaller time usage, validating its efficiency in practice.

Weaknesses

1. Though the time usage is decrease, the accuracy is also decrease from Tab.1&2&3. 2. The paper lacks significant citations in feature matching methods, such as Efficient LoFTR[1], RoMa[2]. Some of these works also focus on improving efficiency in feature matching and should be referenced to provide a comprehensive background. Including these methods in the experiments would offer a more thorough comparison of the proposed approach's performance. 3. Though the authors acknowledge that some other methods (e.g., [14, 38]) are better in certain aspects (line356-357), it would be beneficial to include a comparative analysis with these methods. This would provide a clearer understanding of the strengths and weaknesses of the proposed approach. [1]. Wang, Yifan, et al. "Efficient LoFTR: Semi-dense local feature matching with sparse-like speed." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [2] Edstedt, Johan, et al. "RoMa: Robust dense feature matching." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Questions

No question to the authors.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have addressed the limitations in their paper.

Summary

This paper proposes a novel framework for efficient transformer-based local feature matching. Transformer-based local feature matching usually contains two stages: a coarse matching stage which applies self-attention and cross-attention on coarse-level features (usually H/8 x W/8) to obtain coarse matches, and a refinement stage that refines the coarse matches in a local window based on fine-grained features. This paper proposed two methods to improve the efficiency of this pipeline. First, the coarse matching is performed on a even coarser level (H/32 x W/32) to reduce the computational cost of the costly attention operations. Additionally, homography hypothesis for each patch is estimated at this level. Leveraging the piece-wise smooth prior, the matching at H/8 x W/8 resolution is directly approximated by selecting the most probable homography hypothesis in local windows using a proposed segmentation technique. Second, in the refinement stage, the bi-directional cross attention is reduced to uni-direction one. Experiments show that the proposed framework achieves comparable performance with less inference time compared with state-of-the-art transformer-based and local feature matching algorithms.

Strengths

1. The idea of using piece-wise smooth prior to accelerate transformer-based feature matching is novel and promising, and the supervision and the homography hypothesis re-selection algorithms are carefully designed. 2. Comprehensive ablation studies show that the key design features, including homography hypothesis proposal, homography hypothesis re-selection and uni-directional cross attention have positive effects on the performance and are necessary.

Weaknesses

1. The symbols used in the method part are too complicated, making it hard to read. 2. The symbol $\mathscr{H}$ used in figure 4 is undefined in the text. 3. The framework adopts two-stage training, possibly making it hard to train and limiting the accuracy.

Questions

1. Do you have any plan to reorganize the symbols? 2. If the framework is trained end-to-end, will it yield acceptable performance?

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors have adequately addressed the limitation of this work in the conclusion and limitations section of the paper. Enabling the network to be trained end-to-end, allowing for dense matching and further improving accuracy are my suggestions for directions of improvement.

Summary

This paper presents a local feature matching method based on the multiple homography hypotheses. The paper explicitly introduces the homography hypotheses for coarse matching, combined with homography segmentation, cross-attention, and sub-pixel refinement to obtain fine matching results. The introduction of the homography hypotheses significantly reduces the number of input tokens for the attention mechanism, and with further model modification, the algorithm proposed in this paper has achieved a substantial increase in inference speed while maintaining performance as much as possible. Experiments on multiple datasets have proven the effectiveness and efficiency of the proposed method.

Strengths

Clear motivation, reasonable structural design, convincing experiments

Weaknesses

-The pre-compiled transformer model may have some impact on inference speed, can additional tests be conducted on the efficiency of a normal transformer module? -Lack of the latest state-of-the-art comparison: Some of the latest methods, such as RoMA[1] and EfficientLoFTR[2], have been made public at an early stage, and thus it is necessary to compare and analyze with these methods. [1] Edstedt, Johan, et al. "RoMa: Robust dense feature matching." CVPR 2024. [2] Wang, Yifan, et al. "Efficient LoFTR: Semi-dense local feature matching with sparse-like speed." CVPR 2024.

Questions

-In Section 3, it is 2-stage training adopted, but in Implementation Details, the training phase contains 3 stages. Why there is a joint training stage at first? Moreover, the joint training is usually conducted for the final finetuning, why it is the first stage in your strategy? -For Basic Refinement w/ Segmentation in Section 4.4, why it is the same training hours instead of the same training samples? Early stopping may lead to performance drop. -In the introduction, the authors claim that the multiple self- and cross-attention in the fine-level stage are redundant, is there any numerical results? What if you stack several uni-directional cross-attention in your method?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

see Weaknesses

Summary

This paper propose an efficient framework to reduce the computational load of transformer-based matching approaches. It is a coarse-to-fine two-stage solution. During the coarse matching phase, multiple homography hypotheses are estimated to approximate continuous matches. Each hypothesis encompasses few features to be matched, reducing the number of features that require enhancement via transformers. In the refinement stage, the unidirectional cross-attention is proposed to replace the classical bidirectional self-attention and cross-attention mechanism, which further decreases the cost of computation. Comprehensive evaluations on other open datasets such as Megadepth, YFCC100M, ScanNet, and HPatches are presented to demonstrate its efficacy.

Strengths

The whole idea makes sense to me. The experimental studies are quite comprehensive.

Weaknesses

Some parts of the description are quite vague to me, which may hinder its reproducibility. Some of the experimental results may need further clarification.

Questions

1. I think some descriptions in Sec 3.4 need further clarification, e.g., Line 192-193 $\hat{f^3_j}$ is computed by querying the $f3_k$ with …, is the query done by nearest neighboring searching? 2. How is $\triangle P_j^t$ in Line 188 obtained or is it predicted? 3. In Figure 3, is there an MLP after cross-attention block? If yes, what is the input/output. Also, the presence of “Module 3” is confusing, should it contain the cross attention? 4. In Table 4, it seems the usage of bidirectional cross-attention performs worse than that of uni-directional cross-attention at 5. Is that reasonable, please give some analysis. 5. From Table 2, the performance of lightGlue is much worse than ASpanFormer, LoFTR. Interestingly, ASpanFormer is better than LoFTR with a small margin in the Megadepth dataset, similar to the results reported in LightGlue. The performance gap between LightGlue and LoFTR is much larger than that in LightGlue, could you please explain? Moreover, why LightGlue is not reported in the dataset of Scannet?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

1. Some of the network architectures are not depicted in figures, making it difficult to know the overall diagram of the method. 2. The two-stage training process may not be easily trained. 3. It would be better to present some failure case studies.