Activating Self-Attention for Multi-Scene Absolute Pose Regression

Summary

In this paper, the authors focus on improving the performance of multi-scene absolute pose regression models based on transformers. From statical analysis, the authors assume that the distortion between Q and K features in self-attention and the learnable position encoders are the reasons. Therefore, a Q-K alignment loss and the fixed-pose encoding method are adopted and experiments demonstrate the efficacy on indoor 7scenes and CambridgeLandmarks datasets.

Strengths

The strengths of this paper are as follows. 1. Originality. The undistortion of Q-K features in self-attention is not a new direction as also mentioned in the related works section. While this paper may be the first one to apply it to the multi-scene absolute pose regression task and approve the efficacy. Besides, the authors also demonstrate that the fixed-position encoding works better than the learnable position encoding because learnable position encoding breaks the order the input sequences. These contributions are useful for improving the accuracy of the pose regression as shown in the experiments. 2. Quality. The proposed algorithm is easy to follow, and the paper is well written.

Weaknesses

The proposed method is easy to follow as I mentioned before, so I don’t have many concerns. Several minor concerns are as follows. 1. Contribution. I can understand this paper starts from analyzing the application of the absolute pose regression, but the key technique about the distortion of Q-K features in self-attention mechanism comes from previous works [19]. As this is the major contribution of this paper, I am not very sure if it is enough. Although an additional Q-K alignment strategy is proposed as an additional contribution, its improvement over prior strategies such as [19] is not significant (0.02m and 0.17 deg). These improves may come from a suitable learning rate or more proper hyper-parameters balancing different losses. 2. Results. Overall, the improvements against the baseline method MSTransformer [17] on indoor (0.02m, 0.64deg) and outdoor (0.09m, 0.44deg) datasets are not significant, which further degrades the contribution of this paper.

Questions

Please see the weakness.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

Yes.

Summary

The paper investigates multi-scene absolute pose regression from a new perspective: query-key embedding space. Focusing on distortion of queries and keys, the paper solutions to activate self-attention, which includes an auxiliary loss to align queries and keys and fixed sinusoidal positional encoding. Experimental results demonstrate that the proposed method outperforms existing MS-APR methods in outdoor Cambridge Landmarks and indoor 7 Scenes.

Strengths

1.The paper proposes new analysis that focuses on distortion of query-key embedding space. 2.The paper presents solutions to reduce the distortion that covers an auxiliary loss that aligns queries and keys and fixed sinusoidal positional encoding. 3.Experimental results demonstrate that the proposed method can reduce the localization error.

Weaknesses

1.The motivation of Multi-Scene Absolute Pose Regression is somewhat insufficient. The paper declares that Multi-Scene Absolute Pose Regression can satisfy the needs of speed and memory efficiency across multiple scenes. However, these are not validated in the experiments, especially compared with single-scene methods. 2.The proposed method seems to only support transformer based APR, which are limited to its further applications. 3.Some references are missing. Although the paper focuses on Multi-Scene Absolute Pose Regression, the references about single-scene camera relocalization methods still should be discussed, including single-scene ARP methods and 2D-3D correspondence based methods. 4.The experimental results are limited, reflected by the following aspects. (1)The paper only lists the localization results in comparison with transformer based Multi-Scene APR methods, but the comparisons with other single-scene state-of-the-art methods are missing, especially the speed and memory efficiency which is declared as advantages of Multi-Scene methods. (2)From Tables 1,2, the improvements of the proposed method seem not obvious, which can not show the method superiority. More discussions are preferred.

Questions

1.Currently, the 2D-3D correspondence based methods (also called coordinate regression based methods) still achieve the state-of-the-art localization performance in both static and dynamic scenes, such DSAC*, KFNet and so on. It is curious that whether the proposed method can apply to 2D-3D correspondence based methods? 2.Does the proposed method affect the network training convergence time?

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors have adequately addressed the limitation.

Summary

The paper analyzes the collapse of self-attention map in Multi Scene Pose Transformer model and proposes two simple but effective methods: auxiliary loss and fixed 2D sinusoidal encoding to solve this problem. The improved method delivers SOTA performance on the Multi Scene Pose Regression task.

Strengths

1. The proposed auxiliary loss seems a good solution for query-key distortion in MS Transformer which is simple and effective. 2. The figures and tables in this paper are exceptionally clear and well-organized, making the paper easy to understand and interpret.

Weaknesses

1. Since query-key distortion is studied in the transformer literature and the fixed 2D sinusoidal positional encoding is an off-the-shelf module, the methods in this paper seems to be lack of novelty. It would therefore be nice to discuss more on why APR task will lead to query-key distortion rather than saying "The model tends to avoid the self-attention mechanism due to the learning difficulty." in line 127-128. The paper reads as if it's discussing improvements to the transformer and not related to APR. 2. Ablation study in Table6 indicates that most of the performance improvements come from the using of fixed sinusoidal positional encoding, a more complete ablation study may be helpful to show the ability of the proposed loss function. 3. Section5: Activating Self-Attention for MS-APR is overly detailed, most of the information is a repetition of MS-Transformer.

Questions

The results in Table5 indicates that the loss function is a very hard constraint that limits the purity of query region to [0.4, 0.6), such constraint is not suitable for all situations. Do authors have comments?

Rating

5

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

The manuscript is limited in scholarship, missing references to more APR methods like [A]. [A] Sc-wls: Towards interpretable feed-forward camera re-localization, ECCV 2022

Summary

This paper is about a improving self attention in the transformer architecture for multi-scene APR. The show that the self attention module in the SOTA transformer model for APR is actually not helping much and offer an potential explanation. The paper claims that the keys and queries end up in different spaces, such that the inner product between keys and queries is very close to zero in most cases, leading the attention to collapse to zero. The paper proposes an additional loss term that encourages the mixing of queries and keys leading to much more overlap. The addition of this term results in a noticeable improvement in APR metrics across indoor and outdoor datasets.

Strengths

+ The motivation for the paper is clear and concise and overall well laid out. Table 1 lists metrics with and without self attention showing the limited utility. It is shown empirically that the keys and queries are isolated form one another as well as theoretically explained why this is an issue. The proposed approach is well explained, straightforward, and has the desired effect of quantitatively improving attention and qualitatively improving APR metrics. This will clearly be used with mulitscene transformer based APR methods going forward because it's simple without requiring any extra data and is effective. + The problem analysis Section 4 is thorough and useful. The side effects of the commonly used techniques are clearly explained and the arguments for why this is a problems is compelling. + There are sufficient implementation details. I could easily reimplement this paper from the information provided. The hyper parameters are shared with [17] so it is unlikely that the performance gain is due to hyperparameter tweaking. + Thorough ablation of different methods for solving the SA problem as described as well as different positional encoding methods. The methods proposed in the paper are validated as being the best on these datasets. + This is one of the few papers I've seen using attention maps in pose regression where I feel like the visualizations and discussion around attention are actually meaningful.

Weaknesses

- Table 5 is difficult to parse. I feel like it could easily be represented with histograms as in Figure 2a. Similarly I feel like the histograms a fairly course. The point comes across okay but I'm not sure why such a course histogram would be used.

Questions

In Table 3, MST and +Ours is incorrectly bold for the position error for the Office scene.

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

3

Limitations

Yes.

Response Uploaded

Thank you for all the valuable feedback. We have uploaded the responses according to the reviewers' requests on the global rebuttal page and individual comments. Today is the only day left for discussion, so please verify our responses, and if we have addressed your concerns satisfactorily, we kindly request a positive reassessment.

BA

I have updated the recommendation to BA. This insight has value in APR, echoing insights from other methods like [A] that attending to useful regions help APR. However, a proper literature discussion is needed to help the APR community converge to useful scientific conclusions. [A] Sc-wls: Towards interpretable feed-forward camera re-localization, ECCV 2022

Response Uploaded

Thank you for taking the time and effort to review our rebuttal. We have uploaded our responses according to the reviewer's request. With only one hour left for discussion, we kindly ask you to verify our responses. Your verification will help us improve the overall quality and clarity of our work. If we have satisfactorily addressed your concerns, we would appreciate a positive reassessment.

Response Uploaded

Thank you for taking the time and effort to review our rebuttal. We have uploaded our responses according to the reviewer's request. With only one hour left for discussion, we kindly ask you to verify our responses. Your verification will help us improve the overall quality and clarity of our work. If we have satisfactorily addressed your concerns, we would appreciate a positive reassessment.