Library

Unsupervised Homography Estimation on Multimodal Image Pair via Alternating Optimization

Estimating the homography between two images is crucial for mid- or high-level vision tasks, such as image stitching and fusion. However, using supervised learning methods is often challenging or costly due to the difficulty of collecting ground-truth data. In response, unsupervised learning approaches have emerged. Most early methods, though, assume that the given image pairs are from the same camera or have minor lighting differences. Consequently, while these methods perform effectively under such conditions, they generally fail when input image pairs come from different domains, referred to as multimodal image pairs. To address these limitations, we propose AltO, an unsupervised learning framework for estimating homography in multimodal image pairs. Our method employs a two-phase alternating optimization framework, similar to Expectation-Maximization (EM), where one phase reduces the geometry gap and the other addresses the modality gap. To handle these gaps, we use Barlow Twins loss for the modality gap and propose an extended version, Geometry Barlow Twins, for the geometry gap. As a result, we demonstrate that our method, AltO, can be trained on multimodal datasets without any ground-truth data. It not only outperforms other unsupervised methods but is also compatible with various architectures of homography estimators. The source code can be found at:~\url{https://github.com/songsang7/AltO}

cs.CVcs.AIComputer Science
View graphSource

Paper

References (38)

01POS-GIFT: A geometric and intensity-invariant feature transformation for multimodal imagesZhuolu Hou, Yuxuan Liu, Li Zhang2023 · Information Fusion · 85 citations
02Recurrent Homography Estimation Using Homography-Guided Image Warping and Focus TransformerSixi Cao, Runmin Zhang, L. Luo et al.2023 · Computer Vision and Pattern Recognition · 55 citations
03Iterative Deep Homography EstimationSixi Cao, Jianxin Hu, Zehua Sheng et al.2022 · Computer Vision and Pattern Recognition · 95 citations In Library
04deepNIR: Datasets for Generating Synthetic NIR Images and Improved Fruit Detection System Using Deep Learning TechniquesInkyu Sa, Jongyoon Lim, H. Ahn et al.2022 · Italian National Conference on Sensors · 34 citations In Library
05VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised LearningAdrien Bardes, J. Ponce, Yann LeCun2021 · International Conference on Learning Representations · 1.4k citations In Library
06Deep Lucas-Kanade Homography for Multimodal Image AlignmentYiming Zhao, Xinming Huang, Ziming Zhang2021 · Computer Vision and Pattern Recognition · 88 citations In Library
07Perceptual Loss for Robust Unsupervised Homography EstimationDaniel Koguciuk, E. Arani, Bahram Zonooz2021 · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · 37 citations In Library
08Barlow Twins: Self-Supervised Learning via Redundancy ReductionJure Zbontar, Li Jing, Ishan Misra et al.2021 · International Conference on Machine Learning · 3.1k citations In Library
09Exploring Simple Siamese Representation LearningXinlei Chen, Kaiming He2020 · Computer Vision and Pattern Recognition · 5.1k citations In Library
10Contrastive Learning for Unpaired Image-to-Image TranslationTaesung Park, Alexei A. Efros, Richard Zhang et al.2020 · European Conference on Computer Vision · 1.7k citations In Library
11An Integrated Approach to Registration and Fusion of Hyperspectral and Multispectral ImagesYuan Zhou, A. Rangarajan, P. Gader2020 · IEEE Transactions on Geoscience and Remote Sensing · 66 citations
12RAFT: Recurrent All-Pairs Field Transforms for Optical FlowZachary Teed, Jia Deng2020 · European Conference on Computer Vision · 4.0k citations In Library

Scroll for more · 26 remaining

Similar papers

Content-Aware Unsupervised Deep Homography EstimationJirong Zhang, Chuan Wang, Shuaicheng Liu et al.2019 · European Conference on Computer Vision · 197 citations In Library
UMHE: Unsupervised Multispectral Homography EstimationJeongmin Shin, Jiwon Kim, S. M. Kwon et al.2024 · IEEE Sensors Journal In Library
Content-Aware Unsupervised Deep Homography Estimation and Its ExtensionsShuaicheng Liu, Nianjin Ye, Chuan Wang et al.2022 · IEEE Transactions on Pattern Analysis and Machine Intelligence · 48 citations In Library
Uncertainty Guided Deep Lucas-Kanade Homography for Multimodal Image AlignmentZhen Zhou, Jianqiao Luo, Qing Zhu et al.2024 · IEEE Transactions on Geoscience and Remote Sensing In Library
Unsupervised Deep Learning Model for Homography EstimationLang Zhou2024 · International Journal of Computer Science and Information Technology In Library
Unsupervised deep homography with multi‐scale global attentionWei Hu, Chu He, Mingyuan Lin et al.2023 · IET Image Processing In Library
Perceptual Loss for Robust Unsupervised Homography EstimationDaniel Koguciuk, Elahe Arani, Bahram Zonooz2021 In Library
Perceptual Loss for Robust Unsupervised Homography EstimationDaniel Koguciuk, E. Arani, Bahram Zonooz2021 · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · 37 citations In Library
SSHNet: Unsupervised Cross-modal Homography Estimation via Problem Reformulation and Split OptimizationJunchen Yu, Si-Yuan Cao, Runmin Zhang et al.2025 In Library
SSHNet: Unsupervised Cross-modal Homography Estimation via Problem Reformulation and Split OptimizationJunchen Yu, Sixi Cao, Runmin Zhang et al.2024 · Computer Vision and Pattern Recognition · 5 citations In Library

Peer review

Accept (poster)NeurIPS 2024 · via OpenReview
Reviewer EsVS3/10 · confidence 5/52024-06-26

Summary

The paper proposes an unsupervised homography estimation method for multimodal image pairs using an alternating optimization approach. The claimed key innovation is the introduction of the Geometry Barlow Twins loss function for the alternating optimization. The authors show that their approach works on 3 multimodal datasets and different homography estimation architecutres.

Strengths

The alternating optimization framework together with Geometry Barlow Twins loss seem to be a fresh perspective on unsupervised multimodal homography estimation.

Weaknesses

Weaknesses 1. Discussion on the Feasibility and Rationality of the Proposed Method: First, for unsupervised training of networks based on iterative prediction, such as RAFT, to ensure stability during training, related methods [1-2] typically apply some form of direct supervision to the motion predicted by the network. This is different from the approach proposed in this paper, which only uses the Geometry Barlow Twins loss for brightness supervision. Second, how RAFT can be used for homography estimation should also be explained, because it is designed for optical flow estimation. Moreover, the paper does not explain how the proposed Geometry Barlow Twins loss supervises the intermediate stages of iterative prediction, whereas RAFT, IHN, and RHWF, along with methods leveraging their structures [1-2], generally provide details on their supervision mechanisms on the intermediate stages. This raises concerns about the feasibility of the proposed supervision method in this paper. Additionally, the effectiveness of the Modality-Agnostic Representation Learning (MARL) introduced in section 4.3 is questionable because it lacks spatial information in its supervision. As mentioned in section 3.2, the projector removes spatial information from the feature maps. The authors should provide a convincing and thorough explanation for these issues. 2. Doubt about the Effectiveness of the Proposed Method: For example, the paper proposes the alternating optimization (AltO) method but does not provide sufficient experimental results to demonstrate its superiority over other strategies, such as directly cascading all the modules. Furthermore, the paper lacks a comparative demonstration of the features extracted with and without the MARL phase, making the advantages of introducing this phase less convincing. 3. Insufficient Experimental Validation: The paper conducts experiments on only 3 cross-modal datasets, among which only the GoogleMap dataset exhibits significant modality differences. The GoogleEarth dataset mainly consists of images taken in different seasons [3]. Part of the DeepIR dataset is simulated multispectral data [4], which will significantly reduce the difficulty of homography estimation. It would be beneficial to conduct experiments on more challenging multimodal datasets, such as those involving VIS-SAR modalities. [1] Stone, A., Maurer, D., Ayvaci, A., Angelova, A., & Jonschkowski, R. (2021). Smurf: Self-teaching multi-frame unsupervised raft with full-image warping. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition (pp. 3887-3896). [2] Liang, Y., Liu, J., Zhang, D., & Fu, Y. (2023). Mpi-flow: Learning realistic optical flow with multiplane images. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 13857-13868). [3] Zhao, Y., Huang, X., & Zhang, Z. (2021). Deep lucas-kanade homography for multimodal image alignment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 15950-15959). [4] Sa, I., Lim, J. Y., Ahn, H. S., & MacDonald, B. (2022). deepNIR: Datasets for generating synthetic NIR images and improved fruit detection system using deep learning techniques. Sensors, 22(13), 4721.

Questions

Please refer to the Weaknesses.

Rating

3

Confidence

5

Soundness

2

Presentation

2

Contribution

2

Limitations

The paper discusses the additional training cost arising from the inclusion of an additional module in the two-phase network, and explores potential solutions for addressing this issue in future research. However, the method’s generalization capabilities are not thoroughly explored, with experimental datasets limited to satellite images, maps, RGB, and NIR images. Future research could involve testing the method on a broader range of datasets to validate its generalization capabilities.

Authorsrebuttal2024-08-12

Looking forward to your post-rebuttal comment!

Dear Reviewer EsVS Thank you once again for participating in the review process and for providing such thoughtful feedback. We wanted to kindly follow up regarding the rebuttal we submitted. We understand that this is a busy time, and we greatly appreciate your efforts. To summarize our rebuttal to your review: * Provided detailed answers to all the questions you raised. * Conducted a new experiment to demonstrate the effectiveness of MARL and the global average pooling (GAP) layer. * Conducted a new experiment to demonstrate the effectiveness of alternating training. * Conducted new experiments on additional datasets to demonstrate the generalization capability of our AltO framework. If there are any further questions or clarifications needed from our side, we would be more than happy to provide them. We look forward to any feedback you might have and are eager to engage in any further discussion to improve our work. Thank you once again for your time and consideration. Best regards, Authors

Reviewer EsVS2024-08-12

Thanks to the authors for the rebuttal. I have read the rebuttal and other reviewers' comments. However, I still have the following concerns: (1) In Weakness 1, my concern is mainly about why the GBT loss can enable the homography network to converge well without direct supervision of global/local motion. In my view, the GBT loss can be seen as the 'cross-modal image intensity similarity' mentioned in SCPNet, which may lead to non-convergence. Is there any theoretical basis or proof to support this? (2) Figure R2 further raises my concern about the experimental results. The visualized feature maps are not similar at all. They barely retain the structural information of the original images and contain many artifacts. I am not convinced that such feature maps can effectively supervise the registration network. What's more, in Table 2 in the paper, the author claims that simply using MSE on such feature maps (the mean of squared error of the intensity of the feature maps in Figure R2) as the Geometry loss can produce a relatively accurate result, which also reduces the credibility of the experiments. (3) Insufficient references and comparison experiments. As mentioned by reviewer sFim, this paper did not discuss many references. Moreover, comparison experiments with other methods should be conducted on all datasets to demonstrate the effectiveness of AltO, instead of only on Google Maps. (4) The homography estimation accuracy on the OS Dataset is unsatisfactory with the MACE of 14.12, which may not be regarded as a converged training. The effectiveness of the proposed method is insufficient on such cross-modal dataset. For the reasons mentioned above, I am still inclined to reject.

Authorsrebuttal2024-08-13

Thank you for your continued attention to our work. We would like to address the remaining concerns and the additional questions you have raised. (1) Due to the encoder’s mapping, the GBT loss is calculated under uni-modal conditions in the feature space, not cross-modal conditions in the image space. Although the feature spaces might not perfectly match, a few corresponding regions(or 4-corresponding points) are sufficient to estimate the homography. This is evident in the feature map in Figure R2, where a few corresponding regions are strongly activated. Additionally, experiments (Table R1) show that GBT alone, without MARL, fails because the encoder doesn't learn the mapping without MARL. (2) The purpose of the geometry loss is not to make the features identical. Even if the given pair of feature maps differ somewhat, the goal is to estimate the homography by matching the most strongly activated corresponding points. The encoder plays a crucial role in this process, ensuring that these key points are strongly activated in both input images, which makes the geometry loss convex at these points. Without the encoder, achieving convexity would be difficult when the modalities differ. Since the feature maps are not perfectly identical, as seen in Figure R2, the lower bound (bias) of the geometry loss may be higher. However, if at least 4-corresponding points are well-matched, the homography can still be accurately estimated. The same applies when using MSE: as long as a few strongly activated local regions are well matched, the homography can be accurately estimated even if the two feature maps are not identical. Furthermore, while MSE achieves its minimum when the two feature maps are exactly the same, the GBT loss, being based on normalized similarity, can reach its minimum as long as the trends are similar. This property makes the GBT loss more suitable for this scenario, as shown in the ablation study of our paper. (3) Conducting experiments for all possible cases requires significant time, so we focused on the most meaningful ones. As you mentioned, the Google Map dataset has a significant modality gap, which is why we prioritized it. However, we agree that additional experiments would strengthen our paper. (4) Since the OS dataset is completely new to us, the existing hyperparameters may not have been suitable. Additionally, our proposed method is a learning framework, so improvements could be made through architecture exploration of the encoder and projector, but this has not been done yet for the OS dataset. Therefore, the absolute performance may appear lower than on other datasets. Despite these challenging conditions, our method shows better performance than other baselines, further demonstrating its tendency to converge and its overall feasibility. We apologize for the lack of detailed explanation regarding point (2) in our paper, and we hope this clarification is helpful. If you have any further questions or concerns, please feel free to ask. We would be more than happy to provide a thorough response to any additional inquiries.

Reviewer EsVS2024-08-14

Thank you to the authors for their response. However, it seems that some of my questions were misunderstood, and the reply does not fully address my concerns. Additionally, some statements are not well-supported and contain significant conceptual errors. Below is a point-by-point discussion: (1) The original issue (1), corresponding to Weakness 1.1 in the review and comment (1), questions how training can converge effectively under large homography deformations (e.g., [-32, 32] pixel displacement of the four corner points of a 128x128 image) when only using supervision based on content similarity, such as GBT, especially for iterative networks. Previous unsupervised works, such as Smurf [1] and MPIFlow [2], attempt to directly supervise the motion field by simulating ground truth for training iterative networks, rather than merely supervising on content similarity. Moreover, based on my understanding, the proposed GBT loss and the “cross-modal intensity-based loss” in SCPNet [3] both measure the content similarity of input images. According to experiments in SCPNet, training could not converge only under the supervision of the cross-modal intensity-based loss. Is there any theoretical basis or proof to support why the proposed GBT loss, which also measures content similarity, can enable unsupervised training to converge under such large homography deformation? Furthermore, the authors' explanation for this issue is conceptually incorrect. The authors state, "Although the feature spaces might not perfectly match, a few corresponding regions (or 4-corresponding points) are sufficient to estimate the homography". While it is possible to estimate homography with only a few matching points using manually designed algorithms like RANSAC [4], due to strong prior knowledge, the problem under discussion is the unsupervised training of homography estimation networks based on content similarity, not the design of manual homography estimation methods. The network training method proposed by the authors in the paper lacks prior knowledge like RANSAC. At the beginning of training, it can only infer the homography based on content similarity in the feature maps, making the requirement for consistency in corresponding regions significantly higher. Finally, the authors also acknowledge in Figure R2 that only "a few corresponding regions are strongly activated," and the feature map consistency shown in Figure R2 is poor. Such feature maps are unconvincing and do not guarantee the success of the proposed network training methods. Even when using manually designed homography estimation methods like RANSAC, a small number of corresponding regions could lead to poor estimation results. When corresponding regions are concentrated in a small part of the image, even a slight matching error can be magnified in the overall homography estimation. I am concerned about the authors' understanding of the fundamental concepts in this field. (2) The authors seem to misunderstand my original question. I fully acknowledge that “The purpose of the geometry loss is not to make the features identical,” and my original question did not suggest that the geometry loss needs to do so. As the authors stated in the paper (Lines 173-174), “The Modality loss aims to constrain z_A and z_B to share a similar representational space,” meaning that the consistency of the feature maps in Figure R2 is expected to be ensured by MARL. My original question was whether the Modality-Agnostic Representation Learning (MARL) process, which uses global average pooling (GAP) and thus does not supervise spatial information in the loss, might lead to poor local consistency of the constrained feature maps. In fact, Figure R2 seems to confirm this concern, as the feature map consistency is poor, with even noticeable artifacts. Additionally, I question how the proposed geometry loss can still produce good homography estimation results on such low-quality feature maps. Even more puzzling, Table 2 of the paper shows that replacing the geometry loss with MSE still achieves relatively good results. It raises the question: why wouldn’t the stripe artifacts in Figure R2 mislead homography estimation? The authors' explanation, however, does not address the core of this issue. (3) I understand that conducting more experiments may not be easy during the rebuttal period. However, since the OS dataset is more multimodal compared to the Google Map dataset, conducting additional experiments in the future is indeed necessary. (4) I agree with the authors’ explanation for the performance on the OS dataset. [1] Smurf: Self-teaching multi-frame unsupervised raft with full-image warping. In CVPR, 2021. [2] Mpi-flow: Learning realistic optical flow with multiplane images. In ICCV, 2023. [3] SCPNet: unsupervised cross-modal homography estimation via intra-modal self-supervised learning. In ECCV, 2024. [4] http://6.869.csail.mit.edu/fa12/lectures/lecture13ransac/lecture13ransac.pdf

Authorsrebuttal2024-08-14

(1) We apologize for the misunderstanding. It seems we may have overcomplicated your question. To clarify, let’s first look at UDHN [1], the pioneering unsupervised method in a same-modality scenario, which might address most of your concerns. UDHN predicts four corresponding points, then converts them into a homography using Direct Linear Transformation (DLT). The source image is then warped using the predicted homography, and a reconstruction loss or intensity-based similarity is calculated with the target image. This entire process, including DLT and warping, is differentiable and converges very well in a same-modality scenario. A similar approach is taken by biHomE [2], and both UDHN and biHomE are incorporated into our proposed method. Our method targets multimodal datasets. Therefore, we internally convert them to a uni-modal condition to apply an approach similar to UDHN or biHomE. To facilitate this multimodal-to-uni-modal conversion, we introduced MARL and alternating training. Thus, the GBT loss is calculated under uni-modal conditions thanks to the encoder's mapping. In the aforementioned processes, although methods like RANSAC are not used, the approaches work well. It seems you might be considering feature-based frameworks like SIFT, SURF, ORB or methods like LIFT [3] and Super Point [4], which replace parts of this process with deep learning. However, our method falls within the category of end-to-end homography estimation methods, like DHN [5], UDHN, and biHomE. These methods are composed entirely of differentiable processes for end-to-end learning. Rest assured, we are fully aware of the fundamentals of feature-based frameworks and understand practical know-how, such as: * If the corresponding points are clustered in one region or many lie on a straight line, the error in homography estimation increases significantly. * Conversely, the homography estimation becomes more accurate when the corresponding points are widely distributed across the entire image. * If only a few corresponding points are known, even with methods like RANSAC, the homography estimation error can increase due to outliers. Finally, regarding Figure R2, please consider that this feature map is simply an average across 128 channels. Since our GBT is based on the Pearson correlation coefficient, the strength level (bias) difference between the two feature maps is not important; only the similarity in trends between the two feature maps reduces the loss. Additionally, both MSE and GBT, it can allow a few strongly activated regions (not just single points, but multiple points) to accurately infer the overall homography. This can be empirically observed by looking at the regression values computed by the registration network. Again, unlike explicit methods like RANSAC, the registration network implicitly calculates these values through learning. [1] Ty Nguyen et al., Unsupervised deep homography: A fast and robust homography estimation model. IEEE Robotics Autom. Lett., 2018. [2] Daniel Koguciuk et al., Perceptual loss for robust unsupervised homography estimation. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2021. [3] YI, Kwang Moo, et al. Lift: Learned invariant feature transform. In: Computer Vision–ECCV 2016 [4] DETONE, Daniel et al., Superpoint: Self-supervised interest point detection and description. In: CVPRW 2018 [5] Daniel DeTone et al., Deep image homography estimation. CoRR, abs/1606.03798, 2016. (2) We now understand that your main concern is whether the encoder's output can maintain geometric consistency. We agree that the lack of an explicit mechanism could be a potential weakness, as you mentioned. However, as shown in Table R2, removing the global average pooling layer is not an option. We think that further improvements may require additional processing directly on the encoder’s output to address this issue. Additionally, regarding the discussion on MSE, please refer to the last paragraph of (1). (3) We agree that the OS dataset is more multimodal than the Google Map dataset, so further research will need to focus on this dataset. Thank you for your detailed and ongoing discussion.

Reviewer EsVS2024-08-14

The reasons provided by the authors are not convincing and have not addressed my concerns. Therefore, I maintain my rating. (1) The works cited by the authors, namely UDHN and biHomE, **neither have successful precedents of training on cross-modal data nor iterative networks like IHN, RHWF, and RAFT**, making their explanation unpersuasive. (2) The authors mentioned that their feature maps are averaged across 128 dimensions. First, I reviewed the original paper, and this detailed information was never mentioned. Second, to my understanding, averaging should weaken artifacts, contrary to what is shown in Figure R2, where there are many artifacts with poor consistency. (3) Furthermore, I have never requested a comparison with RANSAC-type algorithms. RANSAC was mentioned because the authors explained the reason for successful unsupervised network training by stating, "Although the feature spaces might not perfectly match, a few corresponding regions (or 4-corresponding points) are sufficient to estimate the homography." In my view, this explanation may have a few possibilities that make sense for RANSAC-type algorithms, **but it is highly unreasonable and unfeasible for the cross-modal unsupervised homography training adopted by the authors**, and they have not clarified this issue.

Authorsrebuttal2024-08-14

We regret that our explanation did not fully satisfy you, but we appreciate your engagement in the discussion until the end. (1) As mentioned, UDHN and biHomE assume a uni-modal scenario. **Our main contribution lies in expanding this to a multimodal application by introducing MARL and alternating training.** Regarding the iterative framework, we retained the original structure of RAFT, IHN, and RHWF, simply replacing the ground truth with AltO. This is why we did not emphasize it in the paper, but we demonstrated that it can be applied without significant issues through the main experiments. (2) Evaluations of the visualized feature maps are subjective, so we believe further discussion on this might not be productive. (3) In our previous comment, we mentioned that "it can allow a few strongly activated regions (not just single points, but multiple points) to accurately infer the overall homography." In other words, even with only a few distinctive regions in a solid image, the registration network can capture these and perform homography estimation relatively accurately.

Reviewer sFim7/10 · confidence 4/52024-07-03

Summary

This paper proposes a new unsupervised homography estimation approach for multimodal images. This method is designed as a two-phase optimization framework named AltO. The first phase named "Geometry Learning" trains a registration network to align the input multimodal images geometrically. The second phase named "Modality-Agnostic Representation Learning" trains an encoder and a projector to extract the image-level features invariant to modality changes. Experimental results demonstrate that AltO outperforms several existing unsupervised approaches on the multimodal registration datasets.

Strengths

1. The proposed framework is intuitive and interesting. This framework trains a registration network to align the input multimodal images geometrically, and trains another encoder to match the image-level features of the warped multimodal images. This framework has the potential to capture the pixel-level and image-level information in an unsupervised manner. 2. The organization and presentation of this paper are good. I think I can understand the core idea of this paper.

Weaknesses

**1. Some central claims of this paper lack experimental evidence.** 1.1 The "alternating" optimization framework is a central design in this paper. However, why is "alternating" optimization necessary? Will optimizing the "geometry loss" and "modality loss" simultaneously hurt performance? 1.2 The superiority of the proposed Geometry Barlow Twins (GBT) loss was not verified. The original Barlow Twins loss can be straightforwardly applied to the proposed model by considering both the spatial axis (indexed with "h,w") and batch axis (indexed with "n") as the batch dimension. This straightforward implementation should be compared with the proposed GBT loss. 1.3 The proposed approaches should be compared with some recent unsupervised approaches. Here are some approaches with released codes. [1] Unsupervised global and local homography estimation with motion basis learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. [2] A Multiscale Framework with Unsupervised Learning for Remote Sensing Image Registration, IEEE Transactions on Geoscience and Remote Sensing, 2022. **2. This paper did not discuss the recent hand-crafted approaches for multimodal image registration.** Many recent hand-crafted methods have been published in the top journals, so this kind of approach should not be ignored. The experiment should also compare the proposed approaches with the recent hand-crafted approaches. Here are some hand-crafted approaches with released code. [3] Histogram of the orientation of the weighted phase descriptor for multi-modal remote sensing image matching. ISPRS Journal of Photogrammetry and Remote Sensing, 2023. [4] POS-GIFT: A geometric and intensity-invariant feature transformation for multimodal images. Information Fusion, 2024. **3. The discussion of the motivation is not sufficient.** The Introduction section mentioned some typical unsupervised approaches designed for the images from the same modality (e.g., UDHN and biHomE). However, the unsupervised approaches [2,5] designed for multimodal image registration are not discussed. What is the motivation of the proposed method compared with this kind of approach? [5] A Novel Coarse-to-Fine Deep Learning Registration Framework for Multi-Modal Remote Sensing Images. IEEE Transactions on Geoscience and Remote Sensing, 2023. **4. This paper misses some references to hand-crafted and unsupervised approaches.** I have listed some of them in the above weaknesses. The authors should further survey more papers and carefully revise the "Related Work" section.

Questions

Please provide more discussions and experimental results to address the above weaknesses. Moreover, is the 3D reconstruction task related to "Homography Estimation" (line 21)? Generally, 3D reconstruction focuses on non-planar scenes, while homography estimation is designed for the planar scenes. Is there some literature that mentions the relationship between 3D reconstruction and homography estimation?

Rating

7

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitations.

Authorsrebuttal2024-08-12

Looking forward to your post-rebuttal comment!

Dear Reviewer sFim Thank you once again for participating in the review process and for providing such thoughtful feedback. We wanted to kindly follow up regarding the rebuttal we submitted. We understand that this is a busy time, and we greatly appreciate your efforts. To summarize our rebuttal to your review: * Conducted a new experiment to demonstrate the effectiveness of alternating training and MARL. * Conducted a new experiment and presented a case to compare GBT with the new method you suggested. * Added more baselines for performance evaluation by conducting a new experiment and incorporating reported values. * Provided detailed answers to all the questions you raised. If there are any further questions or clarifications needed from our side, we would be more than happy to provide them. We look forward to any feedback you might have and are eager to engage in any further discussion to improve our work. Thank you once again for your time and consideration. Best regards, Authors

Authorsrebuttal2024-08-12

Thank you for increasing your score!

Thank you very much for raising the rating of our paper. We appreciate your recognition of the additional experiments and discussions we provided, and we are glad that they addressed your main concerns. We would like to provide further clarification: 1. Intuitive Explanation for End-to-End Training without Alternating: If the encoder and registration network collapse as we mentioned, then all reconstruction-based losses, including L1-loss or our GBT loss, would become zero, the minimum possible value for such losses. This occurs because the encoder's outputs are constant, resulting in the difference or similarity always converging to a trivial value, regardless of the homography. Consequently, the registration network would not make any effort to find the true homography, outputs only identity matrix. 2. Difference Between Our Approach and Literature [2]: As you mentioned, extending from 6-DOF to 8-DOF is indeed straightforward, but it is a more challenging task, and some performance degradation is to be expected. The CFOG, used in [2], seems to be a key element. The multi-scale framework is aimed at more precise estimation, but it does not address different modalities. Additionally, since the code is not fully available at the moment, quickly reproducing and modifying it is difficult. However, we fully agree that the extended version of [2] would certainly be a valuable baseline to consider. We hope these clarifications address your questions, and we are more than happy to discuss further if needed. Thank you once again for your valuable insights and support.

Reviewer PrYW6/10 · confidence 5/52024-07-12

Summary

The paper addresses unsupervised homography estimation from multi-modal image pairs. The authors propose to cope with the issue of 1) modality, 2) registration in two distinct networks that are trained in an interleaved fashion. The networks architecture derives from the Barlow Twins framework, with changes in the loss function. Results are illustrated on several public benchmark of small images (128^2) and compares favorably wrt to related unsupervised approach.

Strengths

1- I enjoy reading the paper. I walked through the paper, first with curiosity and skepticism, then with strong interest. The approach is intuitive (adjust the two representations then compute the transformation) and compelling. I am somehow surprised that it works :) The constrastive-like loss used in Barlow Twins is contributing much for the network to learn the correct solution. 2- Overall, the authors are tackling an important problem (unsupervised learning) for which an original solution is proposed --while based on previous recent work. The methodology is clearly presented. Results are convincing (thought only on small images 128x128) and illustrated on various modality pairs. Quantitative results show improvement wrt related unsupervised work

Weaknesses

1- Not a weakness, but a points which could have been discussed: why not simply transforming the inputs into edge maps before learning a matching/homography function (and putting aside the modality discrepancy). It would not be a very fancy approach, but I believe it could be a baseline for comparison. 2- The approach would be more convincing if each of the two modules (GL and MARL) had demonstrated their effectiveness also individually (ie same image pair modality using only GL).

Questions

- What is the size of the embedding? What is the training time? Are the Barlow Twins trained from scratch? - Illustration seems to show strong geometric features (ie lines) in the input images. Is it a strong limitation of the approach?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

From a practical point of view, the size of the image and the strong overlap between the pairs show that the work need to be further developped for applications at full scale. From a methodology point of view, the authors have discussed the limitation of having two networks trained in an interleaved way, with potential collision or collapse.

Authorsrebuttal2024-08-12

Looking forward to your post-rebuttal comment!

Dear Reviewer PrYW Thank you once again for participating in the review process and for providing such thoughtful feedback. We wanted to kindly follow up regarding the rebuttal we submitted. We understand that this is a busy time, and we greatly appreciate your efforts. To summarize our rebuttal to your review: * Conducted a new experiment to demonstrate the performance of the edge-based method in multimodal case. * Conducted a new experiment to show the effectiveness of alternating training and MARL. * Provided detailed answers to all the questions you raised. * Conducted a new experiment to demonstrate that our method, AltO, is also effective for larger image set. If there are any further questions or clarifications needed from our side, we would be more than happy to provide them. We look forward to any feedback you might have and are eager to engage in any further discussion to improve our work. Thank you once again for your time and consideration. Best regards, Authors

Authorsrebuttal2024-08-13

Thanks to Reviewer PrYW

We would like to thank you again for highly appreciating the strengths of our work. If you have any further questions or concerns, please feel free to ask. We would be more than happy to provide a thorough response to any additional inquiries. Best regards, Authors

Reviewer PrYW2024-08-13

Thank you to the authors for the responses and clarifications, and overall rebuttal. - Comments regarding the use of edge maps make sense, I appreciate the additional experimental results which support the observations of the authors. - My question was about confirming the viability of each of the two networks (MARL and GL) by testing then independently: experiments that would be using same modality inputs (with some deformation) to evaluate GL independently of MARL. (I do not see how MARL could be tested independently of GL); Overall, the authors mainly addressed my concerned. Additional experiments have consolidated the work. I keep my score as accept.

Authorsrebuttal2024-08-14

Thank you for your continued feedback. We noticed that there may have been a misunderstanding regarding one of the points you raised. We would like to clarify this and provide a more accurate response. We have conducted the following additional experiments using the MS-COCO dataset, which has the exact same modality. Due to the limited time, we used the simple DHN model as the registration network. In the case where only GL is used, the encoder was frozen from scratch. | Method | MACE | |------------|:--------:| | (No warping ) | 24.89 | | DHN + GL | 5.75 | | DHN + MARL | 25.00 | In the case where only MARL is used, the spatial information is lost due to the global average pooling layer, leading to ineffective learning. We hope this response addresses your concerns. If you have any further questions, please feel free to ask. We will do our best to respond promptly. Thank you.

Reviewer sFim2024-08-12

Thank the authors for the responses. The additional experimental results and discussions address my main concerns. In my opinion, the ablation study about the alternating optimization and the comparison with the recent approaches make the proposed approach's superiorities more convincing. Therefore, I’d like to raise my rating from "Weak Accept" to "Accept". It would be better to discuss the following two problems further. 1. In the response to W1.1, the authors pointed out that "Training the encoder and registration network together end-to-end without alternating can cause the encoder to output a constant value and the registration network to always output the identity matrix as the homography." It would be better to provide some intuitive explanations. Why does the end-to-end training make the encoder/registration network tend to output a constant value/identity matrix? 2. In the response to W3, the authors claimed that this paper’s motivation is different from the literature [2] because the proposed approach considers the 8-DOF homography estimation while the method [2] focuses on a 6-DOF problem. However, such a difference seems to be minor because both the two methods utilize learnable regression models. It is straightforward to extend the method [2] to handle 8-DOF homography estimation. More intrinsic differences should be discussed to highlight the motivation of this paper.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC