G3: An Effective and Adaptive Framework for Worldwide Geolocalization Using Large Multi-Modality Models

Summary

In the paper, authors propose a novel framework, G3, for worldwide geolocalization of a given photograph anywhere on Earth. The authors address the challenges of capturing location-specific visual cues and handling variations in image data distribution across the globe. G3 utilizes a three-step process: Geo-alignment, which learns location-aware image representations, Geo-diversification, which employs multiple retrieval-augmented prompts for robust location prediction, and Geo-verification, which combines retrieved and generated location data for final prediction. The authors also introduce the MP16-Pro dataset to support location-aware visual representation learning. Experiments on the IM2GPS3k and YFCC4K datasets demonstrate the superiority of G3 over existing methods.

Strengths

* All the modules in the G3 framework: Geo Alignment, Geo Diversification and Geo Verification seem logical and rational. Three kinds of embedding coming from the vision encoder are used for retrieval. LLM is used to generate a set of plausible coordinates by providing positive and negative examples. * The method achieves superior performance over several baselines at various levels of granularity on IM2GPS3k and YFCC4K. * Overall, the method is interesting and novel, the writing and flow of the paper is meaningful.

Weaknesses

* The only limitation discussed is regarding the efficiency of inference. However, there is no mention of how much compute time and memory (in number) is required to geo-localize a given input image. * There are no concrete qualitative example of failures reported in the paper. Can the system be fooled easily? For example, if an image from Italy contains a human with a flag of The Netherlands, is the system capable of correctly geolocalizing the image? How does the RAG system along with the LLM perform in such a case? * Limited evaluation considering the state-of-the-art. No mention of recent works such as Pigeon [1] or a GeoReasoner [2]. * Why did the authors choose to use CLIP vision encoder for extracting image features? Recent works have shown that purely image-based pretrained models such as DINO-v2 are better feature extractors than CLIP. No ablation study done for the choice of the vision encoder. * Overall, from discussion in **L316-L333** and Figure 4, it looks like the number of references provided to LLM highly depends and varies based on the image content. The performance is highly sensitive to this hyperparameter and a single value cannot guarantee optimal performance. This can make the framework highly unreliable for practical use cases. [1] Haas, Lukas, Michal Skreta, Silas Alberti, and Chelsea Finn. "Pigeon: Predicting image geolocations." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12893-12902. 2024 [2] Li, Ling, Yu Ye, Bingchuan Jiang, and Wei Zeng. "GeoReasoner: Geo-localization with Reasoning in Street Views using a Large Vision-Language Model." In Forty-first International Conference on Machine Learning.

Questions

* There is only a marginal improvement in performance when including the geo-diversification step considering it is potentially the most expensive step during the inference. * Currently, the text associated with each coordinate only includes the country and city labels. Will the performance of the framework improve by including fine-grained details such as region and/or street name?

Rating

6

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

Limitations are included but failure cases are missing.

Summary

This paper introduces G3, a RAG framework for geo-localization. By introducing a three-step process, the G3 framework achieves superior performance against other SoTA methods. To improve the expressiveness of the image embeddings, the paper proposed a new dataset, MP16-Pro, which adds textual descriptions to the existing MP-16 dataset. By comprehensive experiments, the paper demonstrates the necessity and merits of the G3 framework.

Strengths

1. The proposed method, G3, achieved competitive performance in geo-localization against existing classification-based, retrieval-based and RAG-based methods. 2. Compared to the original MP16 dataset, the proposed MP16-Pro dataset additionally provides textual geolocation-related data, which could be beneficial to the community if fully open-sourced. 3. Comprehensive experiments and explanations are provided to prove the effectiveness and necessity of each G3 component. 4. The authors have open-sourced their project in a clear and instructive manner, which is very positive for reproduction.

Weaknesses

1. The necessity of the Geo-alignment module in the G3 framework has been indicated by experiments results in Table 2. However, the authors did not address the motivation for their particular design choice of the alignment module. Why do the image features have to align with both text features and gps features? Does aligning with simply one modality work just as well? The authors should clarify that by conducting the corresponding ablation study. 2. One of the main contributions claimed by the authors is the introduction of the MP16-Pro dataset. However, the description of the construction process for the MP16-Pro dataset is not sufficiently detailed. 3. The G3 framework was not the first work to incorporate RAG into geolocalization, nor was it the first work to use retrieval-based models. While the proposed method achieved superior performance, it lacks certain novelty to the field. 4. In the experiment setup in section 5(line 226), no retrieved coordinate is considered when evaluating G3 on IM2GPS3K, which is inconsistent to figure 2 of the paper. If no retrieved coordinate works better in certain cases, I wonder about the necessity and applicability of such design.

Questions

1. Since the proposed dataset, MP16, is relatively large, I wonder if the authors have run decomtamination procedures to ensure there is no overlapping data between training and evaluation. 2. The authors of Img2Loc provided experiment results with other LMMs (LLaVA), I wonder what is G3’s performance when switching the LMM to LLaVA compared to Img2Loc. I think by incorporating this result the authors can more robustly state their superiority over existing methods. 3. As shown in Figure 2, the text descriptions of the location in MP16-Pro are not used during inference, I wonder if adding text descriptions to the prompt would work better.

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors of this paper have addressed the limitation of the G3 framework by pointing out its high computation cost. The introduction of alignment and diversification brings on more computation cost and latency compare to existing methods, which limits the retrieval and inference speed.

Summary

This paper proposes three steps, i.e., geo-alignment, geo-diversification, and geo-verification to optimize both retrieval and generation phases of word-wide geo-localization.

Strengths

1. The motivation is clearly stated. 2. The experimental results show the effectiveness of the proposed method. 3. The proposed method achieves state-of-the-art performance. 4. Code is publicly available.

Weaknesses

The experiments are not insufficient. The model seems too large, so the author should provide the number of parameters and gflops experiments.

Questions

1. What is the purpose of image vectorization? Please provide a detailed explanation by the author. 2.Figure 6 is difficult to understand, what does the author want to express? What does it mean that the number of references has a significant impact on the model? 3. The model seems too large, the author should provide the number of parameters and gflops experiments. 4. The author proposed a new dataset MP16 Pro, but it seems that the results have not been published on this dataset. 5. The author obtain textual descriptions by geographical reverse encoding during the Database Construction. Are the descriptions generated for longitudes and latitudes with similar geographical locations consistent? Is the text description useful for images taken at similar locations? The author can add descriptive text experiments to the ablation experiment to prove that the text is helpful for feature representation. 6. The experiment of paper lacks the results of baseline 7. the author does not introduce enough details about the branch of Geo-diversification Module. 8. Fig.5/8 lacks a comparison with the visualization results of baseline.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors discuss the limitation in the paper, but not in enough depth.

Summary

This work focuses on the task of "worldwide geolocalization" with an effective and adaptive framework based on large multi-Modality models. A novel framework, i.e., G3, is proposed, including Geo-alignment, Geo-diversification, and Geo-verification. This work also releases a new dataset MP16-Pro. The experiment results show that G3 has superior performance on two well-established datasets IM2GPS3k and YFCC4K.

Strengths

1. The task of "worldwide geolocalization" is very important and quite interesting. 2. This paper is easy to follow, it is well-written. 3. The experiment results are solid. The G3 model achieve better results than GeoCLIP and Img2Loc on IM2GPS3k and YFCC4K.

Weaknesses

1. In the Geo-diversification part, there is no ablation study on different LMMs and different RAG templates. I wonder whether it still works on open-source LLMs. 2. Minor: a. Figure 1 is often set as a teaser figure, which could show the basic design of the whole work. It would be better to indicate the solution, instead of only showing the limitations. b. Table 1 needs citations for each previous work. And GeoCLIP should be noted as NeurIPS 2023 instead of arXiv.

Questions

1. Please address my above concerns on Weaknesses. 2. The qualitative results show that most of the retrieved images are photos for tourists. What about using real world images from official company (e.g., Google Map)? 3. Please briefly describe how does the MP16-Pro Dataset help (or improve) the G3 model.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

None

Summary

This paper proposes a RAG-based framework for worldwide geo-localization. The first Geo-alignment stage projects input images to embedding spaces to align with GPS coordinates, and text description with contrastive learning. Given new input image, the system is able to retrieve similar GPS and text description. Then the retrieved candidates GPS and text prompts are fed to GPT4V with a pre-defined prompt template to generate GPS coordination. The final stage conducts a similarity-based verification based on multi-modal representations. The method is evaluated on two worldwide geo-localization datasets, i.e., IM2GPS3k and YFCC4K, with state-of-the-art performance.

Strengths

+ The RAG-design for geo-localization is interesting and promising. + The writing is easy to follow. + The performance is much better than previous methods. + Ablation result is provided for the three stages. + The case study and failure cases are informative and interesting.

Weaknesses

- My major concern is that the current pipeline is highly dependent on the LMM which is the powerful closed-source model, GPT-4V. This model is expensive for large-scale applications and also hard to reproduce due to unannounced updates for API across time. It would be better to provide the results with open-source large multi-modal models, for example, LLaVA. I would expect a lower accuracy with open-source models. - The proposed MP16-Pro dataset is also claimed as a contribution, but there is no guarantee that the data will be released. Hope this will be provided in final version.

Questions

See the weaknesses.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

The limitation is included in the appendix.