Summary
This paper tackles the image geolocalization problem. Instead of formulating the task as image retrieval or classification over a set of geospatial bins as is typical, the authors frame it as image-to-GPS retrieval by matching image features with location features. The proposed training strategy leverages Random Fourier Features (RFF) to encode geospatial location across multiple scales and align them with features from a CLIP visual encoder, with several training enhancements. The resulting method achieves state of the art performance on three benchmarks, across all scales, and showcases interesting applications in text geolocalization and classification.
Strengths
- Addresses an important and interesting problem. Image localization is a fundamental problem in vision that continues to receive attention in the literature.
- The approach has technical novelty and is of the first to formulate the localization task as an image-to-GPS retrieval problem. The key idea is to relate images directly to locations via aligning image features to location embeddings using contrastive learning. Such an approach uses a continuous model of location whereas traditional geolocalization approaches depend on a large reference database of images with known locations to support image-level matching (via image retrieval or classification) and inferring location from the closest match. Instead, the authors propose the concept of a GPS gallery. Though training still requires geotagged images (image + location), the authors show the proposed approach requires less training data. Further, at inference, the GPS gallery can theoretically be any set of locations to query against, instead of being limited by the reference image set. Geospatial location is represented using NeRF-style location embeddings (Random Fourier Features with an MLP). Results show that infusing the location representations with multiple spatial scales improves geolocalization performance.
- Evaluation is extensive and compelling. State-of-the-art (SOTA) performance on multiple datasets at different threshold scales (street, city, region, country, and continent). Near SOTA performance on Im2GPS3k training using only 20% of the training data.
- Intriguing applications. The results on text geolocalization and geotagged image classification showcase the utility of the location encoding beyond image-based geolocalization.
Weaknesses
- A central idea of this paper is the concept of modeling location continuously. I was disappointed to see that during evaluation, the GPS gallery (the set of possible locations) was limited to locations sampled from the training set (L230). Though I understand why this was done from an evaluation perspective, it significantly hurts the "global localization" theme. At the very least, I would have liked to see an evaluation of how selecting the GPS gallery at inference (choice of sampling distribution, granularity, etc.) impacts performance. This also raises the issue that the location encoder is not truly global, it is limited by where images have been captured on the globe (as training requires image/location pairs).
- Some minor details missing. Section 4.2 does not mention which dataset the experiment was performed on. Comparing the numbers from Figure 3 and Table 1a, it can be assumed to have been conducted on Im2GPS3k, but the dataset should still be mentioned to improve clarity. Similarly, section 4.3 does not mention which dataset(s) was/were used for the ablation study.
- The application for text query-based geolocalization is interesting, but the results are lacking. It is only demonstrated qualitatively on a continent scale using a single keyword (desert) in the main paper. While this is interesting, it leaves a lot to be desired in terms of understanding the full or even partial capability of the text encoding for tasks such as geolocalization or digital forensics.
- Quality of the presentation could be improved. For example, the introduction could provide additional motivation for the underlying problem. In its current form, the introduction mentions that there are a variety of applications for geolocalization, but does not support this claim with any citations or descriptions of how geolocalization benefits them. Instead, the majority of content is related to why the problem is difficult and how existing approaches are inadequate, repeating much of the related work section content (e.g., binning, classification, and scaling methods). The conclusion section, in particular, lacks flow and could benefit from additional editing. Unclear mathematical notation on L138. What is K? Minor grammatical errors (L49, L107, L111, L372, etc.).
- Word choice in the conclusion. L353 suggests that this work “solved worldwide geolocalization”. Of course this problem is far from solved!
Questions
My initial rating is weak accept. This paper introduces a method for training a high-quality geospatial location encoder that interacts with, widely used, CLIP features and demonstrates its use for localization and several applications. This has the potential to benefit the wider community.
Quantitatively, how does the proposed approach compare in terms of runtime and memory footprint with existing approaches?
How does the text encoding perform at smaller scales (region or city)?
Can the text encoding benefit the image-based geolocalization?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
Limitations addressed in the manuscript.