Paving the way toward foundation models for irregular and unaligned Satellite Image Time Series

Although recently several foundation models (FMs) for satellite remote sensing (RS) imagery have been proposed, they fail to address major challenges of operational applications. Indeed, representations that do not take into account the spectral, spatial, and temporal dimensions of the data as well as the irregular or unaligned temporal sampling are of little use for most real-world applications. As a consequence, we address some existing shortcomings in the design of FMs for RS data. In particular, we propose an ALIgned Sits Encoder (ALISE), a novel approach that leverages the spatial, spectral, and temporal dimensions of irregular and unaligned satellite image time series (SITS) while producing aligned latent representations. ALISE provides easy-to-use fixed-size SITS representations that preserve the spatial resolution of the input SITS required for most mapping tasks. Moreover, to learn informative representations of SITS, we investigate the integration of instance discrimination losses within a masked autoencoding pretraining task, utilizing a multiview framework. The model is pretrained on a custom-built Sentinel-2 (S2) multiyear SITS unlabeled dataset. The genericity of the provided representations is assessed on three downstream tasks: crop segmentation, land cover segmentation, and an unsupervised crop change detection task. The results suggest that the use of ALISE’s aligned representations is significantly more effective than previous self-supervised learning (SSL) methods for linear-probing segmentation tasks. Additionally, the experiments show the interest of using ALISE representations for unsupervised change detection. Finally, the impact of the pretraining hyperparameters and the proposed method for aligning irregular and unaligned time series are examined in detail. The code, the pretrained model, and the datasets are released at https://src.koda.cnrs.fr/iris.dumeur/alise