SS-MAE: Spatial-Spectral Masked Auto-Encoder for Multi-Source Remote Sensing Image Classification

Masked image modeling (MIM) is a highly popular and effective self-supervised learning method for image understanding. The existing MIM-based methods mostly focus on spatial feature modeling, neglecting spectral feature modeling. Meanwhile, the existing MIM-based methods use Transformer for feature extraction, and some local or high-frequency information may get lost. To this end, we propose a spatial–spectral masked autoencoder (SS-MAE) for hyperspectral image (HSI) and light detection and ranging (LiDAR)/synthetic aperture radar (SAR) data joint classification. Specifically, SS-MAE consists of a spatialwise branch and a spectralwise branch. The spatialwise branch masks random patches and reconstructs missing pixels, while the spectralwise branch masks random spectral channels and reconstructs missing channels. Our SS-MAE fully exploits the spatial and spectral representations of the input data. Furthermore, to complement local features in the training stage, we add two lightweight convolutional nerual networks (CNNs) for feature extraction. Both global and local features are taken into account for feature modeling. To demonstrate the effectiveness of the proposed SS-MAE, we conduct extensive experiments on three publicly available datasets. Extensive experiments on three multisource datasets verify the superiority of our SS-MAE compared with several state-of-the-art baselines. The source codes are available at https://github.com/summitgao/SS-MAE.