Cross Teaching between Single-Spectral and Multi-Spectral Detection Transformers for Remote Sensing Object Detection

Abstract

In recent years, to enhance all-weather observation capabilities, remote sensing platforms have been increasingly equipped with thermal infrared (TIR) sensors in addition to visible spectrum (RGB) sensors. Consequently, remote sensing object detection methods started utilizing images from both modalities to improve detection accuracy. However, the inconsistency of target visibility across the two spectrums would introduce confusion of the multi-spectral model, leading to lower detection performance compared to the single-spectral TIR model. In this study, a fine-grained cross-teaching method is proposed to mitigate confusion caused by visibility inconsistency. In detail, leveraging the one-to-one matching mechanism of Detection Transformer (DETR), if a target is visible in both images, knowledge is distilled from a multi-spectral DETR to a TIR-only DETR. Conversely, if an object is only visible in the TIR image, reverse distillation is conducted. Experiments show that cross teaching aids both the single-spectral and multi-spectral models, achieving the state-of-the-art performance on the DroneVehicle dataset.