Modality-Adaptive Mixup and Invariant Decomposition for RGB-Infrared Person Re-Identification

RGB-infrared person re-identification is an emerging cross-modality\nre-identification task, which is very challenging due to significant modality\ndiscrepancy between RGB and infrared images. In this work, we propose a novel\nmodality-adaptive mixup and invariant decomposition (MID) approach for\nRGB-infrared person re-identification towards learning modality-invariant and\ndiscriminative representations. MID designs a modality-adaptive mixup scheme to\ngenerate suitable mixed modality images between RGB and infrared images for\nmitigating the inherent modality discrepancy at the pixel-level. It formulates\nmodality mixup procedure as Markov decision process, where an actor-critic\nagent learns dynamical and local linear interpolation policy between different\nregions of cross-modality images under a deep reinforcement learning framework.\nSuch policy guarantees modality-invariance in a more continuous latent space\nand avoids manifold intrusion by the corrupted mixed modality samples.\nMoreover, to further counter modality discrepancy and enforce invariant visual\nsemantics at the feature-level, MID employs modality-adaptive convolution\ndecomposition to disassemble a regular convolution layer into modality-specific\nbasis layers and a modality-shared coefficient layer. Extensive experimental\nresults on two challenging benchmarks demonstrate superior performance of MID\nover state-of-the-art methods.\n