MultiSiam: Self-supervised Multi-instance Siamese Representation Learning for Autonomous Driving

Autonomous driving has attracted much attention over the years but turns out\nto be harder than expected, probably due to the difficulty of labeled data\ncollection for model training. Self-supervised learning (SSL), which leverages\nunlabeled data only for representation learning, might be a promising way to\nimprove model performance. Existing SSL methods, however, usually rely on the\nsingle-centric-object guarantee, which may not be applicable for multi-instance\ndatasets such as street scenes. To alleviate this limitation, we raise two\nissues to solve: (1) how to define positive samples for cross-view consistency\nand (2) how to measure similarity in multi-instance circumstances. We first\nadopt an IoU threshold during random cropping to transfer global-inconsistency\nto local-consistency. Then, we propose two feature alignment methods to enable\n2D feature maps for multi-instance similarity measurement. Additionally, we\nadopt intra-image clustering with self-attention for further mining intra-image\nsimilarity and translation-invariance. Experiments show that, when pre-trained\non Waymo dataset, our method called Multi-instance Siamese Network (MultiSiam)\nremarkably improves generalization ability and achieves state-of-the-art\ntransfer performance on autonomous driving benchmarks, including Cityscapes and\nBDD100K, while existing SSL counterparts like MoCo, MoCo-v2, and BYOL show\nsignificant performance drop. By pre-training on SODA10M, a large-scale\nautonomous driving dataset, MultiSiam exceeds the ImageNet pre-trained MoCo-v2,\ndemonstrating the potential of domain-specific pre-training. Code will be\navailable at https://github.com/KaiChen1998/MultiSiam.\n