Summary
The authors propose a multi-level contrastive learning (MCL) method tailored for dense prediction downstream tasks. The core idea is to downsample original images in different scales and assemble them in a montage style as input. Since the location of each sub-image is known as a priori, the authors use this supervision signal to perform contrastive learning on multi-scale features. Extensive experiments demonstrate that the proposed method achieves superior performance when transferred to various dense prediction tasks.
Weaknesses
- **Effectiveness on non-object-centric datasets.** In the introduction section, the authors claim that object-level SSL methods use off-the-shelf algorithms to produce object proposals, which are not accurate enough on non-object-centric datasets like COCO. To tackle this issue, the authors should mainly conduct pre-training experiments on COCO to compare with previous object-level SSL methods. However, in the paper, the authors mainly pre-train MCL on the object-centric ImageNet dataset. From my understanding, MCL still relies on the object-centric bias of ImageNet since the proposed montage strategy should ensure that each sub-image only contains a single object. Otherwise, pulling sub-images containing multiple objects together will mislead the network. Therefore, I am afraid that the proposed montage strategy may not be suitable for non-object-centric datasets like COCO. According to Table 1 (the only pre-training experiments on COCO in the whole paper), the COCO-pretrained MCL (41.8 $AP^{bb}$, 37.7 $AP^{mk}$) performs worse than the ImageNet-pretrained MCL (42.5 $AP^{bb}$, 38.2 $AP^{mk}$). Considering that the authors have adjusted the number of epochs to match the training iterations on ImageNet, such results further indicate that MCL may not be effective on non-object-centric datasets.
- **Potentially unfair comparisons.** Apart from the backbone, the authors also pre-train FPN neck and detection head. However, most previous methods do not pre-train FPN and detection head. To ensure a fairer comparison, the authors should also pre-train FPN and detection head for previous methods (e.g., DenseCL). Another option is to pre-train MCL without FPN and detection head. This can better verify whether the montage assembly itself is superior over other pixel-level or object-level pretext tasks without the interference of other factors. According to Table 10 (e), only pre-training the backbone (41.1 $AP^{bb}$, 37.2 $AP^{mk}$) does not perform better than many previous methods that also only pre-train the backbone, as shown in Table 3. Given the current results, I am concerned that the main gains come from using additional multi-scale features in FPN for contrastive learning rather than the montage assembly. Furthermore, the claim that MCL can reduce pre-training epochs may be simply due to pre-training FPN and detection head. In other words, previous methods may also be able to reduce pre-training epochs by pre-training FPN and detection head.
- **Some related works are missing.** There are also several closely related dense/object-level SSL works (e.g., [1, 2, 3]) that should be discussed and compared in Sec. 2 and Sec. 4.
References:
[1] Self-Supervised Visual Representation Learning by Contrastive Mask Prediction. In ICCV, 2021.
[2] Unsupervised Object-Level Representation Learning from Scene Images. In NeurIPS, 2021.
[3] Point-Level Region Contrast for Object Detection Pre-Training. In CVPR, 2022.
Questions
I have additional questions:
- The authors use a large batch size of 4096 during pre-training. Considering that the number of composed images is correlated with the batch size, I am curious about the effect of the batch size on MCL. Whether a small batch size can also work well?
- The authors use a contrastive loss with positive and negative samples. Considering that many recent works (e.g., BYOL, SimSiam) remove the necessity of negative samples, what if a similar loss without negative samples is used in MCL? Will the performance be degraded?
- The authors use an additional predictor for the online network. Could the authors provide a justification on this?
- I am confused about the setting that extends MCL to the supervised learning scenario. According to the implementation details in the main text and the supplementary material, if I understand correctly, all models use supervised pre-training, while MCL is only used together with supervised fine-tuning. If this is the case, other dense/object-level pretext tasks (the authors could choose some baselines in Table 3 for comparisons, e.g., SoCo) should also be implemented in this setting to demonstrate the superiority of MCL.
Some other suggestions:
- Sec. 4.2: Pre-training on Non-object-centric Dataset: "1.4 $AP^{bb}$/1.3 $AP^{mk}$ over SoCo" -> "1.2 $AP^{bb}$/1.3 $AP^{mk}$ over SoCo".
- Some numbers in Table 1 are not correct. For example, Self-EMD $AP^{bb}$ should be +1.5 rather than +1.3, SoCo $AP^{bb}$ should be +1.7 rather than +1.5. Please also check other parts.