Weaknesses
I will merge my questions and weaknesses in this section. Specifically, I want to point out major concerns that are critical to me and some minor concerns that do not affect my decision much.
### [Major Concerns]
First of all, the strongest criticism I have for this paper is that **it is built on many strong and unverified assumptions** and I will elaborate below:
**(1) [Regarding limitation of previous methods]**: In the introduction, the authors motivate the problem by pointing out weaknesses of previous works, which seems problematic to me, for example, they mention that for SSL-backdoor (Saha et al.
2022):
> As a result, the backdoored encoder fails to build strong correlations between the trigger and images in the target class, leading to suboptimal results.
This claim is not immediately clear to me in two regards: (a) the failure to build strong correlations is not supported by empirical evidence. The authors ought to quote similar statements from the original paper or design experiments to verify this; (b) the relationship between such correlations and attack effectiveness is again not supported by either theoretical or empirical results and seems handwaving to me.
Next, the authors criticize PoisonedEcndoer (Liu et al. 2022) as having limited effectiveness due to the following reasoning:
> The limitation arises due to the absence of a theoretical
analysis that guides the optimization of feature similarity between the trigger and objects in the target class.
I later realized that the authors use this claim to highlight their theoretical contribution (which also does not explore the optimization of the feature similarity for CL in my opinion), but still, it does not make sense to me that the limitation on scaling up to large dataset is due to "the lack of theoretical analysis".
Overall I suggest the authors be extra careful in discussing prior works. These claims need to be properly validated to serve as reasonable motivations for the paper.
**(2) [Unsupported claims and missing aspects on contrastive learning]**: Again in the introduction session, the authors mention that:
> Our attack crafts poisoned images via exploiting the random cropping mechanism as it is the key to the success of CL (The encoder’s utility sacrifices substantially without random cropping).
I generally agree with the argument as I am fairly familiar with contrastive learning. However, it is essential to provide relevant evidence (i.e., linear evaluation accuracy with/without random cropping).
Additionally, throughout the paper, the authors seem to only consider random cropping while other augmentation methods exist (e.g., random flip, color distortion, Gaussian blur). These augmentations would significantly affect both the reference object and the trigger. These effects should not be neglected and need to be properly discussed by the authors.
**(3) [General assumption of the proposed method]:** The proposed CE method is built on the following assumption:
> Therefore, if one augmented view includes (a part of) a reference object and the other includes the trigger, then
maximizing their feature similarity would learn an encoder that produces similar feature vectors for
the reference object and any trigger-embedded image. Therefore, a downstream classifier would
predict the same class (i.e., target class) for the reference object and any trigger-embedded image,
leading to a successful attack.
The above argument may sound intuitively reasonable, but is too strong for me and deserves verification. For example, the authors can design a simple experiment to verify this: one can handcraft such positive pairs (i.e., one with the reference objective only, another with the trigger only) by writing a simple customized clipping function (of course it is not random anymore). If the assumption is true, the experiment should lead to a "successful attack" (i.e., 100% ASR). In fact, the authors could also compare with a simple baseline to verify the claims on SSL-backdoor by placing the triggers randomly or according to the original paper.
**(4) [Reference images and poisoned ratio]:** The authors claim that they use fewer reference images (I assume it is 3 vs 650) and a lower poisoned ratio, but it appears that in the experimental settings, the poisoned ratio is 0.5% across all baseline methods. In the baseline paragraph, the authors mention:
> We follow the same setting for their attacks, which gives advantages to them.
What are the advantages? To me it seems to be an unfair comparison, I assume that you are using 3 reference images, repeated 650 times, while the baseline methods use 650 distinct reference images. In that case, does repeating help? What if you only apply 3 poisoned images?
**(5) [Problematic assumption for the theory]**: One page 4, the authors presents:
> For simplicity, we assume the regions are squares and they have the
same size s.
This assumption is not reasonable to me as it is not the case in practice. Here I quote the description of random clipping in SimCLR (Chen et al. 2020), Appendix A:
> The crop of random size (uniform from 0.08 to 1.0 in area) of the original size and a random aspect ratio (default: of
3/4 to 4/3) of the original aspect ratio is made.
We can immediately see that this largely deviated from the assumption, where the authors assume that the random size is a constant, in contrast to uniform from 0.08 to 1.0. Therefore, the following theory seems meaningless to me as it only applies to constant random size and does not provide insights into real-world scenarios. Thus it appears to me that the proposed method can be introduced as heuristics and could be probably verified by Figure 2(b), but is not well-supported by the proposed theory.
**(6) CorruptEncoder+**: I don't understand what this part is achieving. Why is a data poisoning algorithm optimizing weights in Equation (4)? Are you interfering with the training process? If that's the case, it is contradictory to your threat model in Section 2, where the attacker does not know the pre-training settings.
### [Minor Concerns]
(1) [Presentation needs to be improved]: The presentation of the paper could be improved. For example, in Section 2, the notation such as $e_{ti}$ is cumbersome and can be easily avoided as the authors do not even consider downstream tasks in sequences throughout the paper.
(2) [Utility of the attacks]: the author mentions that a good attack should obtain higher ASR and reasonable downstream task performance. The utility should also be reported for the baselines for better comparison.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.