Do SSL Models Have Déjà Vu? A Case of Unintended Memorization in Self-supervised Learning

Summary

This paper thoroughly studies a particular undesired aspect of state-of-the-art self-supervised learning models: deja-vu memorization, where the models end up memorizing specific parts of the image instead of learning just a semantically meaningful association. The authors provide extensive empirical results to show when SSL models might memorize, and what aspects of training might minimize this effect.

Strengths

* The authors introduce novel idea and testing methodology for SSL models * Provides comprehensive empirical results on different aspects of SSL modeling and training and memorization * Additional results on ablations and using different representations and aspects of the setup help understand the author’s proposed method better * Clear presentation of results and setup.

Weaknesses

* Authors do not seem to talk about the impact of different augmentations on memorization, if any. Augmentations are very important for SSL performance, maybe they impact memorization? * It is hard to judge how confidently we can estimate the memorization if deploying. We compare to only one other reference SSL model. * The importance of using the generative model is not quite clear. From the result figures, the kNN images themselves seem to do enough. * Only shows using ImageNet data. No way to know if it will generalize to other datasets as well.

Questions

* How do different augmentations used by SSLs impact memorization? * Is using just one other SSL model enough to give confident results of memorization? Should we include more reference SSL models to increase confidence? * How many samples would be needed in the X public dataset? Could be helpful to understand the scale if deployed. * When is the generative model actually needed, when is it necessary beyond just looking at the k nearest neighbors?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

* The lack of a way to estimate the confidence or uncertainty in the memorization score. * Although shown for many models, the results are shown only for a single dataset. Using more datasets could have been nice, maybe using CelebA to show background info is memorized to learn person’s identity.

Summary

This paper shows how self-supervised learning (SSL) models can memorize information about their training data in ways that are not intended. SSL is a technique for learning representations of unlabeled data by training a model to solve a "pretext" task; for example, we might take training images A, B, and C and crop them in different ways, producing sets of patches (A1,...,An), (B1,...,Bn), and (C1,...,Cn). The algorithm tries to find an embedding of these patches that makes Ai similar to Aj, while minimizing the similarity between Ai and Bj. These embeddings are used for a variety of downstream tasks. This paper shows that the representations of training patches can contain surprising information about the training data. In the headlining example, the authors take an image of a black swan from the training data, extract a water-only patch, and produce an embedding of this patch. Then a generative model, which (ideally? see below) has no access to the training data, produces an image from this embedding: it contains a (different?) black swan. The water-only patch contained no information about swan, so the SSL model "remembered" what was in the image the patch came from. The trick fails when the same patch is run through an SSL model that was trained on different data: it happens only because the patch was seen in the training data. The paper has two methods for detecting such memorization. The first predicts the class label given the patch's representation. The second, mentioned above, visually inspects images generated from the representation. In both cases the classification/generation is done by a model trained on a different data set. The label-prediction method demonstrates quantitatively that the representation contains information about the label, while the visual inspection reveals finer details: the paper contains a great example of how SSL models can learn and memorize the distinction between European and American badgers, even though both share an ImageNet label.

Strengths

If the authors resolve my dataset question below, I am happy to call this a very clean and original demonstration on a significant and relavant topic. It is definitely a paper to generate good discussion. The paper performs extensive experiments, showing how the results change as they turn various knobs. The quantitative and qualitative techniques complement each other nicely. The work lays out clear directions for future work.

Weaknesses

As I understand from Appendices A.2.1 and A.2.3, "private" sets A and B come from ImageNet 1k, which is a subset of ImageNet. The "public" set X is a face-blurred version of ImageNet. This appears to contradict the experimental setup of Section 3.1 and, if it is the case, means I am not sure how to interpret the results. I could be mistaken and am open to raising my score. EDIT: the rebuttal adequately addressed these concerns. I was initially confused the use of the word "reconstruction." For example, Figure 1 talks about "reconstruction of an SSL training image," but the generated image looks quite different from the training image. It seems you use the word differently than recent work does [1-5], where it means producing an example that is pixel-by-pixel, or feature-by-feature, similar to the input. I suggest using a different word or clarifying the issue earlier in the paper. I think synthetic-data experiments might strengthen the paper. They might nail down specific aspects of this phenomenon. One might better probe the types of memorized information. (Of course, easier suggested than done.) I find the related work section adequate, but I think Section 3, "Defining Deja Vu Memorization," would benefit from clearer connections between prior work and the presented definitions. In particular, the definition at line 111 has similarities to the notion of memorization used in [2], which also attempts to capture the idea of memorization of specific information about individual examples, beyond what one can infer from the data distribution. The definition for testing methodology, at line 130, is similar to the definition of memorization used in [6,7]. [1] Balle, Borja, Giovanni Cherubin, and Jamie Hayes. "Reconstructing training data with informed adversaries." 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022. [2] Brown, Gavin, et al. "When is memorization of irrelevant training data necessary for high-accuracy learning?." Proceedings of the 53rd annual ACM SIGACT symposium on theory of computing. 2021. [3] Haim, Niv, et al. "Reconstructing training data from trained neural networks." Advances in Neural Information Processing Systems 35 (2022): 22911-22924. [4] Guo, Chuan, et al. "Bounding training data reconstruction in private (deep) learning." International Conference on Machine Learning. PMLR, 2022. [5] Hayes, Jamie, Saeed Mahloujifar, and Borja Balle. "Bounding Training Data Reconstruction in DP-SGD." arXiv preprint arXiv:2302.07225 (2023). [6] Feldman, Vitaly. "Does learning require memorization? a short tale about a long tail." Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing. 2020. [7] Feldman, Vitaly, and Chiyuan Zhang. "What neural networks memorize and why: Discovering the long tail via influence estimation." Advances in Neural Information Processing Systems 33 (2020): 2881-2891.

Questions

Section 3.1 says the datasets A, B, and B are disjoint. Appendix A.2.1 says that A and B overlap. Furthermore, as mentioned above, my understanding is that X is a face-blurred version of ImageNet, and thus X might contain (possibly blurred) examples from A and B. Can you clarify this? Could you lay out what the "adversary," the person conducting the privacy attack, has access to?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The authors adequately address the limitations of their work.

Summary

Self-Supervised Learning algorithms provide a label-free alternative for representation learning and is widely used in vision, and language. In this work, the authors try to understand the information content of representations learned by SSL algorithms through the lens of memorization. In particular, they define a "deja-vu" memorization notion that measures how much a given sample from the training set can be reconstructed from a pretrained model. The authors focus on joint-embedding-based approaches, exploring the extent of such memorization during pretraining by evaluating label-inference accuracy on peripheral crops of a subset of training images ($A$). A "deja-vu" score at p% confidence measures the gap in accuracy between a target model (trained on $A$) and a reference model (trained on a different subset $B$). The gap essentially measures the extent to which exposing particular samples during training makes it easier to predict the label from partial background information. Building on these metrics, the authors present interesting results comparing different SSL algorithms under different training configurations. Notably, long pretraining increases the risk of memorization, while the size of the dataset doesn't is as consequential (Fig 4/5). With a sequence of ablations, the authors establish empirical findings and provide design recommendations that minimize deja-vu memorization while maintaining high performance.

Strengths

Some strengths of the work are outlined below: **Originality**: The authors examine the representation quality in SSL from a privacy perspective. The work focuses on train-time characteristics and memorization for non-generative SSL algorithms, in line with similar works in language/generative models. The authors propose a novel leave-one-out mechanism for evaluating the extent of memorization (compared to correlation) of specific training-set examples across SSL algorithms. **Significance**: Understanding the information content of representations learned by black-box SSL algorithms is important as these algorithms become widely used in safety-critical applications. In particular, the paper makes a solid case for understanding the privacy risk induced by long-pretraining routines in existing SSL algorithms. **Quality**: The work is well-motivated and grounded in real-world challenging questions related to measuring the quality of representations learned in SSL algorithms. **Clarity**: The paper is well written, with definitions and pedagogical examples that motivate the core insights of the work. The authors include clear, well-labeled figures and plots accompanying their observations. **Reproducibility**: All experiments are performed on public datasets and pretrained models and sufficient details to reproduce the core results of the paper. The authors are encouraged to release the accompanying code at their earliest convenience.

Weaknesses

Some weaknesses of the work are outlined below: **Explainability**: The authors have shared some interesting empirical findings about the effectiveness of SSL algorithms in representing data and their ability to recall examples from the training set from partial side information. Although the authors demonstrate that established methods like guillotine regularization reduce the risk of memorization, the current version of the study could benefit from more discussion on the underlying mechanisms behind these mitigation techniques.

Questions

1. Although the authors primarily concentrate on joint-embedding-based techniques, it is probable that reconstruction-based methods are also susceptible to this type of memorization. To enable a fair comparison across the SSL algorithm family, it would be useful to include a baseline such as MAE.nclude a baseline such as MAE. 2. The focus of the authors is on the linear probe as a potential measure of representation quality. However, they have shown that this metric is inadequate in capturing the full extent of deja-vu memorization. It would be valuable to determine if this inconsistency still exists even when the model is fine-tuned, which involves adapting the backbone as well, in an end-to-end approach.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper explores the privacy-risk profile of SSL algorithms through the lens of training-set memorization. The authors are strongly encouraged to discuss their proposed algorithm's limitations (including assumptions and failure modes).

Summary

This work investigates to what degree self-supervised models exhibit memorization of particular samples in the training set. The authors demonstrate that a simple crop of an image, only comprised of the background, is enough for an adversary to learn the associated class label of the foreground object. Moreover, a conditional generative model can be trained, allowing the adversary to reproduce a very similar-looking image compared to the original. Such memorization capacity can lead to privacy issues and the authors demonstrate that such behaviour occurs in a broad range of models, while at the same time, this effect is not visible in common metrics such as probing accuracy.

Strengths

1. The paper is very well written and builds upon the simple but very interesting idea of memorization in self-supervised models. Such an investigation seems timely, given the similar recent efforts in large language models. 2. The experiments are very well designed, clearly disentangling the notion of correlation (i.e. actually learning something generalisable about the sample) and pure memorization (i.e. only memorizing the specifics of the sample). The splitting of the training data into two sets and training two separate models is a very neat way to measure this effect, which would otherwise remain very difficult to disentangle. 3. It is very interesting that supervised and self-supervised methods behave very differently in terms of memorization. In general, studying the difference in the two learning approaches is very worthwile and timely.

Weaknesses

1. I struggle to really see the privacy concern here. How likely is it really for someone to have access to only a crop of an image, while not being able to access the full image? The authors present the example of an image of a face of a person, which could then be potentially used to infer where that person is. But again, in what situation do I have access to the face of an image without the rest of it? As the authors also demonstrate, such an image has to be present in the training data, it is not enough to simply use another photo of a person and then the model reveals another image of the same person. I also believe that this scenario is different from the language model case, where simply “guessing” a good query as input for the model to complete seems to be more realistic due to the smaller search space and potential knowledge of similar queries. I hope the authors can comment on this. 2. This is more minor but while I really like the experimental setup, I’m still not sure if memorization and generalization are completely disentangled. Assume that you have a model that clearly memorizes images but also performs very poorly at the downstream task, i.e. its linear probe is around random guessing. Due to the complete lack of generalization, it will be tough to establish memorization since the nearest neighbours will all share no correlation. So from a pure theoretical perspective, this seems a bit unsatisfying. Of course in practice, no generalization is usually not the case, so from a practical viewpoint this makes complete sense.

Questions

What happens if you perform the nearest neighbour class inference on the training set (excluding the image itself)? This would circumwent the issue outlined above, but of course defeats the practicality of the method. Still seems like an interesting experiment though.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have addressed the limitations of their work.