AdaptSSR: Pre-training User Model with Augmentation-Adaptive Self-Supervised Ranking

Summary

The paper tackles user-oriented tasks, e.g. personalized recommendation, and proposes a self-supervised method named AdaptSSR to replace the contrastive learning pre-training target task. It adopts a ranking loss that selects samples of smallest similarity differences and assigns dynamic weight coefficients to ranking parts based on the estimated similarity between the augmented views. Experiments on 6 downstream tasks from 2 datasets and several empirical analyses are conducted to verify the effectiveness of AdaptSSR.

Strengths

The pre-training method and objective are clearly explained, and the equations in the text concisely demonstrate the proposed loss function.

Weaknesses

The Multiple Pairwise Ranking loss, which is the core of the method, is not an original contribution of this paper, but an adaptation from Yu et al [49]. However, there is almost no mentioning of this work except for the source of the loss function, casting doubt on the novelty of the paper.

Questions

Please elaborate on the differences between the approach proposed in this paper and [49], and highlight the contribution of this paper.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have not discussed the limitations and broader societal impacts in the paper.

Summary

Recent studies have explored pre-training user models with contrastive learning tasks to address data sparsity issues in user-oriented tasks. However, existing augmentation methods may introduce noisy or irrelevant interests, leading to negative transfer. To overcome this, a new approach called Augmentation-Adaptive Self-Supervised Ranking (AdaptSSR) is proposed, which replaces contrastive learning with a multiple pairwise ranking loss. An augmentation-adaptive fusion mechanism is also introduced to combine learned ranking orders based on the similarity between augmented views. Extensive experiments demonstrate the effectiveness of AdaptSSR across various tasks and datasets.

Strengths

1. The paper's motivation is reasonable, directly adopting contrastive learning may lead to consistency problems in recommendations. 2. This paper proposes a novel approach to combine implicit and explicit augmentations.

Weaknesses

1. The main contribution of this paper is adding an order constraint in the loss function, which is a rather incremental modification of existing contrastive learning framework. The main idea of paper is a fusion of explicit augmentation and implicit augmentation by the loss function. Thus, the novelty is limited. 2. It's unclear whether the added constraint is necessary. Since "$u$ and $u^+$ originate from exactly the same input behavior sequence" as the authors commented in line 123, I think $sim(u, u^+)\ge sim(u, u^-)$ and $sim(u, u^+) \ge sim(u, \hat{u}$ should always hold. I don't understand why we need the $sim(u, u^+)$ term here. Without the $sim(u, u^+)$ term, the proposed method reduces to common contrastive learning. 3. Even if the constraint is meaningful, the authors' analysis cannot convince me why such constraint may help generalization. Why may $sim(u, \hat{u}) > sim(u, u^+)$ harm the downstream performance? Intuitively, suppose the objective of original contrastive learning is overly strong, we should loose the constraints. For example, $sim(u, \hat{u}) \ge sim(u, u^-) - \epsilon$. However, the authors make the constraints even stronger by adding another constraint term. This does not make sense to me. --- Edit after rebuttal: The authors' response resolved my primary concern about technical correctness. I'd like to raise my score to a borderline reject regarding the novelty of this work.

Questions

1. Please explain what is implicit augmentation, how is $u^+$ generated, and why equation (1) should hold. 2. The authors introduce a Augmentation-Adaptive Fusion coefficient $\lambda$ needs further discussion. Is the $\lambda$ fixed during training, i.e. stop_gradient($\lambda$), so that the gradient calculation does not involve $\lambda$?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

1 poor

Presentation

1 poor

Contribution

2 fair

Limitations

Please address the issues highlighted in the Weaknesses section.

Summary

This paper proposes Augmentation-Adaptive Self-Supervised Ranking (AdaptSSR), a new user model pre-training paradigm, which alleviates the requirement of semantic consistency between the augmented views while pre-training a discriminative user model. Conventional methods assume that different views of the same behaviour sequence constructed via data augmentation are semantically consistent, while in practice existing augmentation methods tend to lose certain interests of the user or introduce noisy interests that the user does not have. AdaptSSR addresses this issue by adopting a multiple pairwise ranking loss which trains the user model to capture the similarity orders between the explicitly augmented views, the implicitly augmented views, and views from other users. An explicit hard negative sampling strategy and an augmentation-adaptive fusion mechanism are also introduced to facilitate model training. Extensive experiments on both public and industrial datasets verify the effectiveness of AdaptSSR.

Strengths

- The proposed approach is technically sound and the empirical results validates the effectiveness of the method. - The paper is well-written with very clear figures. - Code is available which makes it easy to reproduce the results.

Weaknesses

An important hyperparameter sensitivity analysis is missing: how does the value of $\lambda$ affect the model performance? Compared with existing models, AdaptSSR introduces an extra SimCSE-inspired implicit augmentation approach. It remains unclear in the paper if the performance improvement is primarily due to the introduction of implicit data augmentation.

Questions

Only three random augmentation operators are used in the paper. Does introducing more augmentation operators help improve the performance? Do informative augmentation operations introduced by CoSeRec help improve AdaptSSR's performance?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The extra computational cost introduced by AdaptSSR is not analyzed in the paper, it would be useful if the authors can demonstrate the tradeoff between training time and performance for various models.

Summary

The authors tackle the problem of doing self-supervised learning for user modeling. Inspired by the successes of contrastive learning approaches in the image setting, they adapt contrastive learning to the user modeling setting. However, in user modeling the augmentations typically used are not very suitable for contrastive learning because they can change the semantics of the data, thus forcing similarity between augmented views can be problematic. They instead produce three views: the anchor, a similar "implicitly" augmented view, and a less similar "explicitly" augmented view. The implicitly augmented view is trained to be more similar to the anchor than the explicitly augmented view. This escapes the problematic similarity training that plain contrastive learning would have in user modeling.

Strengths

1. The paper was well-written and the diagrams were easy to understand. It made the paper easy to read and review. The contributions were clearly stated and explained in the paper. 2. The method is novel and original as far as I know. This method could be generalizable to other domains where augmentation choices are not straightforward and could alter the semantics of the data. 3. The method is well-designed: the ranking loss does help mitigate the "make semantically different augmented views the same" problem, and furthermore helps balance the focus of the loss between the implicit vs explicit contrast and the explicit vs other user contrast. 4. The improvements in the empirical results are consistent and seem to be significant.

Weaknesses

1. Going back to the example where the user behavior is represented by a sequence of images, is it possible to just do per-image augmentation (choices for these exist and are widely used) and then perform a typical InfoNCE style contrastive loss on the user embeddings? For text one could do something similar using masking augmentations and such. I did not see a comparison to this baseline and I wonder how well it would perform. I think this is something that would be critical to compare against. 2. While the paper is written well, I think the paper should define what user modeling is and what the downstream tasks are earlier in the paper (or in the abstract). For a while it was not clear to me what problem the paper was trying to solve, as someone who has not worked on user modeling. 3. Adding error bars into the results tables would help in understanding the significance of the results.

Questions

Questions are listed in the weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Seems sufficient.

Thanks for the rebuttal

I will keep my score after reading the rebuttals and the other reviews. Thanks for adding significance test results.