Learning List-Level Domain-Invariant Representations for Ranking

Summary

The authors propose a new domain-invariant ranking method which enforces list-wise feature invariance as opposed to item-wise feature invariance. The authors prove a novel domain adaptation bound specifically for the ranking problem. Authors show strong empirical improvement over item-wise invariance methods.

Strengths

(1) The theorem is new and not a trivial extension of the existing domain adaptation generalization bound from Ben-David (which is for classification). In particular, some new techniques are required to analyze the ranking problem. (2) The method is novel and results are good.

Weaknesses

(1) My main concern is with the empirical improvement between ItemDA and ListDA. I remain unconvinced that this improvement is coming from the list-wise invariance and not from the improvement in discriminator architecture. The authors mention that ItemDA uses a three-layer MLP, while ListDA uses a stack of three transformer blocks. is there anyway to compare the two methods using the same discriminator architecture? (2) While the novel generalization bound and algorithm are both novel. The theory doesn't explain why ListDA is better than ItemDA. Personally, I do not see this as an issue.

Questions

I find the statement in line 141 confusing (even after looking at Appendix A.2). I would think that item-level invariance implies list-level invariance. For example, consider the case where $g$ was trained to output a domain-invariant feature representation of query-document pairs (at the item level). The ranker $h$ operates on these domain invariant features to form a list. How could the resulting list be different between the domains, if the ranker is constant and the output of $g$ is domain-invariant? Perhaps it would be helpful to elaborate on this point further in the paper.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This paper focuses on domain adaptation in the context of ranking problems. The authors propose a novel approach called list-level alignment to learn domain-invariant representations for ranking tasks. They demonstrate the benefits of this approach through theoretical analysis and empirical experiments. Existing domain adaptation methods for ranking, particularly invariant representation learning, have been sporadically used and lack theoretical guarantees. Given the limitations of previous approaches, the authors aim to address the conceptual discrepancy between item-level alignment and the intrinsic list structure of ranking problems. They propose list-level alignment as a solution to learn higher-level invariant representations on rank lists. This approach not only provides a theoretical foundation for domain adaptation in ranking but also achieves better empirical performance.

Strengths

I think this problem setting is interesting and practical. I have the following views on this paper's strengths part. From the perspective of problem and method, this paper addresses the challenge of training ranking models in domains with limited annotated data. It introduces domain adaptation as a transfer learning technique and highlights the need for invariant representation learning in the ranking literature. The authors propose a new approach called \list-level alignment\ that preserves the list structure of ranking problems and achieves higher-level invariant representations on the lists. The benefits of this approach include the establishment of the first domain adaptation generalization bound for ranking and better empirical performance on unsupervised domain adaptation tasks. Experimentally, the authors conducted experiments on unsupervised domain adaptation tasks to evaluate the performance of their proposed approach, list-level alignment. The proposed approach achieved better empirical performance on unsupervised domain adaptation tasks compared to existing methods. The approach also established the first domain adaptation generalization bound for ranking, providing theoretical guarantees for its implementation.

Weaknesses

I have the following views on this paper's weakness part. - The paper acknowledges that existing domain adaptation methods for ranking lack theoretical guarantees, and it claims to have theoretical foundations for its proposed approach of list-level alignment. It's already a good contribution. However, the contribution of this paper in the theoretical aspect may be considered weak, since the contribution of this paper can be seen as an extension of existing domain adaptation theorems[1] applied to the ranking problem. Correct me if I misunderstand it. - One curious aspect here is that while the paper discusses domain adaptation (DA) in the ranking problem, the experiments are conducted on reranking. In the context of reranking, which involves reordering a list of candidate documents generated by an initial retrieval model in response to a search query. However, it does not directly address the fundamental ranking problem. This limitation of the paper should be further discussed to provide clarity. [1] Ben-David, Shai, et al. "Analysis of representations for domain adaptation." Advances in neural information processing systems 19 (2006).

Questions

In my first point at my weakness part, if I misunderstand the theoretical contributions, the author could correct me.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, it's properly stated in the paper.

Summary

This paper proposes to learn domain-invariant representations for ranking. In contrast to prior works that typically consider item-level alignment, they introduce the concept of list-level alignment to learn higher-level invariant representations on the lists. The domain adaptation generalization bound is explicitly established and extensive experiments on benchmark datasets reveal the effectiveness of the proposed approach. Post rebuttal: my major concerns have been adequately addressed, and thus I've decided to change my score from 4 to 5.

Strengths

- Domain adaptation for ranking problems is significant but receives scant attention. - The paper in general is well-motivated and presents major challenges that need to be overcome to address DA for ranking. A new paradigm termed list-level alignment is proposed. - They evaluate and ablate the method in multiple benchmark datasets.

Weaknesses

- The proposed alignment relies on the traditional domain adversarial training, offering no significant advancements. What are the differences between the proposed alignment and prior adversarial alignment, and why the proposed list-level alignment is preferable in the context of ranking problems? The primary distinctions between list-level alignment and other paradigms are not well-articulated. - According to the reported results, the proposed ListDA shows marginal improvements over baseline methods in terms of MAP. - The connection between the theory and the proposed method is not sufficiently clear and precise, leaving readers uncertain about their relationship.

Questions

Please refer to the weakness.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Summary

This paper proposes a novel approach for domain adaptation on list ranking problems. In significant contrast with previous works, they adopt aligning list-level representations directly instead of item-level representations, and argue for its relevance in improving on list-level metrics pertinent to the problem of ranking. They also devise a theoretical bound for the target error in terms of source error, a joint hypothesis error and the list-level domain divergence, under suitable assumptions. Results on passage re-ranking task shows their strength in practical applications.

Strengths

- The paper is very well written, easy to follow and extremely well presented. - The idea of extending UDA approaches to list-level re-ranking problems is interesting and highly relevant. The ideas discussed in the approach might be relevant to problems beyond ranking (like object detection in computer vision which also uses unique evaluation metrics). - The empirical results on the passage re-ranking task is strong. - The simple of idea of using a transformer discriminator to allow permutation in-variance is impressive.

Weaknesses

- Few choices, like the use of transformer discriminator instead of MLP, wasn't evaluated or ablated empirically, although it makes sense intuitively. - The authors might also consider showing results on diverse ranking tasks beyond passage re-ranking, possibly in tasks like fine-grained computer vision as in [1], although I agree that this is much beyond the scope of this work. [1] Wang, Xinshao, et al. "Ranked list loss for deep metric learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

Questions

Is there a typo in L98? Since both $\mathcal{L}_\text{rank}$ and $D(\mu_s,\mu_t)$ need to be minimized according L100-102, but the negative sign in between does the opposite.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The work is self-sufficient, and presents good evaluation to convey the effectiveness of the proposed method. Although, I feel that the scope of this or a follow-up work could be broadened a bit, to also incorporate problems beyond passage ranking into the evaluation.

Response

I thank the authors for addressing my questions, and will keep my rating unchanged.