Summary
Building on the work of Fan et al. (2023), this paper addresses the challenge of computing prediction intervals in an unsupervised domain adaptation setting, where labeled samples are available from a related source domain, and unlabeled covariates are available for the target domain. The primary objective is to generate prediction intervals for the target domain that provide both an asymptotic coverage guarantee and minimal width. The authors also explore the aggregation of prediction intervals on the target domain under both covariate shift and domain shift scenarios. Theoretical guarantees, including finite sample concentration bounds on approximation errors and coverage guarantees, are presented. The proposed methods are illustrated and compared with the weighted split conformal prediction method using a real-world dataset.
Strengths
- The paper tackles the important and challenging problem of uncertainty quantification in the unsupervised domain adaptation setting.
- The authors propose algorithms to generate prediction intervals with both coverage and minimal width guarantees.
- Experimental results show that the proposed method generates prediction intervals with comparable coverage but smaller average width compared to an existing method.
Weaknesses
- The paper assumes familiarity with Fan et al. (2023), making it difficult for readers who have not read that work to understand some notations and concepts. Section 2.1 on problem formulation is particularly hard to read without prior knowledge.
- There is a lack of flow in the paper, with some explanations and definitions missing or unclear. For instance, terms like "valid prediction interval", and expressions such as U(x) and l(x) are not clearly explained.
- The authors write "This translates into solving the following optimization problem:" without any explanations.
- Another example is Line 234, "adding a slight delta to ensure coverage even when F is complex.". This is not clear. It is discussed in Fan et al (2023) in (2.4).
- Even if simple, I suggest adding important details about different expressions in the appendix to help the reader.
- The title mentions "optimal aggregation," but model aggregation is not the central focus of the paper. It is first discussed in Remark 4.2 and briefly mentioned in line 272.
- The paper should highlight the similarities and differences with Fan et al. (2023) in terms of contributions, both theoretical and empirical.
Also, the authors use different quantile levels for estimators compared to Fan et al. (2023) without explaining the reason for these differences. In Fan et al (2023), the following quantile levels are considered: 0.05, 0.35, 0.65, and 0.95, while the authors use 0.85, 0.95, 0.9, and 0.9.
- The statement that "the shape of the prediction band does not change much if we change the level of coverage" lacks clarity on scenarios where this might not be true, such as multimodal distributions (?).
- The discussion on lambda <=1 is confusing and needs improvement. The paper inconsistently explains the shrinkage factor and its implications. For example, the authors write "Therefore, it is not immediate whether the shrinkage factor is smaller than 1, i.e., whether we are indeed shrinking the confidence interval (lambda > 1 is undesirable, as it will widen finit , increasing the width of the prediction band)." and "Although ideally lambda <= 1, it is not immediately guaranteed as we use separate data for shrinking.". The authors also discuss it in Lemma 3.3. and Lemma 4.4.
- The term "shape of the prediction band" with prediction intervals can be confusing as it can have other meanings (see functional data). I suggest the authors clarify this in line 147.
- The paper ignores a significant amount of research on prediction interval estimation. I suggest discussing important related work.
- The limitations of the paper are not discussed. For example, focusing solely on intervals might have implications under multimodal distributions, where High-Density Regions (HDR) would be the smallest regions.
- The relationship between the proposed approach and conformal prediction is not clearly explained. The paper should discuss the guarantees provided by the proposed approach compared to conformal prediction, including the relevant theoretical guarantees and calibration aspects.
For example, the weighted conformal approach also splits the data and estimates a density ratio. What about the (weaker?) guarantees you provide? The following paper could also be useful: "On the expected size of conformal prediction sets".
Also, in conformal prediction, in addition to finite-sample marginal guarantees, the authors often provide asymptotic conditional guarantees. What can you say about this with your approach?
- The comparison with the conformal approach in the experiments might not be fair, as conformal prediction does not optimize for the smallest width. Additionally, reporting the interval score, which is a proper scoring rule for intervals, would be useful for evaluating the methods.
- Typos and clarifications:
- line 137: "remains same."
- line 154: "via by solving"
- line 453 "a.s. on source". On the source domain?
- line 470, "we with have"
- It is not "Leduox-Talagrand" but Ledoux-Talagrand".
- line 481, a minus sign is missing
- Line 490 to 492. It is not clear why tilde tau <= alpha.
- Line 182, what does \mathcal{F} - f* mean?
- About Theorem 3.4, we expect the performance (or convergence rates) to also depend on alpha. Where does this appear in the theoretical guarantees?