Distributional regression: CRPS-error bounds for model fitting, model selection and convex aggregation

Summary

The paper studies distributional regression, a statistical technique that models not just the conditional mean of the response variable given the predictors (as in traditional regression) but the entire conditional distribution. The learning task minimizes the risk function, measured by the Continuous Rank Probability Score (CRPS). The authors aim to estimate the error bounds for the expectation of the predictive distribution. They also investigate model selection and aggregation via CRPS minimization on a validation set.

Strengths

* Comprehensive Approach: The use of distributional regression provides a more holistic view of the data by modeling the entire conditional distribution, offering more insights compared to traditional regression methods. This work explores various aspects of this model and provides a thorough analysis. * Theoretical Contributions: The authors present a robust theoretical framework, including the estimation of error bounds for the expectation of the predictive distribution. This adds valuable understanding and reliability to the model's predictions. * Model Selection and Aggregation: The paper addresses practical aspects such as model selection and aggregation using CRPS minimization on a validation set, which is crucial for effectively applying the methodology in real-world scenarios.

Weaknesses

* Typos in the Text: The text should be checked again by the authors to remove the typos (e.g., line 88: "on can show" should be "one can show"). * Math Symbols and Abbreviations: Several math symbols or abbreviations are used in the text without prior definition, reducing the readability of the text. For instance, "EMOS" in the abstract or $\mathcal{G}$ in line 69. Additionally, some symbols are used multiple times with different meanings. For example, $\mathcal{D}$ in line 19 indicates the training set, while $\mathcal{D}$ in line 71 denotes a subset of the set of all probability measures on $\mathcal{R}$. It would be better to use symbols more carefully and ensure consistency throughout the text. * Comparison with Existing Methods: While the paper focuses on the advantages of the proposed CRPS-error bounds for distributional regression, it would be beneficial to include a comparison with existing techniques, both in terms of theoretical properties and empirical performance. Related works and available baselines have not been properly discussed in the paper. Including such comparisons would provide a clearer context for the contributions of the proposed method and its relative performance. * Complexity Analysis: A detailed complexity analysis is needed to understand the computational cost of the proposed method, especially in comparison with other techniques. * Numerical Analysis Clarity: The results presented in the numerical analysis should be clearer, with a more detailed explanation of what the numbers represent and how they support the claims made in the paper. The numerical analysis is not sufficient, and not enough sensitivity analysis has been provided. Only two small datasets have been used, and the simulation does not cover all aspects of the model. It would be better if the authors designed new experiments with different datasets and settings to comprehensively validate the proposed method. This would help demonstrate the robustness and applicability of the approach across various scenarios.

Questions

See Weaknesses.

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

The limitations have not been properly discussed in the paper.

Summary

This paper considers the problem of conditional distribution prediction. For covariate-response pair $(X, Y)$, the objective is to estimate the conditional distribution of $Y|X=x$ for all $x$. The paper provides concentration bounds for the empirical risk minimization (ERM) estimator with continuous rank probability score (CRPS) as loss. It also provides regret bounds for the model selection and model aggregation with CRPS minimization.

Strengths

This paper is well-structured and clearly written. The proofs are rigorous, and the notations are well-organized. It offers theoretical guarantees that will benefit future research involving the CRPS minimization technique.

Weaknesses

1. The proving technique for the theorems presented in the paper is pretty standard. The work lacks technical contribution. 2. In Theorem 1, the definition of $R$ which denotes the boundary of the parameter space $\Theta$ should be added in the main text. 3. The theorems have not provided insights for the algorithm design. The paper has not proposed new algorithms, but in line 262 it says "demonstrate the effectiveness of our proposed methods". 4. The entire experimental section does not utilize the theoretical guarantees established earlier. Additionally, there is a lack of experiments to validate the effectiveness and tightness of the theoretical bounds.

Questions

As given in the "Weakness" section.

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

As given in the "Weakness" section.

Summary

The paper considers the problem of distributional regression, i.e., learning the distribution of Y conditional on X. In particular, the distributional regression problem is formulated as an empirical risk minimization problem, where standard concentration techniques are applied to obtain non-asymptotic bounds on the excessive risks.

Strengths

1. The paper is well-written and easy to follow. 2. The question is well-motivated, and the technical development is solid.

Weaknesses

After formulating the problem as an empirical risk minimization problem, applying concentration techniques appears to be very standard --- the technical contribution of the current paper is relatively weak.

Questions

The current paper mainly considers the case of iid data, and correspondingly leverages concentration tools for iid data. I wonder if it could be generalized to adaptively collected data, since there are parallel concentration tools for such data.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The paper has discussed its limitation.

Summary

The paper provides theoretical guarantees for distributional regression, which aims to estimate the conditional distribution of a target random variable Y given covariates X. These theoretical guarantees hold when the regression is learned by minimizing a particular proper scoring rule, the Continuous Rank Probability Score (CRPS), in the presence of i.i.d. data (X,Y). Concentration bounds on the estimation error are provided for model fitting, model selection, and model averaging.

Strengths

1.) The paper gives perhaps the clearest presentation I've ever seen when reviewing for one of the top ML conferences. Its acceptance is almost warranted from this fact alone; there is significance in the exceedingly clear overview of distributional regression. 2.) The provided concentration bounds cover multiple modeling goals (model fitting, model selection, and model averaging) and a wide range of popular models for distributional regression (both parametric and non-parametric).

Weaknesses

1.) The experiments do not seem strongly related to the theory, other than that the experiments investigate distributional regression. I expected the theoretical results to play a stronger role here. Is there some way in which the theory guides the data analysis? If not, what is the purpose of including an experiment at all? If nothing else, the experiments would seem to provide an opportunity to perform sanity checks. For example, can the authors provide empirical evidence of consistency to the theoretical limiting values? 2.) Generalizing #1, the authors do not provide guidance on what their theoretical results can be used for.

Questions

1.) The authors write: "[...] distributional regression is way stronger and more challenging than standard regression, where only a point prediction for Y given X=x is provided which typically reduces to an estimation of the conditional expectation." I found this statement a bit puzzling. For instance, both frequentist and Bayesian GLM's would seem to estimate conditional distributions of a target given explanatory variables, rather than simply providing point estimates. Could the authors clarify? Editing notes: 1.) "Way stronger and more challenging" (line 20) is redundant, and the first phrase is too colloquial. Streamline to "more challenging" or perhaps "much more challenging". 2.) It is unclear why "we will mostly use the CRPS for discrete predictive distributions" (line 87). Please clarify. 3.) Can the Wasserstein space `P_1(\R)` (defined between lines 83 and 84) be equivalently described as the set of probability distributions over `\R` with finite first moment? If so, this might be worth stating explicitly.

Rating

6

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

No limitations were provided.

Further comments by the reviewers?

Dear all reviewers: Can you please respond to the rebuttal as soon as possible? Your comments will be greatly appreciated. Many thanks, AC

Response to the authors' feedback

Thank you for your revisions and the effort you’ve put into improving the manuscript. The gap between theoretical claims and numerical experiments still exists. After careful consideration, I have decided to retain my original score.