Scaling laws for learning with real and surrogate data

Summary

This work addresses the challenge of augmenting limited data with more accessible surrogate data to improve generalization. The authors proposed a weighted ERM approach for integrating surrogate data into training and analyzed its performance under various statistical models. It was shown that incorporating surrogate data provably reduces test error, even when the surrogate data is far from the original ones. The paper introduced a scaling law that predicts the optimal weighting scheme and the amount of surrogate data to use, supported by both theories and experiments across different domains.

Strengths

- The effect of weighted ERM on generalization is theoretically analyzed on various statistical models. - The discussion on the connection with the Stein paradox through the toy Gaussian mean estimation example is insightful and provides a nice intuition for why incorporating surrogate data can be beneficial. - The theoretical results are well-supported by experiments in different domains.

Weaknesses

Weighted ERM is a simple and arguably the most intuitive way to incorporate surrogate data. This is a strength rather than a weakness on its own. However, given the ubiquity of such methods, the discussion on related works could be more comprehensive. - For example, in addition to synthetic data from generative models, data augmentations also seem to fall into the category of surrogate data considered in the paper. The results on sample complexities of data augmentation methods could be relevant. - Another potentially related topic is distributionally robust optimization. Despite the different optimization problems, the idea of finding a near-optimal weight for different groups of samples shares a similar high-level goal. - Instead of "introducing weighted ERM", the contribution of this work seems to be more about providing a theoretical underpinning for the effectiveness of weighted ERM with an insight on the choice of weight. This could be made clearer in the paper.

Questions

- In Eq (4), is there any reason for using $\approx$ instead of $\asymp$? - In Theorem 1 and 2, how to interpret $\beta$ intuitively, the difficulty of the learning problem? - In the Sobolev class example (line 161), what's the particular reason for assuming $n=Q^d$? Is this essential or just for simplicity?

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

- Some limitations are listed in the Weaknesses section. - It would be helpful to provide more intuitions for Theorem 3 given the complexity of the notations and the distinct form of the result in the asymptotic regime (cf. the clear sample complexities in Theorems 1 and 2). Remark 3.3 on the connection with the scaling law provides some insights but doesn't seem to be clear enough.

Summary

This work investigates the effects of augmenting training datasets with lower quality data, under a weighted ERM scheme. It is shown that introducing this surrogate data to optimally weighted training can improve predictive performance on the original data distribution, as measured by test error, even when the surrogate data is unrelated to the dataset of interest. A corresponding scaling law is derived, giving insight into both the optimal weighting and the amount of surrogate data that can be used. Numerical examples are provided.

Strengths

The paper is generally well written. A substantial numerical study is provided.

Weaknesses

The theory of this paper is developed for a squared loss function. However, most of the numerics are done on classification tasks without mention of this discrepancy. This should be addressed, with justification provided if possible. The role of the regularizer is barely mentioned, however it appears to play a crucial role in the relatively clean form of the relevant quantities given in the result statements. Analysis of sensitivity of the results to the regularization parameters would seem important and be appreciated. Further, growth estimates on its spectrum do not match the usual ridge regression setup (i.e. setting $\Omega$ to be the identity) in a way that's independent of parameter dimension. This makes the assumption highly artificial, and somewhat redundant (compared to just using the largest eigenvalue), and impacts the statement of the theorems. Clear discussion on the rationale for a scaling law and its specific form is not given.

Questions

Can you please address the above weaknesses?

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

While the phenomena is framed as analogous to a regularization technique, I do not feel that adequate care is taken to ensure that non-expert readers are not encouraged to add junk data to their training regimes without adequate care being taken.

Summary

This study addresses the challenge of computational scenarios where true data are scarce, and surrogate data are available to assist in building statistical models. The authors provide novel theoretical and empirical insights demonstrating that training models with both true and surrogate data, with appropriated selected coefficient $\alpha$, could improve empirical risk reduction on test sets, even when the surrogate data is unrelated.

Strengths

- The scaling law claim is clearly presented and supported by theoretical results across various contexts, including Gaussian sequence model, non-parametric regression, low-dimensional asymptotic, and high-dimensional linear regression. This breadth of application suggests the claim's novelty and relevance. - The link established between training with surrogate data and the Stein paradox is both novel and intriguing, potentially influencing future work in data augmentation, multi-fidelity modeling, and out-of-distribution training. - Theoretical and empirical results are robust and well-articulated, lending significant credibility and depth to the findings.

Weaknesses

- The discussion is limited to certain loss functions, specifically $\lVert \theta_\ast - \theta\rVert^2$ and $\lVert f_\ast - f\rVert^2_{L^2}$. Extending these results to include more common loss functions like L-1/L-inf norms or classification-driven losses could broaden the applicability of the findings. - Some notations are unclear or undefined, which could hinder understanding for readers unfamiliar with the symbols or concepts. For example, the symbol $\asymp$ used in Line 152 and $\Lambda(\alpha)$ in Line 207 require definitions. - Some minor errors: - Line 50 introduces the test error $R_\text{test}$ but this subscript is not used in the following context, e.g. Equation (4). Please make sure the test error notations are consistent; - Line 226: $R_\ast=R(\hat\theta_{0, \infty}(0))$ should be $R_\ast=R(\hat\theta_{\infty, 0}(0))$

Questions

- Equation (4) appears to lack a variable that quantifies the "closeness" between the surrogate and true data, such as the $\gamma$ parameter defined in Theorem 3. Could the authors clarify the absence of such a metric in this model formulation? Note: I realize that $R_\text{su}^\text{ex}(\infty)$ can represent the deviation from surrogate data to true data (indirectly). But let's keep this in case others share similar questions.

Rating

9

Confidence

3

Soundness

4

Presentation

4

Contribution

4

Limitations

The limitation is included and no negative societal impact should be considered.

Summary

This is a technical report on predicting training loss for a model trained on a weighted combination of real (in-distribution) and surrogate (out-of-distribution) data. The report proposes a parametric function for predicting how the training loss scales with the amount of real and surrogate data. The report first conducts theoretical analysis with a number of standard models on stylized cases, such as Gaussian sequence model, high-dimensional linear regression, etc., and derivated conditions for the proposed scaling relationship to hold. The report conducts empirical validation of the proposed scaling law on a range of classic tasks such as Sentiment analysis for the IMDB dataset, ResNet for CIFAR-10 classification, etc., and shows the training loss consistent with the proposed scaling function.

Strengths

The paper adopts the format of a technical report rather than a typical research paper. The structure of the report is clear. The problem is clearly introduced and formulated at the beginning of the paper. Abundant theoretical analyses are provided and a range of models and cases are considered. Experiments are diverse.

Weaknesses

I'm not sure whether this technical report format is suitable for a NeurIPS paper. The lack of context or background is not a critical problem, though the lack of context may make it less accessible to broader audiences. What I found most concerning is the complete lack of comparison with existing works and a serious literature review. The practice of data augmentation or/and using synthetic data from generative models/simulation, etc. is prevalent in the field and has a long history. This also seems not also the first paper to propose the idea of weighted training with real and surrogate data. It is essential for any paper contributing to this topic to first 1. organize and list previous efforts on this topic; 2. introduce the current progress and research landscape; identify and justify there exists a valid research gap; 3. how this paper moves forward compared to previous efforts. All of these are missing from this paper. Section 1.3 simply listed a number of potentially relevant references without commenting on each one or explaining how they relate to or are different from this paper. It is not actually informative. --- The problem definition does not appear rigorous. Though problem formulation and notations come early on in the paper, many are not clearly introduced or defined. Some vague arguments mismatch with the math-heavy writing style of the paper. For example, in the abstract, the authors emphasized > "Integrating surrogate data can significantly reduce the test error on the original distribution. Surprisingly, this can happen even when the surrogate data is unrelated to the original ones. " The word 'unrelated' makes it a very strong argument. Surprisingly, I did not find it clearly defined in the main paper. Actually, I don't think learning with unrelated surrogate data will always reduce model loss. Imagine training the model with mislabeled data. It will only degrade the validation performance. My guess is what the authors are trying to say might be "training the model with not directly related data may also contribute positively due to regularization effect or knowledge transfer". But this is well-known to the community. So I am confused.

Questions

See Weaknesses.

Rating

3

Confidence

3

Soundness

2

Presentation

1

Contribution

2

Limitations

The work does not have a standalone discussion for its limitations.

Summary

In the context of linear regression, this paper proposes to complement training data with surrogate data (generated / synthesized, or from an unrelated task / domain). For an optimal combination of real and surrogate data, new scaling laws are derived which show that that surrogate data can can help reduce the text error.

Strengths

The paper is very well-written. The context and motivations are clearly presented. The paper makes nontrivial contribution to understanding commonplace practice in training large models in modern AI era: namely, the use of synthetic or surrogate data.

Weaknesses

NaN

Questions

- In practice, how is the optimal value of the weighting paramater $\alpha_*$ to be set ? Is there a consistent estimator for it which only depends on the input training data ? Without such a prescription, the proposed scheme might fall short of practical usefulness. - My understanding is that the proof of the main result uses the Gordon comparison theorem. If a nontrivial covariance matrix for the features $x$ is introduced, major modifications might be required, or an entirely different approach (e.g via RMT) might be required. I'd be glad if the the authors can comment on this. - I wonder what this work could tell us about the "model collapse" phenomenon [1,2,3], where a model trained on synthetic data eventually breaks down (becomes biased towards a trivial / useless estimator). [1] Shumailov et al. "The Curse of Recursion: Training on Generated Data Makes Models Forget" (2023) [2] Alemohammad et al. "Self-Consuming Generative Models Go MAD" (2023) [3] Dohmatob et al. "A Tale of Tails: Model Collapse as a Change of Scaling Laws" (ICML 2024)

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The setup considered is extremely: linear regression with Gaussian data with no covariance structure (i.e isotropic covariates)

request for clarification

Dear Author(s), I am curious about the following points. Could you please provide some comments? Thank you very much. 1. Would the theoretical analysis still hold if we were to replace the surrogate data loss term with an l2 penalty centered at θ** (whether carefully designed or arbitrarily selected)? If not, could you please elaborate on the challenges this would pose for the proof? If it would, considering that this substitution could significantly simplify the problem, could you provide insights on why the use of surrogate data remains essential? 2. I notice that Theorem 3 characterizes the concentration of the risk to R_test(α). By the way, could 'Λ(α)' possibly be a typo? Additionally, I didn't see a definition for R(θ). Would the concentration weaken or perhaps improve if α were to fall outside this interval [ε0, 1−ε0]?

Response

I thank the authors for their clarification. I do realize that these losses behave similarly, however in a paper such as this I think it should be addressed when presenting theory and should be explicitly mentioned in the paper. The authors have briefly commented regarding the regularization parameter, setting of Theorem 3.1, and my concerns about partitioners blindly using dummy data. However, they have not provided any indication on whether they intend to address these concerns in their paper, and if so, detailed discussion on how the changes to the document, its framing, and experiments will be made. Without this, I cannot adjust my score.