Performance Scaling via Optimal Transport: Enabling Data Selection from Partially Revealed Sources

Summary

In this paper, the authors address the problem of data selection, acknowledging that complete data availability is often not possible, and data can only be obtained from specific providers. They recognize that different data sources may have varying impacts on the performance of the model, emphasizing the importance of optimizing the data selection strategy to enhance model training. To achieve this goal, the authors leverage optimal transport, validation data, the target model, and scaling law to allocate the selection budget effectively across different data sources. The experimental results presented in the paper validate the effectiveness of the proposed method. Pros: This paper is well-written, presenting a clear and logical flow of ideas, along with a comprehensive description of the method employed. The setting of the study is intriguing, considering the current landscape where data providers offer limited options for data access. Exploring the optimal combination within a constrained budget is a valuable contribution to both academia and industry research. Cons: 1. While the chosen setting is interesting, it may be somewhat impractical since scenarios where different data providers offer data for the same task with identical distribution are uncommon. The authors implicitly assume that data from all providers share the same distribution, which might weaken the persuasiveness of the results. It would be beneficial if the authors could present a real-world scenario to support their assumptions. 2. The assumption that the selection model has access to validation data with the same distribution as the testing data might not be practical in many cases. Often, the real testing data and its distribution are unknown, making this assumption less realistic. 3. Some of the figures presented in the paper are difficult to comprehend. For instance, Figure 5 lacks clarity in terms of color interpretation, as well as the meaning of the x and y axes. Overall, this paper offers a solid contribution to the field of data selection. Despite the presence of certain strong assumptions, the insights provided are valuable for enhancing the data selection process.

Strengths

This paper is well-written, presenting a clear and logical flow of ideas, along with a comprehensive description of the method employed. The setting of the study is intriguing, considering the current landscape where data providers offer limited options for data access. Exploring the optimal combination within a constrained budget is a valuable contribution to both academia and industry research.

Weaknesses

1. While the chosen setting is interesting, it may be somewhat impractical since scenarios where different data providers offer data for the same task with identical distribution are uncommon. The authors implicitly assume that data from all providers share the same distribution, which might weaken the persuasiveness of the results. It would be beneficial if the authors could present a real-world scenario to support their assumptions. 2. The assumption that the selection model has access to validation data with the same distribution as the testing data might not be practical in many cases. Often, the real testing data and its distribution are unknown, making this assumption less realistic. 3. Some of the figures presented in the paper are difficult to comprehend. For instance, Figure 5 lacks clarity in terms of color interpretation, as well as the meaning of the x and y axes.

Questions

None

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The assumptions are a bit strong and may degenerate the usefulness in practical setting.

Summary

Developing ML systems typically requires collecting data from multiple sources. A natural question is how much data to collect from each source. This paper proposes a two-stage estimator that (1) estimates the relationship between data set proportions vs validation loss using Optimal Transport and then (2) optimizes the proportion of each source to emphasize in a dataset given a fixed budget. They validate aspects of the framework on several settings to demonstrate improved performance from good data selection decisions.

Strengths

The topic is timely and of growing importance. The optimal transport perspective to data selection is novel and interesting.

Weaknesses

1. The paper makes some confusing and potentially incorrect statements regarding the related literature, particularly with power laws. For example, - Abstract: “these scaling functions are black-box …”. Neural scaling laws are very interpretive and give clear explanations on how data set size relates with model performance. More so, I do not see how the proposed work is less black-box than a neural scaling law. - The general learning curve assumed in this work is a logarithmic model of “performance $= -\alpha \log N + C$ where $\alpha, C$ are parameters to learn. While logarithmic learning curves have been used to model performance in the past, the current literature typically considers power laws. Moreover, the cited work in this part of the paper, [1], specifically discusses power laws and how they may be more effective than log learning curves (e.g., Fig 23 in [1]). While this paper is free to use a logarithmic model for ease of implementation or other design choices, the surrounding discussion is unclear about the motivation of the log learning curve as well as the implications of using log curves vs power laws in this setup. 2. The problem of understanding how much data to collect given a fixed budget or given a performance threshold from multiple sources has been studied in several recent prior works, for example [2, 3]. It would be important to differentiate this work from this prior literature (e.g., the OT framework). 3. The baselines considered in the numerical experiments are mostly simplified versions of the proposed method. Pertaining to the above 2 points, it would be important to validate the design decisions against more relevant related methods, for example, the data selection methods in [2, 3], or at the very least, a meaningful comparison against power law-based performance estimation strategies. 4. While the OT-based estimator in eq. (1) is interesting, it is not well-motivated in this paper. I appreciated the interpretation of $a_1$ and hope to read an interpretation of $a_0$ as well. Furthermore, while bounds on validation loss w.r.t. OT have been studied, how tight are these bounds? This is particularly important because OT is used as an estimator rather than a bound, suggesting either (i) that OT can very tightly bound validation loss or (ii) that the constant term $a_0$ is carrying a lot of the water in this bound. This is particularly interesting because $a_0$ is assumed to be independent of data set size. 5. If I understand correctly, in the numerical experiments, the initial pilot training data set is fixed to 55% of the full data set size. This generally makes the learning curve estimation problem significantly easier as we are well in the power law stage of the learning curve at this point. Moreover, this experiment setup weakens the motivation of having access to only a small pilot data subset. It would more interesting to consider meaningfully smaller data fractions. [1] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. [2] Tejero, Javier Gamazo, Martin S. Zinkernagel, Sebastian Wolf, Raphael Sznitman, and Pablo Márquez-Neila. "Full or Weak annotations? An adaptive strategy for budget-constrained annotation campaigns." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11381-11391. 2023. [3] Mahmood, Rafid, James Lucas, Jose M. Alvarez, Sanja Fidler, and Marc Law. "Optimizing data collection for machine learning." Advances in Neural Information Processing Systems 35 (2022): 29915-29928.

Questions

Please see the above weaknesses. Most importantly: 1. How does the log learning curve estimation framework in the proposed setup compare vs power law-based estimation strategies? Can we use a power law estimator rather than a log one? How would that affect the empirical performance? 2. Can you give better interpretation of the OT-based estimator? For example, it would be nice to read out an OT-based bound on validation loss and then show how the proposed estimator’s components capture the different elements of the bound. 3. How does the methodology compare against the related prior literature on estimating and optimizing data set sizes?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Limitations are reasonably discussed in the Appendix.

Summary

This paper considers the problem of predicting model performance (and subsequent data selection) under a partially revealed setting. The two challenges are estimating the right proportions as well as extrapolating to dataset scales beyond the observed scales. The paper proposes a two-stage approach called projektor: in the first stage, it predicts model performance as a simple function (either an affine transformation of a quadratic) of the optimal transport distance between target distribution D_val and the input distribution. In the second stage, the performance is extrapolated using ideas and functional forms from neural scaling laws. The paper also proposes a gradient-based method for data selection using the model performance estimator.

Strengths

- Novel technical solution to a well motivated problem. The proposed approach is well-motivated, interesting, and well presented. - Well written overall: motivates problem well (section 1), contextualize well within related work (section 2), clear set up (section 3),

Weaknesses

- The experiments are a bit disappointing in terms of their practicality. As far as I can tell, the data sources are just different subsets of distributions of fixed datasets (potentially unlabeled or mislabeled). Given the recent developments in the field (training language/diffusion models on large unfiltered data sources), I would have liked to see more experiments on more realistic/noisy data sources. Even for ImageNet, a better data source would be raw images from different online sources (e.g., Flickr),. Hence, it's a bit hard to judge the practical utility of the proposed approach (beyond the settings considered in the paper, which I am not sure are settings in practice where people need much better data selection strategies). - Section 5 (evaluation) is a bit hard to follow in terms of setup: what exactly are the datasets used, etc.

Questions

- In stage two, is the intuition that by fitting the two different scales N0 and N1 in stage 1, you eliminate the need to explicitly estimate the scaling parameters (alpha and C) in the scalings laws? - How are OT distances computed for images? Is it done in some feature space? which one?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Fine.

Summary

The paper considers the data acquisition setting where benefit to model performance from acquiring new streams of training data may be supported by the inspection of limited segments of a candidate corpus, such that one may wish to evaluate the benefit to model performance to support selection from a set of candidate corpi. Such forms of data acquisition evaluation for estimating model performance impact has much prior work, the first claimed novelty in this paper is associated with utilizing forms of optimal transport metrics to improve selecting ratios of data to be served from multiple candidate corpi. This estimate is benchmarked to suggest a resulting improvement to estimates of model performance in comparison to prior work, and then by extending the use of the derived estimates of an ideal ratio to another component of evaluation associated with data scaling laws there is another improvement to estimates of model performance.

Strengths

The abstract claims of significant improvements to computational costs of the application with use of this approach was suggestive of a material contribution but I had difficulty evaluating that further from the rest of the writeup. It is intuitive that for real world application in industry the prioritization of data streams is impactful to bottom line, such that scalable solutions to more easily allocate training compute would be impactful.

Weaknesses

The contributions of the paper associated with incorporation of an optimal transport metric on its own strikes me as much more of an iterative contribution rather than a significant one. I recognize this is common to the field, but I had a hard time getting comfort on the statistical significance of the benchmark findings - even a few more details on experimental setup could have possibly helped. I agree that the figures appear to demonstrate improvement but I don’t think the paper convinced me that it was anything more than suggestive.

Questions

In figure 5 do you have any theories as to why with increasing sample scales the method appears to increasingly underestimate model performance?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

R2. ASR(cont’d)/R3. Contributions

***(continued from the rebuttal)*** With **almost-precise predictions** of target model performance, **the optimal data selected based on the predictions is nearly as good as the best solution found by intensive grid search. [Results are visualized in the PDF attached to the rebuttal.]** We are compiling these results into a short section in Appendix, highlighting the promising usefulness and broad applications of the proposed methods. *** ### R3. Additional discussions on technical and conceptual contributions **On the technical front**, this work proposes an original framework that incorporates the information on data distance into the prediction of model performance, which **enables the construction of highly reliable predictors in a way that is substantially more efficient and scalable**, requiring orders of magnitude less training time. Compared to existing methods based on fitting **non-informative surrogates** for performance prediction–i.e., they predict the performance solely based on the size of data or its composition ratios (how much from each data source) while neglecting the information of the content of data., leveraging the addition of information from data, the resulting performance prediction tools proposed in this work **fundamentally prevent large deviations from overfitting higher-order parametric surrogates**. This property is especially favorable for industrial applications where algorithmic robustness is often a critical concern. For example, Figure 2 gives a direct visualization that the proposed methods achieve a much higher prediction accuracy on extrapolation tasks (predicting model performance on data compositions where training data does not directly cover), **showing a drastic reduction in large prediction errors compared to baselines.** Further, this work extends the state-of-the-art discoveries on scaling laws and adapts them to the context of data selection. A prominent challenge in scaling laws is that model performance for different data scales at different rates. Constructing a universal predictor for data from arbitrary combinations of sources using fixed scaling laws could lead to unsatisfactory results. This work threads the needle to **organically incorporate the performance predictor with the scaling laws in a parameter-free projection, allowing predicting model performance on any data composition and data quantity in a self-adaptive manner,** automatically fitting scaling parameters for the data based on the prediction tools. This combination enables a totally different type of tools to help inform machine learning practitioners in data selection. In contrast to conventional data acquisition schemes where the algorithm simply acquires data to maximize the utility function and returns the set of collected data, **the tools proposed in this work could visualize the entire landscape of model performance** for any data composition at any data quantity. The practitioner can then make tradeoffs on what type of data to acquire and by how much so that the objectives are best met. **On the conceptual front**, in recent years, distributional discrepancy measures have been popularly used as a proxy of validation performance. A considerable amount of literature has made implicit assumptions that distance implies model performance and subsequently built their works on top of such assumptions. It is largely left unexamined whether, or/and how well, these discrepancy measures correlate with the actual model performance in reality, posing a prominent conceptual gap in research. **This paper is the first to directly use data distance for the prediction of the empirical performance of the model trained on the data**, providing functional relationships that connect them together and successfully developing applications based on these functions. **This work sheds light on a new path that shall provide inspiration for future work on data selection, data valuation, performance prediction, etc.**

R4. Clarifications on Figure 5

### R4. Clarifications on Figure 5: Settings and Visualization **Figure 5 provides a direct visualization of the landscape of prediction errors in model performance** for all data compositions as the scale of data to be predicted grows. We consider the setting of 3 data sources with unlabeled samples from CIFAR-10. **We construct the proposed performance predictors within 1k samples** that are considered accessible to the practitioner. **We then predict model performance for all data compositions and on larger data scales (2-10k samples).** In Figure 5, X-Y axes represent the proportion of data from data sources #1 and #2 (consequentially, the proportion of data from data source #3 will be 100%-X-Y) and the **prediction errors are visualized in Z-axis** (horizontal plane represents zero error, **red spikes blue dips represent deviations from over-/under-prediction**). As the data scale gets larger, the performance predicted from the initial 1k samples becomes slightly less accurate, but **prediction error remains mostly within 5% even at the scale of 10k data** (10 times larger than the pilot data), **retaining its effectiveness and functionality.**

C1: “performance scaling” and “scaling laws”

> *"This paper proposes a two-stage estimator that (1) estimates the relationship between data set proportions vs validation loss using Optimal Transport and then (2) optimizes the proportion..."* (*Summary*) **Re:** The authors would like to thank the reviewer for providing detailed and in-depth feedback and comments on the manuscript. **We would like to first clarify a potential misperception and the concerning confusion it may have caused.** This work proposes a two-stage scheme for **performance prediction (Stage I) and projection onto large scales (Stage II)**, referred to as **"performance scaling with data composition"** and **"performance scaling with data quantity"** in the manuscript, **respectively.** **The first stage is to obtain a functional relationship between data composition from different sources and the resulting model performance, which is achieved leveraging Optimal Transport data distances.** Then the function can be used to construct a predictor for model performance on arbitrary data composition. **This stage is conducted on the small pilot dataset, and is referred to as "performance scaling with data composition" in the manuscript.** The conceptual contribution is to incorporate data distances (via Optimal Transport) into the representation of the functional relationship. Compared to previous works that rely on fitting non-informative surrogates, the construction of the OT-informed predictors is orders of magnitude faster (efficient and scalable) and the resulting prediction tools fundamentally prevent large deviations from overfitting high-order nonlinear surrogates (robust and reliable) **Then, the second stage is to project the prediction of model performance to target data scales (data quantity) that are much larger than that of the pilot data.** The technical contribution is to organically incorporate the performance predictor with the scaling laws in a parameter-free projection. **This stage is referred to as "performance scaling with data quantity" in the manuscript.** In previous works, a prominent challenge in scaling laws is that the model performance for different data would scale at different rates thus constructing a universal predictor for data from arbitrary combinations of sources using fixed scaling laws could lead to unsatisfactory results. The novel integrated performance projection proposed in this work allows for predicting model performance on any data composition and data quantity in a self-adaptive manner, automatically fitting scaling parameters for the data based on the prediction tools. So far, main technical challenges for the conceptual problem have been solved. The proposed pipeline with the two-stage scheme would provide a complete landscape of target model performance for any data composition on any scale (data quantity), helping inform practitioners in decision making. **Additionally, we showcase how this can help find the precise optimal operating point, which is formulated as an optimization problem** based on the predicted performance and solved via efficient gradient-based methods. This framework is highly extendable and allows the practitioner to make tradeoffs on what type of data to acquire and by how much so that the objectives are best met. In empirical studies, improved performance is also demonstrated, outperforming comparable baselines in many important aspects. The authors acknowledge the responsibility for the presentation of this work and appreciate having the chance to revise the manuscript accordingly for better readability. **Especially, the use of term "performance scaling" may be somewhat abused that causes unnecessary ambiguities in different contexts.** We are considering replacing the phrase “performance scaling with data composition” with “functional relationship between model performance and data composition”, and reserves the term “scaling” exclusively for the scaling laws w.r.t. data quantity (data scales). **The authors would appreciate it if the reviewer could recommend phrases that have better clarity in the context.**

C2: black-box scaling relationships

> *"...some confusing and potentially incorrect statements regarding the related literature, particularly with power laws. For example, Abstract: “these scaling functions are black-box …” "* (*Weaknesses: 1.1*) **Re:** The authors acknowledge the responsibility for the choice of phrases that may have caused unnecessary ambiguities under different contexts and apologize for the confusion it may have caused. **As discussed above, we may have abused the term "performance scaling" and used it for different meanings in contexts outside of scaling laws (w.r.t. data quantity/data scales).** In the manuscript, we refer to the functional relationship between data composition from different sources and the resulting model performance as "performance scaling with data composition" and refer to the projection of the predicted model performance onto larger data scales (data quantity) as "performance scaling with data quantity", where **the latter has been more commonly associated with the term "performance scaling".** **We agree with the reviewer that these scaling laws are intuitive and interpretable, and "black-box functions" are references to the use of non-informative surrogates in representations of the relationship between data composition from different sources and the resulting model performance ("performance scaling with data composition").** Non-informative surrogates–e.g., rational functions [a] predicts the performance solely based on the size of data or its composition ratios (how much from each data source) **while neglecting the information of the content of data.** High-order nonlinear functions are essentially black boxes as the **implication of their parameters becomes impossible to interpret.** For example, [a] predicts the model performance L as the following $L\_{\lambda}(p)=\frac{1}{\lambda\_{11}\cdot p\_1+\lambda\_{12}\cdot p\_2...+\lambda\_{1k}\cdot p\_k}+\frac{1}{\lambda\_{21}\cdot p\_1+\lambda\_{22}\cdot p\_2...+\lambda\_{2k}\cdot p\_k}...+\frac{1}{\lambda\_{k1}\cdot p\_1+\lambda\_{k2}\cdot p\_2...+\lambda\_{kk}\cdot p\_k}$ where k is the number of data sources, $p=\\{p_1, p_2... p_k\\}$ are the proportion of data from each data source, $\lambda\_{ij}$ are a number of $k^2$ parameters of the surrogate to be fitted. **With these inherently nonlinear rational functions being added together, there is no way to associate parameters $\lambda$ to the effects of data from each source or interpret how each data source contributes to the predicted model performance.** Again, the authors apologize for the ambiguity and the confusion it may have caused. We appreciate the reviewer for pointing these out to us and for the effort in helping improve the presentation of this work. > *[a] Tatsunori Hashimoto. Model performance scaling with multiple data sources. ICML, 2021.*

C3: log-linear vs. power law scaling laws

> *"... the current literature typically considers power laws... Moreover, the cited work in this part of the paper, [1], specifically discusses power laws and how they may be more effective than log learning curves..."* (*Weaknesses: 1.2*) **Re:** Thanks for the excellent comment. We are appreciative of your insights and pointing out our negligence in elaboration of equations. **In short, we are using power laws rather than log functions. It looks like log-linear relationships because we brought everything to log space for better numerical properties**–we took log operations on both sides of the equation and the power law in the original space became log-linear in the log space. In existing studies on scaling laws (e.g., [1]), the variable of interest is the validation loss, which is often the cross-entropy loss as in language tasks. In our paper, we center on directly predicting the target model "performance". For practitioners, it is typically the accuracy for classification tasks or MSE (mean squared error) for regression tasks. **Following the line of research on data selection/data acquisition, we use the residual error as our prediction variable**, which is (100%-accuracy) for classification tasks and MSE for regression tasks. The reduction of residual error is often exponential (e.g., accuracy 90%->99% with residual error 10%->1%), **and working in the log space often gives better numerical properties.** Then, when implementing the scaling laws, for the **power law relationship** in the original space given as > $L = a N^{-\gamma}$ **taking log operation on both sides, we have** > $\log L = \log (a N^{-\gamma}) = \log a - \gamma \log N$ **where $\log a$ and $-\gamma$ are the constants $C$ and $-\alpha$ in our expressions, $L$ and $\log L$ are the residual error in the original space and log space, respectively.** **We admit that we were unaware of the line of research studying directly using log-linear relationships to fit the scaling laws, thus not paying adequate attention to clearly stating the expressions and their derivations to distinguish them from alternative forms of scaling laws.** We appreciate the reviewer for providing us with additional information on current investigations on scaling laws (Figure 23 is especially helpful to us for a better understanding of their difference). The authors apologize for the confusion and will improve the presentation.

C4: Comparison to existing data selection methods

> *"...validate the design decisions against more relevant related methods… data selection methods in [2, 3], or … against power law-based performance estimation strategies."* (*Weaknesses: 2, 3*) **Re:** We appreciate the reviewer for providing additional references and we would be glad to discuss them. At the time of submission, we were aware of [3] and cited a related paper [4] while [2] was published after our submission. **These papers are indeed interesting and provide inspiration for the line of research on data selection/data acquisition, though, we would like to point out that their scope and target problems are different from ours and the conceptual and technical contributions are non-overlapping.** **Both of these works are built for iterative data acquisition**, where the data collector aims to acquire data in an adaptive manner and gradually improve the strategy until the target amount of data is collected or other end goals are reached (e.g., time). **These frameworks aim to collect the best set of data at the end, not to directly predict the target model performance on all data compositions from the beginning.** This is a different field of research. There are tasks where such settings align with the need (such as the image annotation task [2] is built for), but are distinct from the data market problem we set our work into. **We focus on one-shot decision processes–given a small amount of pilot data (e.g., samples), we aim to directly provide the complete landscape of target model performance for any data composition on any scale (data quantity) such that the practitioner can be informed to decide the data acquisition strategy.** This is also true for the industrial instances that this work intends to apply to. **For example, training large models typically cannot accommodate changing training data during the process**. For example, in state-of-the-art automatic speech recognition (ASR) tasks, a number of datasets are available consisting of many audio sources (e.g., accents, recording quality, environmental conditions), the practitioners are able to curate a validation dataset representing their target user groups and scenarios (e.g., North American market, home environment) that they hope their applications to perform best. Due to the large scale of samples, training the model each time is resource-intensive. Finding an ideal combination of data for training on the target tasks can be prohibitively expensive if conducted with manual searches. The tools proposed in this work would provide a complete landscape of target model performance for any data composition on any scale (data quantity), helping inform practitioners in deciding what type of data to acquire and by how much so that the objectives are best met. ***[We are adding this as a set of NEW experiments with additional results provided in the summarization rebuttal to all reviewers (R1).]*** **Iterative data acquisition frameworks are not designed to accurately predict target model performance from the beginning.** Both [2] and [3] use surrogate models to represent the relationship between utility function (e.g., model performance) and the amount and proportion of data acquired from each source. In these works, [2] uses Gaussian Process (GP) and [3] uses Kernel Density Estimation (KDE), **which are both non-informative surrogates**–i.e., they predict the performance solely based on the size of data or its composition ratios (how much from each data source) while neglecting the information of the content of data. **KDE is a nonparametric method and its predictions are simply interpolations smoothened by the kernel function, while GP’s predictions are based on the estimated mean performance of each data source and the covariance between each two data source pair, resembling quadratic predictors (the quadratic baselines in our work).** Both of these methods need to fit on the model performance from a substantial number of model training on different data to function properly, and in both [2] and [3], **these models rely on adaptive improvements of their accuracy during the iterative data acquisition process. On a small amount of data (such as the pilot dataset considered in our work), their initial estimates are inaccurate and not intended for making final selections.** **Data selection via performance prediction based on scaling laws considers *only* the data size and not its composition** (which data sources it is from or/and by how much). As far as we are aware, [5] (cited in our work) provides a benchmark result on this strategy ("datasize" baseline). **Given that its performance is subpar compared to other baselines which consider both data composition and data size, we omit it in this work.** > *[4] Mahmood, Rafid, et al. How much more data do i need? estimating requirements for downstream tasks. CVPR, 2022* > *[5] Tatsunori Hashimoto. Model performance scaling with multiple data sources. ICML, 2021*

C5: OT bounds and performance prediction

> *"... while bounds on validation loss w.r.t. OT have been studied, how tight are these bounds? ... suggesting either (i) that OT can very tightly bound validation loss or (ii) that the constant term is carrying a lot of the water in this bound."* (*Weaknesses: 4*) **Re:** This is also an excellent question. We would appreciate the chance to thoroughly explain it. Starting from the classic result on **Kantorovich-Rubinstein Duality** (KR-duality [a]) $W (p, q) = \inf\_{\pi∈\Pi(p,q)} E_{(x,y)∼\pi} [||x − y||\_{2}]= \sup\_{||h||_L\leq k} \left[ E\_{x∼p} [h(x)]− E\_{y∼q} [h(x)] \right]$ which gives that the 1-Wassersterin distance (Wasserstein distance is the distance defined by Optimal Transport) between two distributions p and q upper bounds the gap between the expected empirical performance of some model h trained on samples x from p and y from q, assuming the model h is k-Lipschnitz. **The bound is tight and is attained when the Lipschitz constant k is minimal everywhere for the model on the data manifold of interest. For modern machine learning problems with neural network models, training error can be reduced to near-zero and Wasserstein distance would provide a direct indicator of expected validation performance.** **Yet, in practice, the precise value of Lipschitz constant is hardly known in priori.** Besides, in practical problems, we do not have access to the actual representations of underlying distributions p and q and instead, we use empirical distributions from their samples as approximations to the underlying distributions. This introduces **<sample noises>** to the calculation of Wasserstein distance. Also, OT problems are solved numerically using the efficient Sinkhorn algorithm, which carries some small approximation error from entropy regularization and adds **<entropy bias>** to the computed Wasserstein distance. **Despite the bound being tight in principle, the empirical Wasserstein distance calculated from finite samples and entropy-penalized OT includes certain noises added to the true Wasserstein distance between underlying distributions. These noises are typically invariant for the same problem and only depend on the sample size.** Thus, we use an affine transformation to represent the relationship between empirical Wasserstein distance and model performance of interest, which is a natural and simple choice. The affine transformation takes 2 parameters that represent the slope and intercept, respectively. **We refer to this approach as "center-scaling"**, where we denote slope as "scaling ratio" and intercept as "centering constant", corresponding to fitting the terms in the above-described relationship. **Intuitively, the "scaling ratio" serves as an empirical estimation for the value of Lipschitz constant on the data manifold and the "centering constant" fits the invariant noises in empirical Wasserstein distance and aligns the predictor with the model performance.** These values are important in connecting the empirical performance of the model to the Wasserstein distance between training and validation data, but are impossible to be obtained analytically. **This work develops a new approach that demonstrates the plausibility of estimating these quantities empirically and successfully constructs predictors and develops applications based on these estimated relationships.** This work sheds light on a new path that shall provide inspiration for future work on data selection, data valuation, performance prediction, etc. We would compile this discussion into a section into the Appendix for the benefit of readers. Additionally, the technical pipeline of this work is built on the novel result of class-wise hierarchical OT distance first proposed in [b], which treats distances between labels as the OT distance between their features, where the analysis resembles the classic Kantorovich-Rubinstein Duality. For detailed derivations, please refer to [c] where comprehensive elaborations are provided in its Appendix. > *[a] David A Edwards. On the kantorovich–rubinstein theorem. Expositiones Mathematicae, 29(4):387–398, 2011.* > *[b] Alvarez-Melis, David, and Nicolo Fusi. "Geometric dataset distances via optimal transport." Advances in Neural Information Processing Systems 33 (2020): 21428-21439.* > *[c] Just, Hoang Anh, et al. "LAVA: Data Valuation without Pre-Specified Learning Algorithms." The Eleventh International Conference on Learning Representations. 2022.*

End of discussion period and thanks note.

Dear Reviewers and Area Chair, The authors express their sincere appreciation for your valuable time and efforts spent reviewing this paper and helping us improve it. Despite the end of author discussions, we strive to consistently improve the paper. Please don't hesitate to let us know anytime if you have any further comments or feedback. If there are questions from the discussion between the reviewers and the AC, we would be more than glad to know and could take the last few moments today to respond to them and help explain. Many thanks, Authors of Submission2041