## **Motivation and usefulness**:
*Q. Why is this an important problem? And what the outcome of our method suggests for the practitioner to act.*
Thanks for noting the importance of the motivation and usefulness of the problem.
> Why is this an important problem?
Knowing what is missing from the sample is a crucial problem in statistics and machine learning. It is directly associated with the reliability/accuracy of the model; consider the case where previously unseen events are critical (e.g., medical diagnosis, software vulnerability detection). Thus, many studies have been conducted to estimate the properties of the underlying distribution from the sample [1-7]. **The missing mass [4-7] is the most fundamental property among them**; it is the probability mass of the unseen categories.
In machine learning, the **missing mass measures how representative the training data is of the unknown distribution**. If the missing mass is high, the training is not very representative, and a trained classifier is unlikely to predict the correct class. If we manually label training data, the missing mass also measures saturation. We may decide that the labeling effort has been sufficient and saturation has been reached when the missing mass is below a certain threshold.
While the probability of an observed event $x$ can at least be (over-)estimated using the empirical probability $\hat p_x^{Emp}=N_x/n$, the missing mass cannot be estimated directly since the total empirical probability mass over unobserved events is zero by definition, i.e., $0=\sum_{x\in X_0}\hat p_x^{Emp}$ where $X_0=\lbrace x|x\in X\wedge N_x=0 \rbrace$. Thus, existing methods, including the famous Good-Turing estimator, use the (frequencies of) frequencies $f_k$ to estimate the missing mass. In this paper, we 1) **precisely characterize the bias and the error of the well-known Good-Turing estimator in the multinomial distribution**, and 2) **propose a new distribution-free method** to, given only the sample, find the distribution-specific estimator with a **minimized mean-squared error (MSE) outperforming the existing methods**.
> What does the outcome of our method suggest for the practitioner to act?
***Tl;dr:*** The proposed method allows practitioners to determine whether additional data collection or labeling is necessary to address unseen components of the underlying distribution. Also, it ensures efficient use of resources for data collection while maintaining model reliability, not wasting resources on unnecessary data collection.
Our method provides practitioners with a reliable way to assess the representativeness of their data and identify gaps affecting model reliability. If unseen events are prevalent, practitioners can decide whether additional data collection or labeling is necessary to address them. Conversely, when the model is already reliable, unnecessary data collection can be avoided, enabling efficient use of resources. This is particularly valuable in domains like software vulnerability detection or medical diagnostics, where proactive risk management is critical, and in resource-constrained environments where data collection is expensive or time-consuming.
## **Meaning of distribution-free**
*Q. "I'm not sure if it is appropriate to call the proposed method a 'distribution-free' method."*
We follow the definition of distribution-free estimation in the established literature on the missing mass estimation problem: Since Good's formulation of the problem [5], missing mass estimation has been defined in the context of drawing N i.i.d. samples from a categorical population, where $p_1+\dots+p_S=1$ and $n_1+\dots+n_S=N$. Here, $p_i$ represents the probability of sampling species $i$, and $n_i$ is the number of occurrences of species $i$. In this context, following Good's terminology [5], we define a distribution-free estimator as one that makes no assumptions about $p$.
- [1] Chao, A. et al., Unveiling the species-rank abundance distribution by generalizing the Good-Turing sample coverage theory. Ecology, 2015
- [2] J. Acharya, et al., Optimal probability estimation with applications to prediction and classification. Proceedings of Machine Learning Research, 2013
- [3] Y. Hao & P. Li. Optimal prediction of the number of unseen species with multiplicity. Advances in Neural Information Processing Systems, 2020
- [4] I. J. Good & G. H. Toulmin. The number of new species, and the increase in population coverage, when a sample is increased. Biometrika, 1956
- [5] I. J. Good. The population frequencies of species and the estimation of population parameters. Biometrika, 1953
- [6] A. Painsky. Convergence guarantees for the good-turing estimator. Journal of Machine Learning Research, 2022
- [7] A. Orlitsky and A. T. Suresh. Competitive distribution estimation: Why is good-turing good. Advances in Neural Information Processing Systems, 2015