Summary
The paper "When are ensembles really effective?" discusses the question asked in the title: When are ensembles really effective? To do so, the authors introduce novel theoretical measures called the ensemble improvement rate and the disagreement-error ratio. The ensemble improvement rate measures the relative rate an ensemble improves over a single learner, whereas the disagreement-error ratio measures the expected disagreement between members of the ensemble scaled by the expected performance of each model. The authors relate both terms to each other through a series of inequalities, i.e. provide an upper and lower bound for the ensemble improvement rate wrt. to the disagreement-error ratio. Moreover, the authors introduce the concept of competent ensembles which more closely resemble ensembles we are seeing in practice: While we might construct ensembles with arbitrarily bad members, we usually see ensembles that, on average, have well-performing members. Last, the authors conclude the paper with an empirical investigation that shows that a) ensembles are indeed competent in practice and b) that the disagreement-error ratio is valuable in predicting if an ensemble might improve performance compared to a single model.
Strengths
- The paper is well-written and easy to follow. The authors introduce remarks where necessary and re-visit existing results from the literature if it adds valuable information to the paper.
- The authors introduce new theoretical concepts, argue in favor of them, and show their practical value in experiments. In addition, the authors also validate any assumptions they are making in their empirical investigation. Hence, the authors offer a very complete overview of their ideas, studying them from a theoretical and a practical point of view. This should be the gold standard for all papers introducing novel concepts.
- The introduced ensemble improvement rate and disagreement-error ratio seem to work well in practice and hence are valuable. Second, I think the idea of competent ensembles can be potentially impactful for future studies about ensembles, as it brings the theory about ensembles closer to what we see in practice
Weaknesses
- In some form, the paper presents many things we already know in a new form. For example, the disagreement-error ratio can be seen as some form of diversity that has been studied for a long time in the literature. Similarly, the concept of weak learnability and competent ensembles seem to be closely related to each other (see my question below). While the authors did a decent job relating their work to existing work, a deeper discussion here would be interstring
- There is a minor typo in the paper: Section 3 uses "K" as the number of classes, whereas in Section 4 "C" is used for the number of classes
Questions
- In the introduction, you present the following scenario: "In particular, consider the following practical scenario: a practitioner has trained a single (perhaps large and expensive) model, and would like to know whether they can expect significant gains in performance from training additional models and ensembling them" -- How can you answer this question with EIR and DER? As far as I can see, EIR and DER require already trained ensembles and explain their performance and not predict the performance of a yet untrained ensemble? In this sense, it is similar to the bias-variance decomposition that explains the performance of a model/ensemble post-hoc (i.e. after training) and, therefore might be used as a theoretical guidance tool but not necessarily as a prediction of what model/ensemble performs better or worse in a given scenario. Did I miss something here?
- Did you explore the relationship between the disagreement-error ratio and the bias-variance decomposition, and if so, what is it? I assume that given a suitable disagreement rate D, one could come up with a term that is close to the variance (i.e., diversity), but the normalization via the expected error rate this novel here.
Rating
9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Limitations
The authors empirically validate their assumption of competent ensembles that might limit the applicability of the presented theory. Moreover, they point out the unique case of Random Forests that arguably justifies a study in its own right. I agree with the authors that there do not seem to be any negative societal impacts from the theory presented in this paper.