Summary
This submission studies the problem of comparing multiple classifiers under multiple evaluation criteria. It presents the construction of the GSD-front, i.e., the set of GSD-undominated classifiers, and an empirical variant called the $\epsilon$-empirical GSD-front.
Then, theoretical aspects of the GSD-front and the $\epsilon$-empirical GSD-front, such as satistical consistency, and the inclusion concerning the respective Pareto-front, are investigated.
Two statistical tests, namely static GSD-test and dynamic static GSD-test, on whether a given classifier $C \in \mathcal{C}$ is an element of the true GSD-front are proposed. Both tests are shown to be valid level-$\alpha$ tests. A robustified static GSD-test is also proposed to deal with the setting where the available data sets are drawn under non-i.i.d.-scenarios.
An empirical study is conducted to assess the proposed tests. It covers 80 binary classification data sets from OpenML [70]. The set of classifiers consists of Support Vector Machine (SVM) with Random Forest (RF), Decision Tree (CART), Logistic Regression (LR), Generalized Linear Model with Elastic net (GLMNet), Extreme Gradient Boosting (xGBoost), and k-Nearest Neighbors (kNN). The set of evaluation metrics consists of predictive accuracy, computation time on the test data, and computation time on the training data. Empirical evidence suggests that the GSD-approach tends to be more contrastive, compared to Pareto-front and another test that aggregates the single-metric tests as in [19].
Another empirical study is conducted using 62 datasets from the Penn Machine Learning Benchmark (PMLB). The set of classifiers consists of rule ensembles of trees (CRE) [53], CART, RF, SVM with radial kernel, kNN and GLMNet. The set of evaluation metrics consists of classical accuracy (metric), ii) robustness of accuracy w.r.t. noisy features (ordinal), and iii) robustness of accuracy w.r.t. noisy classes (ordinal). The results provided by GSD-tests are discussed.
Strengths
S1: The submission is well-written and organized.
S2: Theoretical aspects of the GSD-front and GSD-tests are carefully investigated.
S3: The authors also take into account an interesting aspect that the different evaluation metrics may have different levels of impact. Non-i.i.d.-scenarios are also taken into account.
S4: Empirical evidence seems to suggest that GSD-tests can complement the existing statistical tests for comparing classifiers meaningfully.
Weaknesses
W1: The authors might consider mentioning more scenarios where comparing multiple classifiers under the multiple evaluation criteria is crucial. An example might be multi-label classification, where multiple evaluation metrics have been proposed to evaluate multi-label classifiers. The set of evaluation metrics can also be extended with others, such as training time, storage memory, and so on.
W2: The motivation to compare multiple classifiers using evidence on multiple data sets, which can come from very different application domains might need to be strengthened. For example, it would be not easy to convince practitioners in safety-critical applications, such as lung cancer detection to use some classifier that has shown to be promising given evidence aggregated from multiple domains.
W3: Additional recommendations for the end-user might be beneficial. For example, what should one do if the GSD-front contains multiple classifiers?
Questions
Q1: Could you elaborate more on the case where only one training data is taken into account? This is related to the comment in W2.
Q2: Regarding W3, should one randomly choose one element of the GSD-front when having to make predictions? Yet, one might consider doing ensemble learning on the set of promising classifiers. In safety-critical applications, such a strategy might lower the interpretability and explainability of the predictive system. Do you have any recommendations in such cases?
Q3: Could you also discuss the Pareto-front and results of the test that aggregates the single-metric tests as in [19] for the empirical study on PMLB data sets?
Limitations
Please refer to "Weaknesses" and "Questions" for detailed comments.