Weaknesses
The paper was an enjoyable read, but I found the following potential weaknesses and questions. I am willing to raise my score once these issues are addressed.
**Lack of highly related baselines.**
The work is positioned as a variant of Early Classification of Time Series (ECTS), especially given the focus on “real-time processing” and early decision-making. However, there is no mention or evaluation of ECTS algorithms [1, 2, 3], making it difficult to understand how PrAViC compares to existing work. ECTS literature often employs loss functions that balance classification accuracy and timeliness, much like the approach taken in Eq. (5).
Several existing early classification models, such as video early classification [4], could be applied to the problem setting defined by the authors. Although many ECTS algorithms are feature-vector-based [5, 6, 7] and not intended for video stream per se, they can also be applied to the aggregated feature in Eq.(6) under PrAViC framework.
To justify the novelty and effectiveness of their loss function, the authors should compare PrAViC against these established methods. While the proposed framework demonstrates solid results across diverse real-world scenarios, evaluating against related baselines would strengthen its position.
[1] Xing+, Early prediction on time series: A nearest neighbor approach. IJCAI 2009
[2] Gupta+, Approaches and applications of early classification of time series: A review. IEEE TAI 2020.
[3] Mori+, Reliable early classification of time series based on discriminating the classes over time. DMKD 2016
[4] Ma+, Learning activity progression in lstms for activity detection and early detection. CVPR 2016
[5] Chen+, Early time series classification using TCN-transformer. ICCASIT 2022
[6] Hartvigsen+, Adaptive-halting policy network for early classification. KDD 2019
[7] Bilski+, CALIMERA: A new early time series classification method. Information Processing & Management 2023
**Lack of proper evaluation for online classification.**
While early exit or ECTS models should be evaluated on *both* classification performance and earliness (such as NET), some experiments report only accuracy (e.g., Table 1), or only PrAViC's NET (e.g., Table 2). Ideally, early exit models should not be evaluated based on a single point in the accuracy-NET space; rather, they should be assessed using a set of points that represent the tradeoff between accuracy and earliness (i.e., the Pareto front). Benchmarking the proposed model and the baselines across this tradeoff would provide a more comprehensive evaluation, showcasing how well the model balances early exits and accuracy, which is crucial for early classification tasks.
Additionally, given the real-world datasets used, which often suffer from class imbalance, accuracy may be misleading [8]. More appropriate metrics would be balanced accuracy or mean macro-averaged recall, which would offer a clearer picture of the model’s performance across all classes.
[8] Branco+, A survey of predictive modeling on imbalanced domains. ACM Comput. Surv 2016
**Potentially overstated theoretical contribution.**
The authors claim that PrAViC is a theoretically-based framework, contrasting it with "hand-devised methods." However, I found no formal theoretical guarantees such as convergence rates, statistical consistency, or classification-calibrated loss. If the "theoretically-based" claim stems solely from the thresholding mechanism using class probabilities, this risks misleading readers and undermining the significance of PrAViC’s novelty.
Furthermore, there’s no engagement with well-established literature on Sequential Hypothesis Testing [9, 10, 11], which already provides theoretical guarantees for similar problems. Including a comparison or discussion of this body of work would strengthen the theoretical claims of the paper.
Thus, to substantiate the claim of being "theoretically-based," the authors could strengthen their work by providing formal guarantees, similar to what is done in Sequential Probability Ratio Test (SPRT) [9]. For example, SPRT guarantees the minimal number of samples (or equivalently, exit time) under a predefined error rate, assuming certain statistical conditions. If PrAViC could offer a theoretical guarantee of exit time (or the balance between exit time and error rate) under similar assumptions, it would provide a strong theoretical foundation and significantly enhance the contribution of the paper. Such guarantees would better differentiate PrAViC from "hand-devised" methods and provide formal support for its early exit mechanism.
[9] Wald, Sequential tests of statistical hypotheses. Annals of Mathematical Statistics, 1945
[10] Tartakovsky. Asymptotic optimality of certain multihypothesis sequential tests: Non-i.i.d. case. Statistical Inference for Stochastic Process, 1998
[11] Miyagawa+, The power of log-sum-exp: Sequential density ratio matrix estimation for speed-accuracy optimization. ICML, 2021
**Inappropriate ablation study**
The ablation study is not conducted in the conventional sense. Typically, I would expect the ablation study to evaluate the contribution of individual components (e.g., removing batch normalization, disabling the loss term). However, the authors introduce a new model, CoX3D, and conduct new experiments, which is not in line with what is usually meant by an ablation study. This adds confusion and detracts from the clarity of the experimental analysis.
**Reproducibility concerns.**
The code for PrAViC is mentioned but not provided (there’s a broken link to GitHub), raising concerns about reproducibility. Furthermore, the authors report only scalar accuracy values, without error bars or statistical tests, making it difficult to evaluate the robustness and generalizability of their results. For a complex training process, such as "light retraining" of models, more details should be provided to ensure that others can replicate the experiments.
**Unclear PrAViC framework for multiclass classification.**
Section 3 formulates the problem in a binary classification setting, but it’s not clear how the framework extends to K-class classification (e.g., UCF101 with 101 classes). Are the authors treating it as a K+1-class problem (one background class and K event classes), or are they performing standard K-class classification without a background class? Clarifying this would enhance the understanding of the framework's flexibility in real-world, multiclass scenarios.
**Unclear motivation and novelty.**
The authors state that PrAViC is motivated by the need for "more generalizable solutions that can accommodate diverse forms of data." However, the terms "generalizable" and "diverse forms of data" are not clearly defined in the paper. It would be beneficial to clarify whether "generalizable" refers to performance across different datasets or some specific capability of PrAViC that makes it more adaptable than existing online classification methods.
Additionally, "diverse forms of data" seems to refer to the range of real-world datasets (video streams) used in the experiments. However, since the paper only addresses video classification, it would help to rephrase or elaborate on this claim to avoid confusion. Classical CNN-RNN architectures [12] already perform online video classification effectively, so it would also be useful to highlight the specific contributions PrAViC offers that go beyond these established methods.
[12] Karpathy+, Large-scale Video Classification with Convolutional Neural Networks, CVPR 2014
**Minor comments**
- l.063 “two publicly available datasets” → should be "four"?
- When I saw Fig.2 for the first time, I had no idea what does NE(V) stands for. I would add a definition or at least reference in the legend, but this may be a personal preference.
- The term "unified" is used repeatedly, but it’s unclear what aspect of PrAViC is unified compared to the literature.
- l.178 “BCE” loss is introduced without spelling out. I can tell it’s a binary cross entropy-loss, but please avoid letting a reader guess.
- l.203~ “In this example, the model terminates with a probability of 0.42 at the 1st step, a probability of 0.18 at the 3rd step, and a probability of 0.29 at the 4th step.” → should be 0.60 at the 3rd, and 0.89 at the 4th?
- Figure 6: The caption should specify which dataset is used (UCF101).
- Figure 6: Adding a horizontal line to show the baseline model performance would help clarify the comparison.