Summary
This paper introduces OnlineTAS, the first fully-supervised online framework for temporal action segmentation (TAS). The main contributions include:
1. A context-aware feature augmentation (CFA) module that incorporates an adaptive memory to enhance frame representations with temporal context.
2. An adaptive memory bank that accumulates short-term and long-term context information.
3. A post-processing technique for online boundary adjustment to mitigate over-segmentation.
4. State-of-the-art performance on three TAS benchmarks in an online setting.
Strengths
1. The CFA module and adaptive memory bank provide an innovative approach to incorporating temporal context in an online setting.
2. The proposed post-processing technique effectively mitigates the over-segmentation problem common in online TAS.
3. The authors demonstrate the effectiveness of their approach on three standard TAS benchmarks, showing consistent improvements over baselines.
Weaknesses
1. The paper lacks a thorough discussion of the computational complexity and resource requirements of OnlineTAS. For an online method intended for real-time applications, this is a critical aspect. The authors mention that their approach uses a "single-stage TCN" for efficiency, but don't provide concrete details on inference time, memory usage, or how these scale with video length. A comparison of computational requirements with offline methods and other online video understanding tasks would be beneficial. This analysis should include considerations for both the online and semi-online inference modes.
2. The paper presents quantitative results but lacks a qualitative error analysis. A detailed examination of failure cases could provide valuable insights into the limitations of the approach and guide future improvements. For instance, analyzing scenarios where OnlineTAS performs poorly compared to offline methods or where over-segmentation persists despite post-processing could be informative.
3. While the paper introduces an adaptive memory bank, there's limited discussion on how this approach scales to very long videos or continuous video streams. It's unclear how the method would perform in scenarios where the video length greatly exceeds the memory capacity, which is a likely scenario in many real-world applications.
4. For an online method, real-time performance is crucial. However, the paper doesn't provide metrics such as frames per second processing speed or end-to-end latency on standard hardware. This information is essential for assessing the practical applicability of the method in real-time scenarios.
Questions
The following questions are mostly related to weaknesses directly or indirectly:
1. Have you explored the impact of different feature extractors and different backbones on the performance of OnlineTAS?
2. Can you elaborate on how OnlineTAS might be adapted for weakly-supervised or unsupervised settings?
3. How sensitive is the method to the choice of hyperparameters, particularly the clip size and memory length?
4. Your post-processing technique seems effective in mitigating over-segmentation. Have you explored how this technique might be integrated into the model itself during training, rather than as a post-processing step?
5. The paper introduces both online and semi-online inference modes. How do these two modes compare in terms of latency and accuracy trade-offs? In what scenarios would one be preferable over the other?
Limitations
The authors acknowledge some limitations, such as the computational intensity of the offline segmentation model used as a backbone and the reliance on pre-extracted features. These factors could hinder real-time application of the framework. However, the discussion of limitations could be expanded to include potential challenges in adapting the method to more diverse datasets or real-world streaming scenarios.