ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation

Summary

This paper extends DiffAct to perform both action segmentation and action anticipation. An anticipative masking with a learnable mask token is proposed. Experiments are conducted on the three common benchmark datasets.

Strengths

1. The motivation for unifying action segmentation and action anticipation is reasonable, given the task similarity. It is also intuitive and reasonable to extend a generative framework from segmentation to anticipation, given the generative nature of anticipation. 2. The learnable mask token is interesting. 3. The ablation studies are relatively comprehensive. 4. Codes are provided in the supplementary.

Weaknesses

1. The major technical problem is that the proposed method assumes the ground truth video length is known for action anticipation at the test time. Without the ground truth video length T, the anticipative mask M^A can not be constructed during the inference time for anticipation. This is also shown in the code provided, where 'full_len' is input into the ddim inference function. This is problematic, conflicting with the goal of anticipation, and leading to incomparable experimental results. 2. The technical novelty is limited. The main contribution is extending the DiffAct method with an anticipative mask, while other modules are from existing methods. But given the contribution of unifying the segmentation and anticipation tasks, this is not a deciding factor for me. 3. It will be better to conduct experiments on Assembly101. Given the small data size and the saturated performance on GTEA, I would recommend a transition from GTEA to Assembly101 for this task. Minor: - In Figure 1c, what do the triangle and the circle mean? - In Table 1, some recent methods are missing, such as MVGA ICCV23 and RTK ICCV23.

Questions

NA

Rating

6

Confidence

1

Soundness

1

Presentation

3

Contribution

3

Limitations

The limitations were briefly discussed.

Summary

This paper proposes a new unified diffusion model called ActFusion, which solves the tasks of Time Action Segmentation (TAS) and Long Term Action Prediction (LTA) in a joint learning framework. To unify the two tasks, the model effectively handles the visible and invisible parts of the sequence during the training phase; The visible part is used for observed video frames, while the invisible part is used for future expectations. The experiment showed that the model achieved state-of-the-art performance in standard benchmark tests of 50 Salad, Breakfast, and GTEA.

Strengths

1. The design of the ActFusion model is novel, with its anticipative masking strategy and random masking method unifying the tasks of TAS and LTA. 2. The model enhances its performance on both tasks through mutual promotion of TAS and LTA. 3. The reported results in the paper surpass existing techniques on multiple evaluation metrics, demonstrating significant performance improvements.

Weaknesses

1. What is the difference between the Diffusion model used in the paper and the DiffAct model? They appear to be similar overall, despite using different Masks to unify the tasks of TAS and LTA. 2. The paper employs so many loss functions for model training; the authors should analyze the impact of different losses rather than simply exploring the effects of the encoder and decoder. 3. Since the denoising process of the diffusion model is a time-consuming task, I am interested in the computational efficiency of the model proposed in the paper. The authors are advised to provide information on the model's GFLOPs, parameter scale, and inference time. 4. The authors need to further improve the interpretation of the figures, such as what the circles and triangles in Figure 1(c) represent.

Questions

1. What is the difference between the Diffusion model used in the paper and the DiffAct model? They appear to be similar overall, despite using different Masks to unify the tasks of TAS and LTA. 2. The paper employs so many loss functions for model training; the authors should analyze the impact of different losses rather than simply exploring the effects of the encoder and decoder. 3. Since the denoising process of the diffusion model is a time-consuming task, I am interested in the computational efficiency of the model proposed in the paper. The authors are advised to provide information on the model's GFLOPs, parameter scale, and inference time. 4. The authors need to further improve the interpretation of the figures, such as what the circles and triangles in Figure 1(c) represent.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

In summary, the authors have creatively used different Masks to unify the tasks of TAS and LTA with a single model and achieved significant results. However, compared to existing models, there does not seem to be much modification, highlighting a disadvantage in innovation.

Summary

The author introduce a unified diffusion model for temporal action segmentation (TAS) and long-term action anticipation (LTA), dubbed ActFusion, where a single model is jointly trained to address these two problems effectively. A new anticipative masking is presented for the effective unification of two tasks, along with random masking to learn intra-action relations. ActFusion achieves the state-of-the-art performance on both TAS and LTA, demonstrating the effectiveness of joint learning of two tasks across standard benchmark datasets, 50 Salads and Breakfast, and GTEA.

Strengths

* Originality * The paper presents a approach to integrating two popular vision tasks, temporal action segmentation and long-term action anticipation, within a unified model. This is the first time these two problems have been investigated together in a single framework, highlighting the originality of the research. * Clarity * The paper is clearly articulated and provides sufficient details. It offers an adequate background on diffusion model and how it is utilized. The inclusion of pseudo-code and actual code facilitates a better understanding for the reader. Additionally, the experimental settings are thoroughly described, which aids in the reproducibility of the results.

Weaknesses

* The methodology lacks novelty. The model architecture, loss function (Cross-Entropy Loss, Temporal Smoothness Loss, Boundary Alignment Loss) and mask strategy (no mask, relation mask, boundary mask) closely resemble those used in the Diffusion Action Segmentation (https://arxiv.org/pdf/2303.17959v2). The primary distinction in this work is only its extension to include the long-term action anticipation task, and the introduction of anticipative masking. * The performance improvement relative to other state-of-the-art works is marginal. On TAS task, when compared to DiffAct, this model shows an approximate 1-point improvement across various metrics (F1, edit score, frame-wise accuracy) on different benchmarks (50 Salads, Breakfast, GTEA). * The model consistently achieves sub-optimal results when assessed using frame-wise accuracy as the metric on TAS task. As the authors point out, this could potentially be addressed by employing reconstruction methods for masked features. I am eager to see how these adjustments could enhance the model's performance. * More ablation study for LTA task could be included, e.g., How important is past context for the models (\alpha) ? How far into the future can models predict (\beta)? Table 2 reports only a limited range of settings. A more thorough analysis on these aspects would be highly valuable.

Questions

* The approach is currently limited to predicting action labels within a closed set. To extend this work to predict open-set action labels, what can be done? * While "segmentation helps anticipation" is evident from table 3, "anticipation helps segmentation" is considerably less significant in table 4. What's the reason behind this? * If further training the model to reconstruct the original features from the masked features, would it also improve the LTA task? * Paper writing * What does circle and triangle means in figure 1(c)? * In table2, why there are multiple underlined values (suppose to be second-highest value) in each column?

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

* Limitation * The author notes the sub-optimal performance when using frame-wise accuracy to evaluate the TAS task and suggests a potential solution, though its effectiveness remains unproven. I am eager to see how these proposed adjustments might improve the model's performance. * Potential negative societal impact * There is no potential negative societal impact of this work.

Response

Thanks very much for the response. W1: I am not convinced. This is not an evaluation choice as argued by the authors. This is a test data leakage issue. For any machine learning model, you should not use any ground truth when obtaining predictions on test data. One possible solution might be using the mean length of training videos as the 'full_len' during testing. But I guess this will lower the results. As for previous methods mentioned by the authors, I did not check their code. Did other previous methods except for those mentioned also use this ground truth? Even if ground truth was used in some of previous codes, I would consider this as an issue to be fixed in the following works rather than a convention to be inherited. W2: I appreciate the novelty of the unification, but not the methodology. This is subjective though. W3 and minor have been addressed.