Activity Grammars for Temporal Action Segmentation

Summary

This paper proposes a grammar-based activity segmentation method. The authors proposed a grammar induction algorithm, as well as an improvement over the existing activity grammar parser, the Generalized Earley Parser (GEP). The proposed model shows improvements over prior works on both grammar induction and action segmentation.

Strengths

- The exploration of marrying neural-symbolic representations is an essential aspect of research, especially given the limited application of neural-symbolic methods for real-world problems. - The proposed grammar induction method does improve existing prior works and could potentially be beneficial for future research.

Weaknesses

- The experiments on activity segmentation are mainly compared with baselines that are not state-of-the-art. Though the ablations prove the improvement in grammar induction and parsing, the overall performance can not be justified by the current experiments (i.e., might need better results to justify the motivation of grammar-based activity understanding methods). - Another concern is the design of grammar. Given recent advances in language modeling and unsupervised grammar induction, the motivation of symbolic methods is not very clear or not shown by the experiments. - The overall formulation and algorithm design for the BEP follows from the GEP parser. - The notation used in this paper is error-prone and needs to be clearer, especially given that the grammar-based method largely depends on these notations. (e.g. L.146, should it be $a_i^{M}$).

Questions

See weakness section

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The authors have properly addressed the limitations.

Summary

This paper proposes a new grammar induction algorithm, an effective parser, and a grammar evaluation framework. They improve temporal action segmentation by extracting and handling context-free grammar with recursive rules. The assessment presents good generalization and discrimination capabilities of induced grammar. After reading the rebuttal, my concern about the second and third points are resolved. For the first point, the authors provide the SOTA table as requested. However, the performance of the proposed method is not convincing. Overall, given that this paper introduces a new framework for temporal action segmentation, I still consider the pros outweigh the cons and will keep my initial recommendation.

Strengths

1. KARI proposes a new grammar induction algorithm, an effective parser, and a grammar evaluation framework. They improve temporal action segmentation by extracting and handling context-free grammar with recursive rules. The assessment presents good generalization and discrimination capabilities of induced grammar. 2. KARI outperforms state-of-the-art models on two benchmarks, Breakfast, and 50 Salads, for the temporal action segmentation task. 3. The writing is good.

Weaknesses

1. The evaluation of MIF is based on breakfast and 50salads. Can the author provide results on more benchmarks? The SOTA table is missing in the paper. 2. The method is relatively new, but the details are mostly presented in the form of text. Some illustrations for the method's visual representation are missing. I think this is very beneficial to readers' understanding and the dissemination of the paper. 3. For the first row of MS-TCN results in Table 2, is better than KARI, but the author did not bold it. Also, the author should give an explanation why KARI doesn't work here.

Questions

See weaknesses. In the first row of Section 3.1, the citation is missing.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Limitations are not discussed in the paper.

Summary

This paper addresses the challenge of temporal action segmentation by introducing an activity grammar to guide neural predictions. The proposed approach involves a grammar induction algorithm (KARI) to extract a powerful context-free grammar from action sequence data. Additionally, an efficient generalized parser (BEP) transforms frame-level probabilities into a reliable sequence of actions based on the induced grammar. Experimental results on benchmark datasets show significant improvements in both performance and interpretability of temporal action segmentation.

Strengths

This paper is well written which demonstrates its motivation, methodology and experiments. Especially, the method of this paper is easy to follow and the provided visualization is a plus to understand the proposed algorithm. The idea of introducing the recursive rules is inspiring, which helps to identify the repetitions of actions and action phrases. The proposed grammar evaluation scheme demonstrates the effectiveness of the proposed KARI-induced activity grammar, achieving good recall while maintaining reasonable precision. The experimental results look promising, achieving good performance on various benchmarks.

Weaknesses

The benchmark is relatively small, which generates concerns about the scalability of the proposed method. Also, it’s good to know the computational cost of the proposed method to evaluate its scalability. The design of the proposed KARI is not well justified by using ablation study. It will be good to show the effectiveness of each component and its alternatives. Minor: Missing reference for ln 109 and 119.

Questions

Is the proposed method limited by the action type? For example, it could only work for well-structured activities.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

as is mentioned in previous sections.

Summary

This paper presents a grammar induction algorithm that takes as input sequences of frame-level predictions and outputs structured sequences of actions. The advantage of their approach is that it allows recursive rules, which enhances its generalization abilities.

Strengths

1. The method is more flexible than previously proposed grammar induction methods. 2. The results are better compared to baselines. 3. All the steps are explained in detail and most of them are well justified.

Weaknesses

1. The grammar explanation (section 3.2) is confusing. Especially the large amount of superindices and subindices representing different concepts. For example, what does E^{M(m,n)} mean? Or, in "t \in {b, a, k(m, n)}" what does the "b" represent? Probably simpler notation or a figure would help. I also did not understand what "each sub-string a_i^M starts with a key action and includes all the key actions in K" means. Do you start at a key action and do not close the sub-string until all key actions have been found? But then, I assume there will only be one such sub-string in the whole sequence. Is there any overlap between sub-string? Is it possible that a sequence does not contain any key action? 2. I do not think the synthetic activity grammars are a good setting to judge the quality of the proposed approach. Without delving into the details of the generated random grammars, it is unclear whether the biases introduced in the generation may make them easier to deal with for some kinds of methods than others. Real-world grammars do not have that problem. 3. There are no results showing the importance of the specific contribution. Did the authors identify cases where the inclusion of recursive rules made a difference? Minor weakness: a few typos and grammar mistakes: "due to the reason" -> "due to this reason" (lines 4, 22), "and also be applied" -> "and can also be applied" (line 29), "we proceeds" (line 137), "has" -> "have" (line 70), "other researches" (line 74), a few empty citations (lines 109, 119), etc.

Questions

- Would it be possible to combine this approach to perform end-to-end training with the frame-level predictions Y? This is, train the grammar and the temporal action segmentation network simultaneously. - Accuracy does not improve when using grammars (it actually decreases). Why is that the case? I would assume that having contextual grammatical information about future and past actions would improve the accuracy of the current action prediction. Do you think the accuracy would improve with a flexible/good enough grammar, or this is a limitation of using action grammars?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Limitations and broader impact are discussed in the appendix.

A gentle reminder

Dear reviewers, We'd like to thank again for your effort and time dedicated to our submission. We've addressed your concerns in our rebuttal, and it would be very helpful if you could give us any further thoughts and update your scores before the author-reviewer discussion phase ends. Your opinion would be invaluable to us in improving our work, and we would be glad to respond further to your questions. Thank you for your consideration. Best regards, Authors