Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videos

Summary

The paper presents a novel approach to learning task graphs from procedural activities observed in egocentric videos. Task graphs represent the partial ordering of key-steps needed to complete a task and are crucial for developing intelligent agents that can assist users. The proposed method utilizes direct maximum likelihood optimization of edge weights, enabling gradient-based learning that can be integrated into neural network architectures. The study demonstrates significant improvements in task graph prediction accuracy and online mistake detection using datasets like CaptainCook4D, Assembly 101, and EPIC-Tent. The main motivation behind this work is twofold. Firstly, it aims to enhance the ability of intelligent systems to assist users in performing complex tasks by automatically learning flexible and accurate task graph representations from video observations, rather than relying on manually crafted procedures. Secondly, the authors seek to explicitly model task graphs to provide a more interpretable and human-understandable representation of procedural activities. The authors introduce two primary approaches for task graph learning: Direct Optimization (DO) and Task Graph Transformer (TGT). The DO method uses the Task Graph Maximum Likelihood (TGML) loss to directly optimize the adjacency matrix representing the task graph. This optimization is performed using gradient descent, making the process straightforward and effective. On the other hand, the TGT model employs a transformer encoder to process key-step text or video embeddings and predict the adjacency matrix from these embeddings. This model includes a regularization loss to maintain distinctiveness among embeddings, ensuring accurate and meaningful task graph generation.

Strengths

The main contributions of the paper are as follows: 1. **Approach:** This paper presents an new error detection approach via learning task graphs by directly optimizing a maximum likelihood objective. This approach enables gradient-based task graph learning, allowing task graph methods to be applied to the problem of mistake detection in programs. 2. **Performance:** The proposed methods achieve superior performance in task graph generation and mistake detection compared to existing methods. 3. **Code Availability**: The availability of code for replication enhances the study's transparency and usability.

Weaknesses

1. **Dependence on Labeled Key-Step Sequences:** The method proposed in this work relies on action recognition models or Ground-truth Key-Step Labels. On the one hand, this will bring additional performance overhead, and on the other hand, relying on other action recognition models may introduce noise when errors occur during recognition. It's uncertain whether it will also affect the results of mistake detection. More discussions on these issues are needed. 2. **Limitation in Versatility and Scalability:** In this approach, a new model (task graph) needs to be trained for each new sequence of actions in order to achieve mistake detection. This method to some extent limits the versatility and scalability of the model, as retraining is required for each new task. Additionally, it seems that the proposed method has no versatility on order-unrelated mistakes, such as technique/measurement errors, which are highlighted in latest datasets (e.g., HoloAssist-ICCV23[1], CaptainCook4D-ICML23[2], EgoPER-CVPR24[3]) 3. **Insufficient Qualitative Analysis**: The paper only provides one qualitative result on generated task graph in Figure 5. However, there is no qualitative analysis of the generated task graphs for detecting the mistakes. Also, it is suggested to provide discussions on the corner cases that may be challenging for the proposed method, which would bring insightful observations for the error detection community. [1] Wang, Xin, et al. "Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023. [2] Peddi, Rohith, et al. "CaptainCook4D: A dataset for understanding errors in procedural activities." arXiv preprint arXiv:2312.14556 (2023). [3] Lee, Shih-Po, et al. "Error detection in egocentric procedural task videos." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. Suggestions for authors: My main concerns are about: - The versatility of the proposed method for various types of mistakes. - The detailed qualitative analysis on generated graphs when used to detect mistakes. Please do read the weakness carefully, I would consider to raise my rating if my concerns are well addressed.

Questions

Please see Weaknesses.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes.

Summary

This paper introduced a learning-based task graph generation for procedural actions and mistake detection. Compared to previous approaches which usually consider natural language based descriptions, the proposed approach aims to use backward propagation to adaptively optimize procedural action sequences. To the reviewer, the introduced approach is much more flexible compared to other "hard-coded" language-based descriptions. In summary, this papers introduced a simple yet inspiring approach for graph generation.

Strengths

1. The proposed optimization based graph generation seems inspiring and novel. Unlike other existing hand-crafted methods, learning-based graph generation can be end-to-end trained with task model parameters. This has the potentials in significantly boosting model performances, since the graph is no longer hard-coded, which makes models able to automatically search for the optimal correlations. 2. The methodology is mostly clear and easy to follow. The authors well demonstrate how to establish and end-to-end optimize trainable graphs. 3. The experimental results are quite promising in the experiment sections, and the improvements are non-trivial and encouraging.

Weaknesses

1. The explanations of contrastive loss are somehow hard to follow. For typical contrastive learning, to push away or to pull close the distances always require positive/negative and anchor samples. The authors are expected to provide more detailed discussions to help readers understand. 2. In line 140, the authors make an assumption that no repetitive actions exists in sequences, without any further discussions. It is suggested that authors could provide more explanations on why they have this assumption and what impact this assumption causes to the proposed approach. 3. It is suggested that how the video and text embeddings are sent to the overall framework should be clearly demonstrated in the Figure 3. In the current submitted version, it is somehow confusing to the readers how the video features are considered into the training process.

Questions

Please refer to weakness for details.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations that the authors introduced are reasonable considering current submission, and they do not have any potential negative societal impact.

Summary

The paper proposes a differentiable loss function based on the maximum likelihood of generating task graphs from video features. The two proposed methods (DO and TGT) show strong performance, verifying their effectiveness in generating task graphs. Moreover, the predicted graphs are evaluated on downstream tasks to demonstrate the influence of explicit graph representations for general video understanding.

Strengths

In general, I like the paper from every perspective. It advances the progress of using task graphs for video understanding. - The paper is well-written and easy to understand. - The literature survey is comprehensive enough, which helps readers easily understand the motivation and the problem setting. - The experimental comparison is extensive enough, including a rich set of recent baselines that vary from different types of task graph generation. - The two proposed methods show strong performance on the evaluated datasets, verifying the effectiveness of the proposed differentiable task graph generation. - The generated explicit task graph representation is also evaluated on the downstream tasks. - The discussed limitations, e.g., the assumption of availability of key-step sequences, are insightful and are critical to the future development of task graphs for video understanding.

Weaknesses

- L140 describes that the authors separate sequences if the key-step repetition is available. I am wondering if this will limit the applications for some daily scenarios where a key step needs to be conducted many times, e.g., spreading peanut butter multiple times. - In L329, it would be nice to have a reference of those action recognition works that assume key-step sequences are available.

Questions

See weaknesses. My concern is the potential limitation for applications to long-form daily-life procedure activities where repetition is crucial.

Rating

8

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Yes, the authors have discussed the limitations of this work, which is insightful and can facilitate future works of task graphs for video understanding.