Discovering Intrinsic Spatial-Temporal Logic Rules to Explain Human Actions

Summary

This paper proposes a model for inferring rules based on observation trajectories of (entity, time, location), with the aim of maximizing the probabilities of certain "events". The aim is to predict the next events based on past trajectories, conditioned on the latent "rules" which need to be marginalized over. But this is intractable, so an "encoder" or "recognition" network is used, similar to the logic of variational autoencoder. In the encoder, a transformer is used to define the probability distribution over possible rules based on input trajectories, and a decoder can sample these rules to generate the next event. The model is used for trajectory prediction tasks on two datasets with better results than existing trajectory prediction model. Examples of rules learnt are also given in case of one dataset.

Strengths

1) A nice framework is proposed for representation of spatio-temporal action rules based on logic 2) The "rules" are expressed as a latent variable model, where inference can be done using E-M algorithm applied to an encoder-decoder model 3) Strong results are shown on trajectory prediction tasks, sample rules are also shown In my opinion, the rule definition and rule inference framework is quite general and can find wider applications than the experiments described in the paper.

Weaknesses

It is not very clear how exactly the goal states are estimated (eg. in Fig 3), or how the future trajectories are generated (Fig 3 and experiments), I get the broad idea, but precision is missing.

Questions

1) Can we have an experiment on synthetic setting with ground truth data generated according to specific rules (eg. like a cellular automata), and then show if the proposed model can recover the rules? 2) Please add an algorithm to explain how the trajectory prediction is carried out, step-by-step. 3) What can be other applications of this framework beyond trajectory prediction? Also, can the "rules" here be related to "policies" to choose actions as in Reinforcement Learning? 4) Can the rule probability distribution be modelled by a cheaper model than Transformer?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

NA

Summary

This paper presents a novel approach for human trajectory prediction, introducing a learnable rule-based framework that combines rule generation/reasoning and EM optimization. Unlike previous works in the field, this framework utilizes a neural rule generator to generate rules and treats them as latent variables. Additionally, it selects the top k rules from the entire rule set to predict future trajectories. The experimental results on two datasets demonstrate the effectiveness of the proposed framework.

Strengths

- The paper is well-structured and written in a clear manner, making it easy to follow. The authors have used appropriate notation and included equations where necessary, enhancing the readability of the paper. - The detailed analysis and visualizations presented in sections 5.4-5.6 and the ones in the supplementary materials contribute to a better understanding of the proposed method. I found them enjoyable and informative to read. - The paper introduces a novel framework that incorporates separate steps for rule generation and logic reasoning. This approach is original and reasonable for the given task. - The proposed framework demonstrates improved results on the Stanford Drone Dataset and NBA SportVU dataset. - The authors have put effort into providing numerous illustrative examples that showcase the behavior of the proposed framework. The explanation of logic rules in the NBA dataset (Figure 5) is particularly interesting.

Weaknesses

- The predicates are required to be manually defined and these predicates may need to be redefined when applied to new scenarios, such as transitioning from 2D to 3D scenarios. - ETH/UCY benchmarks have been widely used for benchmarking short-term trajectory prediction but missing from this paper.

Questions

- Regarding the latent rule space, what is the dimension of the latent space z? - I have reservations about fully accepting the argument presented in L207 that "our method can discover more principles-like complex rules." The generation of more rules for complex conditions necessitates a larger latent space (higher dimension), which can pose challenges in generating high-quality rules. - In L207-L208, it is mentioned that "For each query, we aim to identify top K rules $z_I$ from all generated rules $\hat{z}$," with K set to 5 in the experiments. Not all movements require up to K rules. Some movements may be simple and only require 1-2 rules (as seen in the second example in Figure 2). Is it possible to automatically determine the number of rules based on their weights?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have provided a discussion of the limitations in Section 6.

Summary

The paper proses a method for learning spatio-temporal logic rules to explain human actions, utilizing the EM framework. The method results are more easy interpretable by humans, thanks to the logic rules. The method beats some state-of-the-art methods on two real-world motion prediction datasets.

Strengths

The method is well-motived, i.e. that human actions are driven by intention as well as social and environmental factors. The explainability of the model is impressive and well-visualized.

Weaknesses

My major concern with this work is the need for dataset-depended actions. Biasing the model with additional labels, which are not provided to other methods, allows it to spend more capacity on forecasting, effectively producing an unfair advantage. It would have been better if the authors provide an approach that can automatically extract the actions, without human pre-selection. While the general intention is well-motivated, the method overview and method motivation in the introduction section is not well described, making it difficult to get a first good understanding of what the method tries to achieve and how this is done. For example, the authors explain that “[] logic rules present a compact knowledge representation” (L19) but they do not further explain what “logic” they are using and how it is defined. In the method section a lot of important details are missing, making it difficult to follow: * In the Equations in L104.5 and L113.5 the authors utilize variable v and v’ without introducing what this variable means. The authors should explain the variable appropriately. * In L122 the authors introduce various new object types “person”, “block” and “key” which are not explained. * In L117 the paper says that the binary predicated values can be softened into “probabilistic” ones using kernel functions without details or reference to other parts of the paper. This makes it particularly difficult to understand as the paper has not yet revealed how the logic rules are applied to obtain the predictions. Spatio-temporal predicates (L104) contain the time and space dimension. This necessitates that both time and area must be discretized. The discretization rate would be an important hyper-parameter and should be ablated accordingly. Why do the authors not follow the experimental setup in [23] and also evaluate on the ETH-UCY dataset? Can the dataset not be labeled with actions? If so: why? What makes the dataset difficult for this method setup? The organization of the experiments section is not convincing: while the authors conducted a lot of experiments they should have added them into the main paper. **Minor issues**: The authors claim their models “superior interpretability” (L15) but do not properly explained in text (only attempted in Figure 5). This however should be explained briefly. Equations should be numbered to make referencing them easier

Questions

It seems that using the full 3D human pose would be a much better indicator for human actions, interactions with objects, and interactions with other persons. Why did the authors use only trajectory data? For example, the Haggling dataset [a] contains triadic interactions under a well-defined social protocol which could have been utilized. L94: “Consider a set of objects denoted as C”: does object mean humans? This should be explicitly stated if true. L96: could the authors elaborate how k is encoded? — [a] Liu, Zhenguang, et al. "Towards natural and accurate future motion prediction of humans and animals.” CVPR 2019.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The methods challenges are discussed to some degree.

Summary

This study presents a logic-informed, knowledge-driven modeling framework designed to predict and understand human movements, based on the analysis of their trajectories. It takes into account that human behaviors are commonly guided by intentions, desires, and spatial relationships with surrounding objects. The research integrates a set of spatial-temporal logic rules, derived from observational data, into the model, utilizing an expectation-maximization (EM) algorithm to infer model parameters and rule content. The performance of the model is evaluated using datasets of NBA basketball player movements and pedestrian trajectories, demonstrating high levels of interpretability, predictive accuracy, and efficacy in its results.

Strengths

- The objectives and motivation are well-articulated, offering a clear understanding of the reasons behind the research and the problem it aims to address. - The performance of the proposed model is outstanding when compared to other state-of-the-art methods signifying an advantage of the work. - The proposed method is based on theoretical principles (EM method). It's not a product of arbitrary choices, but rather a thoughtful construction based on the EM algorithm, which adds credibility to the model's outcomes.

Weaknesses

- If additional specifics regarding the architecture could be provided, such as the input and output formats utilized by the transformer, it would be incredibly beneficial for a deeper understanding of the method. - The evaluation of this work is currently based on only two datasets. If the authors could expand this to include results from more universally employed pedestrian trajectory prediction datasets (ETH [i] or UCY [ii] ), it would significantly strengthen the claim of the model's generalizability. [i] Improving data association by joint modeling of pedestrian trajectories and groupings ECCV10. [ii] Crowds by example. In Computer Graphics Forum.

Questions

- There is a typo in equation (3); the correct symbol should be $\varphi_f$ instead of $\phi_f$ . - The references for the datasets utilized in the experiments are currently absent and need to be included. - Where did you get the heatmap depicted in Fig3 (b) (d)?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

If my understanding is correct, the spatial-temporal logic rule space, encompassing both static spatial and dynamic spatial relations, needs to be explicitly defined. This implies that prior knowledge or assumptions about the environment and the interactions within it are required to structure and define these rules. Thus, it's not a process automatically derived from the data but requires human intervention and understanding to set up appropriately.