Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approach

Summary

This paper proposes a novel algorithm for return decomposition with causal treatment. To do reward redistribution, GRD uses factored representations to model the Markovian reward function and dynamics function.

Strengths

The writing is clear and easy to follow. It is interesting to see the visualization in section 6.4, especially Fig. 4.

Weaknesses

The technical contribution is somehow limited and the stronger experiments are expected.

Questions

1 The technical contribution of the paper seems to be limited. The major contribution is the learned generative model. However, the proposed framework is similar to [27], which also uses factored MDP, learns a mask on the factors, and optimizes the model in a generative way. Is it correct to say GRD equals [27] adds markovian reward functions? 2 As for the experiments in Fig.3, GRD does not seem to outperform existing methods by a large margin. Except Halfcheetah, swimmer and humanoid stand up, GRD seems to be rather close to the baselines. Maybe it will be better to further test on some environment with more obvious performance gap. 3 It is interesting to see some horizontal lines in Fig. 4(a), where some dimensions have definite contribution to dimensions #27:54. I think it will be helpful to provide illustrations to correspond this to the mujoco ground truth dynamics, which proves that GRD indeed learns the true and interpretable casual structures.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

N/A

Summary

This study introduces a novel approach, termed Generative Return Decomposition (GRD), to address a key challenge in reinforcement learning: identifying the state-action pairs that contribute to future, delayed rewards. While many methods redistribute rewards in a non-transparent manner, GRD offers a clear return decomposition by explicitly modeling the contributions of states and actions from a causal perspective. GRD works by first recognizing unobservable Markovian rewards and causal relations in the data generation process. Then, it leverages these to create a compact representation for policy training over the agent's most favorable state-space subset. The researchers provide theoretical proof of the identifiability of the Markovian reward function and underlying causal structure and models. Experimental data also reveal GRD's superior performance and interpretability compared to other methods. However, some limitations exist due to the assumptions made, such as the stationary nature of the reward function, which may not be applicable in dynamic or online RL scenarios.

Strengths

The paper excels in presenting Generative Return Decomposition (GRD), an innovative method that improves interpretability in reinforcement learning. GRD successfully addresses the identification of impactful state-action pairs for future rewards. Its effectiveness is supported by both theoretical evidence and practical experiments, demonstrating its superior performance over other existing methods. Moreover, the paper demonstrates a high degree of clarity and coherence, enabling smooth comprehension. Additionally, the explicit description of assumptions contributes to a more profound comprehension of the inherent strengths and weaknesses of the study.

Weaknesses

- The quality of the text in Figure 1 could be improved by removing the shadow around the text. The same applies to Figure 2. - Minor typos and errors that need to be edited for the next version of the paper. E.g. in line 172: "provide" -> "provides" - In Section 5, only one policy is considered, which is SAC. How about having the experiments run based on another policy optimization algorithm? What would be the differences in performance and results? - Regarding the last paragraph of Section 5, there are two possible scenarios to train the agent. - First, the generative model is learned while the policy is being updated, in an end-to-end paradigm. - Second, the generative model is first trained (and stays fixed thereafter), then the policy begins to be optimized. In either case, how would that affect the policy training and performance? And how the insights from the GRD interpretations would be changed? - In the experiment section, the visualizations for the learned causal structure are only provided for Ant. Please provide the same type of analysis for other environments. - Considering Figure 4, having GRD, how the agent's robustness and generalizability would be affected? For example, consider the case where there are some anomalies injected into the agent and environment interaction, more specifically changing some values from the state-space. If such anomalies target the features that are less important to the agent, then its performance should not be affected that much, right? If so, could you provide some results in this regard?

Questions

See Weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See Weaknesses.

Summary

Delayed reward in reinforcement learning is the major challenge in reinforcement learning. The return distribution technique is the direct way to resolve this issue while preserving policy. The existing works redistribute the returns in an uninterpretable manner. In this regard, this paper proposes a GRD which generates the Markovian rewards in delayed reward scenarios. GRD first checks the casual relations of state and actions and from a compact representation using causal generative model. The experiment results show that GRD outperforms the baselines and helps visualization.

Strengths

* Experiment results seem promising.

Weaknesses

* Explanation on line 345-352 is not sufficient and hard to understand. This experiment section is very important since authors insist that GRD give the interpretable structure of reward. * The methods have to construct casual inference which is only possible when all the states are exactly defined. If the number of states explodes, the parameters to learn casual structure would explode. If states and actions are given, we can construct the casual structure without learning with parameters. Minor * Equations are too messy and hard to understand. Use under bracket in equation 6. * Notations are not familiar. It is hard to understand C^{\cdot -> \cdot}, d^a , d^s (?). The authors have to redefine all the variables step-by-step to improve the presentation of this paper. * The arrow size is not consistent in Figure 2. The arrow to \hat{r}_3 is narrower. Also, arrow directions which come from R are weird.

Questions

* I could not understand the experiment results in l345-352.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

Yes

Summary

The paper introduces a new algorithm called Generative Return Decomposition (GRD) for return decomposition with causal treatment. GRD addresses the problem by modeling causal relationships among variables, providing advantages over flat representations. It specifies each state and action as a combination of constituent variables and considers causal relationships within the system. The algorithm utilizes a factored representation similar to Factored MDP, enabling the formation and identification of the Markovian reward function based on causality. Unlike previous approaches, GRD uses a graphical representation to determine the contribution of each dimension of state and action to the Markovian reward. It also explains and models the observed delayed return as a causal effect of the unobserved Markovian reward sequence. The framework of GRD visualizes the causal relationships among environmental variables. The paper proves the identifiability of the underlying generative process and introduces a component-wise learning approach for recovering the causal generative process and redistributing rewards. The learned parameters provide a minimal sufficient representation for policy training, aiding in the effectiveness and stability of policy learning. The main contributions of the paper include the reformulation of return decomposition with a graphical representation, the introduction of GRD for learning the causal generative process, and empirical experiments demonstrating the method's superiority over state-of-the-art approaches in robot tasks with sparse rewards.

Strengths

1 - Interpretability: Having interpretable reward redistribution is an advantage over non-interpretable methods. This can be used to diagnose the reason for failures policy optimization. 2 - Reduces the state dimensionality: A very nice side effect of learning causal masks using a dynamics models is that a policy can be learned using very few features of the state. This leads to simpler policies, which could be more robust.

Weaknesses

1 - Writing: The paper needs a lot of work in explaining the method. Especially section 4 and section 5.1. A figure showing how the causal masks are applied would be a good idea. I am willing to improve my score, if the method explanation is improved. 2 - Experiments: The experiments include only Mujoco tasks. It would be interesting to see how the method behaves on delayed reward Atari environments like Bowling. Missing Related work: [1] Modern hopfield networks for return decomposition for delayed rewards

Questions

1 - How do you decide which trajectory to store in the memory for training? 2 - Why is the dynamics model needed? The reward redistribution should be possible without the dynamics model.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

4 excellent

Limitations

Yes, the limitations have been addressed.

Summary

The paper addresses a major challenge in reinforcement learning: identifying which state-action pairs contribute to delayed future rewards. They propose a solution called "Return Decomposition" that redistributes rewards from observed sequences while maintaining policy invariance. Unlike other methods, their approach explicitly models state and action contributions from a causal perspective, making it interpretable. The authors introduce a framework called "Generative Return Decomposition (GRD)" for optimizing policies in scenarios with delayed rewards. GRD identifies unobservable Markovian rewards and causal relationships in the generative process. Using this causal generative model, GRD creates a compact representation to train policies efficiently. The paper proves the identifiability of the unobservable Markovian reward function and the underlying causal structure and causal models. Experimental results show that their method outperforms existing techniques, and visualizations demonstrate its interpretability. The source code for their approach is publicly available.

Strengths

- The authors provide theoretical proof for the identifiability of the unobservable Markovian reward function and the underlying causal structure. This solidifies the theoretical foundation and robustness of the model. - The GRD method outperforms state-of-the-art methods in experimental results across a range of tasks. This demonstrates its practical effectiveness and application potential. - Visualization of the learned causal structure and decomposed rewards contributes to the interpretability aspect, a valued characteristic in contemporary machine learning.

Weaknesses

I have no knowledge about reinforcement learning, and I don't understand why the system assigned me to review papers on this topic. Please disregard my review comments.

Questions

I have no knowledge about reinforcement learning, and I don't understand why the system assigned me to review papers on this topic. Please disregard my review comments.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

I have no knowledge about reinforcement learning, and I don't understand why the system assigned me to review papers on this topic. Please disregard my review comments.

Response to Authors

Thanks for your thorough response. In your response, you referred to the "attached PDF" a few times. By that, do you mean the submitted paper?

Response to Reviewer 6iFi

We appreciate the reviewer for the positive feedback and recognition of our work.