ELDEN: Exploration via Local Dependencies

Summary

This work proposes `ELDEN, Exploration via Local DepENdencies`, a framework with an intrinsic reward that encourages the discovery of new interactions between entities such as the agent or objects that have some influence on each other. The method uses partial derivative of the learned dynamics to model the local dependencies between entities. The uncertainty of the predicted dependencies is used as an intrinsic reward. What distinguishes this from other related work, is that the focus for defining what makes a state interesting. Traditional approaches may focus on the specifics of how entities in an environment interact. In contrast, here interesting states are defined based on whether the entities *can* interact, without focusing on the specifics of how the interaction occurs. Specifically, the algorithm is biased towards exploring states where the relationships between entities are not well understood. To implement this, the authors train an ensemble of dynamics models. Within each model in the ensemble, local dependencies between objects are modeled using partial derivatives of state predictions with respect to the current state and action. Partial derivatives can capture how changes in state or action affect changes in predictions, which can be thought of as capturing local interactions. The uncertainty in local dependencies is then quantified as the variance across all the models in the ensemble where a high variance suggests high uncertainty. For learning the object interactions themselves, they propose using a `Causal Graphical Model` to represent the transition at a given time step 't'. CGM consists of three parts: `1. Nodes`: These represent the state at time step 't' (S_t), the action at time step 't' (A_t), and the state at the next time step 't+1' (S_t+1). `2. Directed Graph G`: This graph illustrates the global dependencies between the nodes. `3. Conditional Distribution 'p'`: For each state variable at the next time step (S_n_t+1), there is a conditional distribution 'p' which represents the probability of reaching that state given the current state and action. The assumption is that the transition probability `P(s_t+1 | st, at)` can be factorized into a product of conditional probabilities for each state variable at the next time step. Here, `Pa(v)` refers to the parent nodes of a node `v` in the causal graph `G`. This means that the probability of transitioning to a new state depends on the current state and action, and can be represented as a product of several smaller probabilities based on the causal relationships. Instead of using these global graph `G` which includes all possible dependencies, the authors use a local causal graph `G_t`. This Local Causal Graph Model is aimed at more accurately and efficiently representing the real-world scenarios by focusing on relevant dependencies at a given time rather than considering all possible interactions. This can potentially lead to more efficient learning and better generalization in reinforcement learning environments. ELDEN's main goal is to identify the dependencies that are locally active among the entities in the environment. In other words, for a specific state 'st' and action 'at', ELDEN tries to determine which entities in the environment have an active causal relationship with each other. This involves constructing a graph where the nodes represent entities, and the edges represent active causal relationships between them. This graph is specific to the given state and action (st, at), and is called the local causal graph 'Gt'. **Experiments** Experiments are conducted with the following baselines: `pCMI (point-wise conditional mutual information)`, `Attn (attention)`: Use the score between each entity pair computed by the attention modules inside the dynamics model to quantify local dependencies. `Vanilla PPO`. Presented tasks : `Kitchen Task` has a very high variance! ` Thawing Dynamics` curiosoty outperorms ELDEN. `CarWash` results also has high variance.

Strengths

- The ideas presented here are very interesting and novel. Creating an intrinsic reward based on local dependency, which aids exploration in RL, especially in environments with sparse rewards is novel.

Weaknesses

- Weak Experimental results. The results as they stand are not very convincing. That being said, these are hard problems so I understand the challenges in making these environments to work.

Questions

- For the PPO baseline this serves as a baseline with sparse reward? So it is not surprising that this baseline doesnt really work ?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

- Underwhelming experiments.

Summary

The paper proposes an intrinsic exploration reward that explicitly takes dependencies between different entities in the environment into account. Such an intrinsic reward helps sparse-reward RL in a few skill-based domains.

Strengths

- I find the idea of modeling dependency graphs between entities interesting and worth exploring. - The related work section is written especially well, making it very easy to understand how the proposed work is contextualized. - Explanation of the method is clearly written and easy to understand. - Results on given domains are promising.

Weaknesses

- The major weakness of the paper is in the evaluation. In particular, the domains that are used for evaluation all use high-level primitives (like goto object and pickup object) which results in relatively few steps in order to accomplish a goal. It's hard to argue that this setting is comparable to the difficulty of something like Franka Kitchen (low-level control in a kitchen environment with sparse reward). And especially given many moving parts are required to make this work, it would limit the applicability in my reading. It's even more unfortunate that this is not justified and only discussed in detail in Appendix C, given that it transforms a long-horizon sparse-reward problem (the typical understanding) into a relatively short-horizon problem. - With regards to the above, ELDEN is proposed as a standalone method, but in the most complex domain tested (Kitchen), an additional exploration method, PER, is required in order to make it work. - Too much is pushed to the appendix, for instance architectural details that are important to understand for evaluation (Attn Mask in section 4.1) are only discussed in Figure 4. Ablations for a method with many moving parts only show up in Appendix E. - All domains are discrete action, and the most convincing experiments (no PER required) are discrete state. The use of partial derivatives through Mixup is a little bit strange and I don't understand where the claim "dynamics models trained on such data... reflect local dependencies" is justified.

Questions

- How is the variance of the dependency graph computed (Algorithm 1)? - Is there a domain with low-level control in which ELDEN can be benchmarked?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Some limitations are addressed in the Conclusion, and more interesting ones are also discussed in Appendix F. In particular Appendix F makes the point that ELDEN is suboptimal in navigation environments, and thus that it is particularly scoped for settings with a big dependency graph.

Summary

This paper proposes a novel intrinsic reward called ELDEN for reinforcement learning. ELDEN encourages the discovery of new local dependencies between entities. Experiments on some robotic tasks are carried out to valid the idea.

Strengths

1. The idea is novel and interesting. 2. The usage of dynamics models to approximate causal dependency is interesting. 3. The method does not make any assumption of knowing groundtruth dependency. 4. The paper is well-written. The authors discussed their method are different from existing works.

Weaknesses

1. My concern is about the significance of proposed method. This work focuses on solving semantic-level tasks and assumes knowing underlying objects. Then, why should we use RL in this case? A better option is probably using large language models (LLM) to do the semantic reasoning directly. Another option is to use LLM to design exploration reward like [1]. I would like to see authors opnion on this. 2. In the experiments, two of the considered tasks are not solved (reach normalize score 1.0). The performance is not strong enough. 3. The authors can compare to more baselines. One important baseline can be RND. [1] Yuqing Du, Olivia Watkins, Zihan Wang, Cedric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, Jacob Andreas. Guiding Pretraining in Reinforcement Learning with Large Language Models. In ICML 2023.

Questions

1. What are the state-space and action-space of the used tasks (including the meanings and number of dimensions)? How long is the horizon for each task? 2. How does proposed method perform in image-based tasks?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors discussed their limitations and potential negative societal impact properly.

Summary

This paper proposes ELDEN, a method for intrinsic reward based on local dynamics dependencies between factored state variables. It learns an ensemble of factored dynamics models, and uses the magnitude of the partial derivatives to detect local dependencies between state variables, and uses the ensemble variance as an exploratory intrinsic reward. This method is tested across a variety of environments such as gridworlds and robotic control tasks, showing great accuracy in capturing local dependencies and exploratory performance.

Strengths

This paper presents a very clear and thorough test of the main ELDEN method. ELDEN itself is straightforward to understand, and section 4.1 does a convincing job of evaluating how well it detects local dependencies compared to reasonable baselines. Section 4.2 also has reasonable baselines, and clearly demonstrates the effectiveness of ELDEN. The ablations on mixup and the regularization are also well done. Overall, many interesting questions around ELDEN are covered by the paper, and the results are very clear.

Weaknesses

A common weakness, not specific to ELDEN, is of course the assumption of the factored state space. While this can be overcome with object-identifying representations, there is still the question of how effective ELDEN may be when paired with learned factored representations as opposed to ground truth. Hopefully future work can look in this direction. While it is mentioned that ELDEN is not meant to excel in all domains, an additional experiment in a more common domain such as the sparse reward tasks of DM Control Suite would be enlightening to see. Perhaps ELDEN would also work well, or it may not compared with the baselines; either way it would provide some additional insight into the limitations of ELDEN. ---- After Author Rebuttals ---- After reading other reviews and author rebuttals, I maintain my score. I think this paper has clearly shown when and where ELDEN can excel, and the limitations when ELDEN is not very effective.

Questions

It is mentioned that ELDEN can help with more complex, chained dependencies, however the basic method is still only looking at 1-step local dependencies. Why would one expect ELDEN to do better than other 1-step methods?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Are addressed appropriately in the paper.

Thank you for the response.

I would like to thank authors for the detailed response. For weakness 1, the authors point out that ``many environments are naturally hard to describe in language and are thus not well learned by LLMs''. I agree with this, but my concern is that if ELDEN is able to solve this kind of problem. For example, it is hard for LLM to solve fine-grained, low-level robotic locomotion or manipulation problems. However, ELDEN could not solve this problem effectively either (e.g. the updated cheetah results). In conclusion, I would like to know if there is any specific task that ELDEN can solve but LLM and other generic exploration methods fail to solve.

Clarifying the 1-step vs chained dependencies

Thank you for addressing my questions! I want to clarify what I meant by asking "Why would one expect ELDEN to do better than other 1-step methods?". This was in part motivated by the following statement about empowerment in Section 2.2: "However, due to the difficulty in measuring the mutual information across a multi-step trajectory, existing empowerment-based methods only measure 1-step empowerment, ...", but also about the repeated emphasis on chained dependencies. The two mechanisms you mentioned in your response are general properties of RL, which would equally apply to other methods such as 1-step empowerment or 1-step curiosity - the RL itself will learn to visit parts of the state space with large aggregate novelty. The fact that "chained dependencies" is mentioned so many times in the paper could give the wrong impression that ELDEN is trying to learn a much more complicated dependency graph, whereas it is actually a 1-step method just like many other methods. So I think it could be useful to just add a bit of clarity that ELDEN is still a 1-step model, but is able to handle chained dependencies because of RL.

Response to authors

I acknowledge that I have read the reviews and I have no further questions.