Summary
The paper proposes a state-confounded (SC-) and a robust-state confounded (RSC-) MDP formulation to account for setups where a confounder satisfying the backdoor criterion confounds the states. The RSC-MDP setup assumes that the confounder lies in an uncertainty set which is part of the MDP parameters, and the aim is to learn a value function that is the lower bound of the different value functions arising out of different instances of the RSC-MDP at different values of the confounder within its uncertainty set. They propose an empirical algorithm to approximate the effect of perturbing the confounder by perturbing single dimensions of the states and setting them to values present in ‘otherwise nearby’ states (defined by distance between the two states in dimensions other than the dimension being edited). The paper claims that this indirectly estimates the effects of perturbing the confounder. They learn a graph-based model that is supposed to be a causal model, to generate new transitions (given a state and action s_t, a_t, generate s_t+1, r_t) based on the perturbed states and (original) action pairs. They augment SAC by randomly replacing part of the sampled batches with the generated transitions and learn as usual, and show that this recipe is robust to the spurious correlation in the nominal envs they define. All experiments are in the state space (non-vision inputs).
Strengths
1. The paper studies an important problem and proposes a set of benchmarks with handcrafted spurious correlations in a CARLA environment and in a robosuite environment which could be useful for more future works to study spurious correlations in RL.
2. The RSC-MDP formulation could also be useful, although I’m not sure how much the notation and proofs scale to an arbitrary number of confounding variables.
Weaknesses
1. The paper structure is somewhat confusing to me: The MDP formulations and the empirical algorithm don’t seem to have much of a connection and seems like both were developed independently - please correct me if I’m missing something here. It would be great to further clarify exactly how the algorithm is helping to solve the robust SC-MDP.
2. The notion of semantic uncertainty is not very precise or clear (I don’t think there’s any references either), and could be removed in my opinion since the RSC-MDP formulation can simply use the “uncertainty set” phrasing.
3. The appendix suggests that the training environments have perfect spurious correlations: the correlation isn’t broken in even a few instances of the training environments, so for eg. there would never be a (non-generated) transition in the SAC buffer that would have the brightness value low in its state space value, in the case that the training envs are following the correlation regime where brightness and traffic are both high during the day. It’s unclear to me then, why the causal model would ever generate a state value that doesn’t also have the brightness value set to high (same logic for the perturbation procedure). Am I missing something else here? Or are the training envs set up such that an agent will train on two kinds of envs at once - night time with less traffic and day time with more traffic, and so it s possible to see samples with the brightness value set to both low and high across otherwise nearby states?
It would be great to clarify what exactly is the train and test env distribution and what is expected of the graphical causal model - is it to generate counterfactual transitions (counterfactual to what?) and conclusively show that it is actually doing that by visualizing or plotting some property of the samples it is generating. It's possible that I have misinterpreted something here, and I'm willing to raise my score if the authors' response clarifies some of these questions.
4. I would also suggest mentioning relevant related work explicitly aimed at resolving causal confusion in online [1] and offline RL [2] (and comparing to the closest version if applicable).
5. It would be great to include the training curves for the results in Table 1 as well, since that gives insight into whether other methods simply converge later to a similar highest reward, or whether they are entirely limited in their ability to converge to a similarly high reward as the best performing method.
[1] Resolving Causal Confusion in Reinforcement Learning via Robust Exploration . Clare Lyle, Amy Zhang, Minqi Jiang, Joelle Pineau, Yarin Gal. ICLR Self-Supervised RL Workshop 2021.
[2] Can Active Sampling Reduce Causal Confusion in Offline Reinforcement Learning? Gunshi Gupta, Tim G. J. Rudner, Rowan Thomas McAllister, Adrien Gaidon, Yarin Gal, NeurIPS Offline RL Workshop 2022, CleaR 2023
Questions
1. How is the experiment to answer R3 designed? It's unclear how to test the w/o P^{c} case since the confounder isn't known anyway?
2. As stated previously, it would help to get more insights into how novel the generated transitions from the causal model really are.
3. Have you observed or quantified the robustness-performance tradeoff mentioned in line 263, which will lead to a performance drop in some envs? I expect the perturbation procedure of states to only work for specific kinds of envs, so it would be good to show the failure cases as well.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
Given how heuristic the technique of generating perturbed states is (swapping single dimensions between otherwise nearby states), the paper should properly discuss the many challenges of scaling this approach to a high-dimensional state space like that in visual domains.