Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learning

This paper studies off-policy evaluation (OPE) in the presence of unmeasured confounders. Inspired by the two-way fixed effects regression model widely used in the panel data literature, we propose a two-way unmeasured confounding assumption to model the system dynamics in causal reinforcement learning and develop a two-way deconfounder algorithm that devises a neural tensor network to simultaneously learn both the unmeasured confounders and the system dynamics, based on which a model-based estimator can be constructed for consistent policy value estimation. We illustrate the effectiveness of the proposed estimator through theoretical results and numerical experiments.

Paper

References (100)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer P4Ev4/10 · confidence 3/52024-07-09

Summary

This paper considers a setting in which there are observed tuples $(O_{i,t}, A_{i, t}, R_{i, t})$ with $ 1\leq i \leq N$, $1 \leq t \leq T$; Here, $O_{i,t}$ represents observable quantities, $A_{i,t}$ is an action generated by a behavioural policy, and $R_{i,t}$ is the associated reward. The goal of this paper is off-policy evaluation; specifically, the authors aim to estimate the expected cumulative reward under a given policy that differs from the behavioural policy. The authors propose a generative model of the observed trajectories. The specific contribution of this work is to consider a two-way confounder, which as I understand a tuple of latent variables $(U_i, W_t)$ representing subject-and time-varying components, respectively.

Strengths

The proposed two-way confounding model can be considered original. The authors considers separating a confounder into a time- and subject-invariant parts, and model their interaction by a neural network. This may be considered as a low-rank approximation (analogously to nonlinear matrix-factorisation), and reduces the number of the latent variables from $NT$ to $N + T$. The empirical evaluation shows that the proposed method performs well. The authors also experimentally demonstrates the sensitivity to the two-way confounder assumption.

Weaknesses

# Clarity ## Major issues: 1. Line 104: It is unclear what $\mathbb{E}^\pi$ denotes. If the expectation is simply taken with respect to the dynamics generated by $\pi$, doesn't this have an effect from the confounders? 2. Introducing the estimator (1) before the model would help the reader to understand the quantities to be estimated and hence the model. 3. The assumption on the data generating process is not straightforward to understand. In the unconstrained unmeasured confounding (UUC) setting, the trajectories cannot be independent, and the data assumption (Line 89) alone already excludes the UUC setting. At the same time, for the proposed TWUC setting, the presence of the subject-invariant latent $W_t$ renders the trajectories dependent on each other. Using a graphical model would help clarify the assumption. ## Minor issues: 1. The $\cup$ notation is undefined. 2. Line 187: Gaussian Mixture Model. The transition kernel does not look like a mixture. Why the term? 3. L 287: forr # Theoretical justification While I realise that the focus of the paper is on the empirical performance, the supporting theory is relatively weak. 1. The validity of the estimator (1) is claimed, but there is not reference to this claim. Moreover, the expectation needs to be estimated, and calling (1) an estimator sound odd to me. 2. I have not been able to read though the proof, but the first paragraph already casts some doubt. Why does estimand $\eta$ depend on the latent and why are they not marginalised? (this might explain what I don't understand about the assumption). It seems that the only variability comes from the initial observation rather than the whole trajectories (since the confounders are fixed). The consistency result therefore has limited reliability,

Questions

NA

Rating

4

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

Yes

Reviewer GM6x7/10 · confidence 3/52024-07-09

Summary

This work proposes a novel technique for performing off-policy evaluation (OPE) in the presence of unobserved confounders that have been classified as "two-way" unmeasured confounders, viz., by assuming that there exist both time-invariant and trajectory-invariant confounders, but not time-and-trajectory-invariant confounders. Roughly outlined, their approach is to first use a network architecture that takes embeddings of these different confounder types, uses a neural tensor network to first obtain estimates of transitions from a transition network and actions from an actor network, and then performs OPE in standard Monte Carlo simulation by estimating the expected cumulative reward from many simulated trajectories. Experiments compare their TWD technique's LMSE and LMAE against a host of traditional and modern techniques that span numerous means of dealing with confounding from traditional model-based methods to POMDPs to TWDIDPs. Across several simulation studies using principled data generation (like the PK-PD model for tumor growth generation), the TWD technique is shown to be the most robust and obtain lower LMSE and LMAE compared to the others compared. Ablation studies following these show how the sum of the TWD components are needed to obtain maximal accuracy.

Strengths

- The review of related work is particularly rich; the authors do a tremendous job at finding recent and relevant work related to the problem space and the various viewpoints in approaching deconfounding. - The writing is exceptionally clear, with well-stated assumptions, defined variables, and illustrations of process in each figure. - Classifying the types of transience to which UCs belong is interesting and exploited in a nice way within the TWD technique. - I'm a fan of the idea of performing MC simulation using the learned transition network to accomplish the OPE. I found the elaboration on line [205] to be particularly compelling. - Compared strategies in Section 5 are comprehensive and recent.

Weaknesses

- [197] One minor point that likely deserves some consideration in general: the causal structure of the system, viz., that certain contexts O should *not* be included in evidence while estimating \eta^\pi -- for example, considering the classic confounding M structure (see Cinelli, Forney, & Pearl, 2022), there are cases where an observed context must be omitted from conditioned evidence to obtain an unbiased estimate. - [5.1] While I completely agree with the risks of ignoring UCs and that the NUC assumption is usually made for convenience rather than realism, would it not also be fair to run an experiment in an environment *without* UCs to see if there is some tax paid by this paper's technique (which seems to *assume* their presence)? It's possible that looking for confounding where none exists comes with a price that the experiments herein should advertise if present. Still, I think the authors did a good job of putting their technique through its paces within the contexts of Section 5, especially the ablation study.

Questions

** Trivial Points ** - Although it has its own section explaining it, two-way unmeasured confounding appears in the introduction without even a short explanation of its meaning; it would aid the clarity if it had a brief, intuitive lead before it gets referenced in the text. - [39] "However, these works require restrict mathematical assumptions that are hard to verify..." -- was "restrict" meant to be "strict" or "restricting"? - [117] "...which pluggs in the estimated latent factors..." -- typo? - [769] "A illustrative example..." -- typo - [Eqn. 1] Why is the subscript on O_{i,1} rather than O_{i,t}? - [236] "Assumptions 4 is concerned..." -- typo ** Substantive Questions ** - [102] "Our objective lies in evaluating the expected cumulative reward under a given target policy π, which depends solely on the observation and not on the latent confounders." The phrasing could be a bit improved here -- one read is that the expected cumulative reward depends solely on the observation and not the latent confounders, but your transition model in the previous paragraph makes it clear that this is not the case. You mean to say that the evaluation itself must be only a function of the observations since we do not have direct access to the UCs? - [Figure 1a] What's the significance of the edges emanating from Z_{i,t}? Just to show that OWUC vs TWUC are different ways of modeling the more general unconstrained UC space? - One part that you might want to develop a bit more: the intended application of these techniques -- is this primarily suited for artificially generated policies or those employed by human decision-makers like given in the examples?

Rating

7

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

N/A

Reviewer 7kZ27/10 · confidence 4/52024-07-10

Summary

This paper studies the problem of confounded OPE. The authors explore a new structural assumption of the data-generating process called the two-way confounding assumption. They also propose an algorithm called the two-way deconfounder that can deal with the new setting they consider. The authors perform theoretical and empirical analyses to validate their proposed assumptions and algorithms.

Strengths

* The idea of two-way confounding is interesting and well-motivated. This paper is the first to introduce the two-way confounder assumption into the field of causal RL. * The two-way deconfounder algorithm is simple and intuitive. And it can deal with challenging causal RL problems with modern functional approximation techniques. * The authors perform solid theoretical analysis and empirical analysis to evaluate their methodology. The theoretical results include a non-asymptotic error bound of the two-way deconfounder estimator. I checked the proof and found the result is reliable. In the empirical analysis, the authors test their method in both simulated and real-world RL scenarios. They also compare their method with various baselines.

Weaknesses

* I suggest the author add more explanations about the problem setting to the intro section, making it more readable for readers outside the field of causal inference. * The algorithm imposes additional modeling restrictions to the MDP by parameterizing $P$ as conditional Gaussians. * (Minor) It seems that some notations may denote both a variable or a set of variables, which is a little bit confusing.

Questions

* In the literature of model-based RL, the transition dynamic can be represented with some kind of generative model (for example, [1]). For the two-way confounder algorithm, is it possible to parameterize $P$ as some generative model and thus eliminate the conditional Gaussian restriction? * Given different confounding assumptions (unconstrained/two-way/one-way confounding), how can we determine which setting fits best with the problem if we do not have full prior knowledge of the data-generating process? Do there exist any hypothesis testing procedures or simple heuristic methods? Can the authors give a short discussion? * (Minor) What does the term "relational neural network" mean? [1] Buesing, Lars, et al. "Learning and querying fast generative models for reinforcement learning." arXiv preprint arXiv:1802.03006 (2018).

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have addressed the limitations of their paper in the appendix.

Reviewer 7kZ22024-08-12

I want to thank the authors for their detailed response, which addresses my major concerns. In particular, the authors present a well-rounded discussion on how to determine the best-fitting setting for the problem without knowing the data-generating process. I have also read the authors' responses to other reviews. From my viewpoint, these responses are convincing and provide interesting ideas beyond the paper. In conclusion, I think this paper is a solid work and will keep my positive rating.

Reviewer wVkG6/10 · confidence 4/52024-07-13

Summary

The authors study off-policy evaluation for longitudinal data with hidden confounders, which are accounted for by assuming they are either time-invariant or state-invariant but not both.

Strengths

* The idea is relatively original, important, and the contribution might be significant. Presentation is somewhat clear. * The section studying real-world MIMIC III data shows that this method might be practically relevant.

Weaknesses

* Despite the term "deconfounder" being featured prominently, the authors do not engage with much of the literature on the deconfounder algorithm initially proposed by Wang and Blei (2019). Not much is said beyond the cryptic statement, "the validity of the deconfounder algorithm relies crucially on the consistent estimation of the latent factors." This method is also inspired by two-way fixed effects regression, but since they are drawing connections to the deconfounder algorithm, the authors should discuss its fundamental assumptions in the context of its limitations and criticisms by D'Amour, Ogburn et al., and others. * The assumption of two-way unmeasured confounding is illuminated with concrete examples starting at line 127, but the Propositions 1-3 that follow are difficult to understand without explicit context. Looking at Appendix A, it seems that additional assumptions need to be made for Proposition 2 or Proposition 3 to be true. Minor comments * Typo on line 117: "pluggs" * Figure 5 is too small to read.

Questions

In Figure 4, why are TWD's estimated values so much more negative? And why is the variance higher?

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors should discuss limitations in the main text rather than the appendix.

Reviewer GM6x2024-08-09

Thank you for the detailed reply, to which I have only one response re: Omitting observations for avoiding biases. I appreciate you looking into the reference I provided and even citing the example of the confounding scenario with treatment $X$, confounders $Z$, and outcome $Y$. While I agree with your assessment that the observations in your work, $O_{i,1}$ are _pretreatment_ covariates (i.e., that are observed before the chosen action), I think it might be too broad to state that all such observations would match the confounding structure wherein controling for $Z$ blocks the back-door -- it is possible, as I mentioned about the M-graph example (Model 7 in Cinelli, Forney, & Pearl, 2022), to have pretreatment covariates that are correlated with both treatment and outcome but that one should _not_ control for, even if they are observed before the treatment choice. I also agree with your ideas for future study / amendments and think that they will strengthen this work, but am maintaining my current score (which I think is quite strong) because I cannot assess these future implementations yet, think the impact of this paper is appropriately scored, and agree with some of the theoretical qualms that some of the other reviewers have raised. Still, I think this paper introduces fresh ideas that the community would find interesting and found your replies to other reviewers to be thorough and convincing as well. Best of luck!

Authorsrebuttal2024-08-10

Again, thank you very much for your valuable feedback and insightful comments. We are largely encouraged by your positive assessment of our paper. After carefully reviewing the paper you mentioned [1], we realize that directly conditioning on $O_{i,1}$ may have been insufficiently cautious. The issue with the M-graph that you highlighted is certainly a valid concern. We will discuss this in the paper. Meanwhile, it remains possible to employ the confounder selection algorithm to determine whether to including observations in a data-dependent manner (see e.g., Example 1 on Page 6 of [2]). However, most confounder selection algorithms only explore the bandit scenario, and further research would be required to extend this to reinforcement learning. We are truly grateful for your insightful feedback once again. [1] Cinelli C, Forney A, Pearl J. A crash course in good and bad controls[J]. Sociological Methods & Research, 2022: 00491241221099552.\ [2] Guo F R, Lundborg A R, Zhao Q. Confounder selection: Objectives and approaches[J]. arXiv preprint arXiv:2208.13871, 2022.

Reviewer wVkG2024-08-11

Thank you for the detailed response. I am glad that you noticed and corrected the underlying issue with Figure 4. I am raising my score accordingly.

Authorsrebuttal2024-08-11

We are sincerely grateful for your thoughtful feedback and for considering our response. We deeply appreciate the time and effort you dedicated to reviewing our work.

Reviewer P4Ev2024-08-12

I thank the authors for the clarifications. As a follow-up comment, I think it is confusing to analyse the case where the latent variables are fixed given that the objective does not contain any latent (as in the general response), and this should be corrected (as proposed). The theoretical analysis currently does not seem to be well linked to the proposed training method, it remains unclear when the four assumptions can be achieved. For this reason, I still find it difficult to provide a strong support for acceptance, while I appreciate the empirical nature (and performance) of the present work.

Authorsrebuttal2024-08-14

We sincerely thank the referee for the positive feedback on the empirical nature and performance of our paper. Below, we address the remaining comments on our theoretical analysis. - __Latent v.s. fixed variables.__ We will revise this as suggested. As outlined in the rebuttal, the error bound remains similar to Theorem 1, with an additional term of $c_2 R_\max (N^{-1/2}+T^{-1/2})$ up to some logarithmic factors, to account for the randomness of the latent factors. We hope this clarification resolves your concerns. - __Clarification on the assumptions.__ This issue was not raised in the official review, but we believe Assumptions 1–4 are mild and achievable. We hope our response addresses your concerns. * Assumptions 1 and 2 are standard in the reinforcement learning literature (e.g., [1], [2], [3], [4], [11], [12], [13], [14]). Though space limits citations, these assumptions are widely applied in offline policy optimization and off-policy evaluation. * Assumption 3 is concerned with the estimation error of the transition function. This assumption is flexible, as $\varepsilon_{\mathcal{P},\delta}$ can be adjusted to a larger value to meet the condition. In our implementation, we use a conditional Gaussian model for the transition function, following [6]. The total variation bound, measured by $\varepsilon_{\mathcal{P},\delta}$, thus reduces to the estimation errors of the mean and covariance functions. Using neural networks for function approximation allows the estimator to achieve an optimal non-parametric convergence rate (e.g., [7], [8], [9]). This yields the specific form of $\varepsilon_{\mathcal{P},\delta}$. Even without the conditional Gaussian model, deep generative learning algorithms with theoretical guarantees can be employed, with error bounds provided in studies like [10]. * Assumption 4 is concerned with the estimation error of the latent confounders. Like Assumption 3, it is flexible, as $\varepsilon_{U,W,\delta}$ can be adjusted to a larger value. Additionally, under a two-way additive model assumption, the factors $\\{U\_i\\}\_i$ and $\\{W\_t\\}\_t$ can be estimated at orders of $\sqrt{T^{-1/2}\log (N/\delta)}$ and $\sqrt{N^{-1/2}\log (T/\delta)}$ with probabilities at least $1-O(\delta)$ (see, [15]). This provides the detailed forms of $\varepsilon_{U,W,\delta}$. [1] Chen J, Jiang N. Information-theoretic considerations in batch reinforcement learning[C]//International Conference on Machine Learning. PMLR, 2019: 1042-1051. \ [2] Fan J, Wang Z, Xie Y, et al. A theoretical analysis of deep Q-learning[C]//Learning for dynamics and control. PMLR, 2020: 486-489.\ [3] Liu Y, Swaminathan A, Agarwal A, et al. Provably good batch off-policy reinforcement learning without great exploration[J]. Advances in neural information processing systems, 2020, 33: 1264-1274. \ [4] Uehara M, Sun W. Pessimistic model-based offline reinforcement learning under partial coverage[J]. arXiv preprint arXiv:2107.06226, 2021. \ [5] Hornik K, Stinchcombe M, White H. Multilayer feedforward networks are universal approximators[J]. Neural networks, 1989, 2(5): 359-366.\ [6] Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J.Y., Levine, S., Finn, C. and Ma, T., 2020. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33, pp.14129-14142.\ [7] Schmidt-Hieber, J. (2020). Nonparametric regression using deep neural networks with ReLU activation function. The Annals of Statistics, 48(4).\ [8] Farrell, M. H., Liang, T., & Misra, S. (2021). Deep neural networks for estimation and inference. Econometrica, 89(1), 181-213.\ [9] Imaizumi, M., & Fukumizu, K. (2019, April). Deep neural networks learn non-smooth functions effectively. In The 22nd international conference on artificial intelligence and statistics (pp. 869-878). PMLR.\ [10] Zhou, Y., Shi, C., Li, L., & Yao, Q. (2023). Testing for the Markov property in time series via deep conditional generative learning. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(4), 1204-1222.\ [11] Rashidinejad P, Zhu B, Ma C, et al. Bridging offline reinforcement learning and imitation learning: A tale of pessimism[J]. Advances in Neural Information Processing Systems, 2021, 34: 11702-11716.\ [12] Yin M, Wang Y X. Towards instance-optimal offline reinforcement learning with pessimism[J]. Advances in neural information processing systems, 2021, 34: 4065-4078.\ [13] Cui Q, Du S S. Provably efficient offline multi-agent reinforcement learning via strategy-wise bonus[J]. Advances in Neural Information Processing Systems, 2022, 35: 11739-11751.\ [14] Wang X, Cui Q, Du S S. On gap-dependent bounds for offline reinforcement learning[J]. Advances in Neural Information Processing Systems, 2022, 35: 14865-14877.\ [15] Bian, Z., Shi, C., Qi, Z., & Wang, L. (2023). Off-policy evaluation in doubly inhomogeneous environments. arXiv preprint arXiv:2306.08719.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC