Sense and Sensitivity Analysis: Simple Post-Hoc Analysis of Bias Due to Unobserved Confounding

It is a truth universally acknowledged that an observed association without\nknown mechanism must be in want of a causal estimate. However, causal\nestimation from observational data often relies on the (untestable) assumption\nof `no unobserved confounding'. Violations of this assumption can induce bias\nin effect estimates. In principle, such bias could invalidate or reverse the\nconclusions of a study. However, in some cases, we might hope that the\ninfluence of unobserved confounders is weak relative to a `large' estimated\neffect, so the qualitative conclusions are robust to bias from unobserved\nconfounding. The purpose of this paper is to develop \\emph{Austen plots}, a\nsensitivity analysis tool to aid such judgments by making it easier to reason\nabout potential bias induced by unobserved confounding. We formalize\nconfounding strength in terms of how strongly the confounder influences\ntreatment assignment and outcome. For a target level of bias, an Austen plot\nshows the minimum values of treatment and outcome influence required to induce\nthat level of bias. Domain experts can then make subjective judgments about\nwhether such strong confounders are plausible. To aid this judgment, the Austen\nplot additionally displays the estimated influence strength of (groups of) the\nobserved covariates. Austen plots generalize the classic sensitivity analysis\napproach of Imbens [Imb03]. Critically, Austen plots allow any approach for\nmodeling the observed data and producing the initial estimate. We illustrate\nthe tool by assessing biases for several real causal inference problems, using\na variety of machine learning approaches for the initial data analysis. Code is\navailable at https://github.com/anishazaveri/austen_plots\n