Thank you and Formulas Clarification
Dear Reviewer,
We extend our heartfelt appreciation for your dedicated time and valuable insights. We are writing to address a technical issue we encountered during the submission process, which led to some of the formulas in our initial rebuttal not being displayed in the final version. We deeply regret any inconvenience this may have caused. In the following, we present a revised version of the rebuttal, specifically addressing the "Confidence Intervals of the ATE Estimator," for enhanced clarity.
- **Confidence Intervals of the ATE Estimator.**
- This is an excellent comment! We totally agree that establishing confidence intervals for the ATE estimators holds significant importance. As elaborated further below, the derivation of a confidence interval for the proposed estimator is quite straightforward. Should our paper be accepted, we will make sure to include these discussions.
- First, let us consider the settings under NMDPs in Section 3. Recall that after generating data through the proposed experimental design, we utilize the online doubly robust estimator $\widehat{\textrm{ATE}}_1$ to estimate the ATE. A key observation is that, under certain regularity conditions, the proposed estimator is asymptotically normal. More specifically, we have
$$\sqrt{n - 2m_0} (\widehat{\textrm{ATE}}_1 - \textrm{ATE}) \overset{d}{\rightarrow} N(0, \textrm{EB}_1(\pi^{b*})).$$
This motivates us to consider the following Wald-type confidence interval
$$
[\widehat{\textrm{ATE}}_1- \Phi^{-1} (1-\alpha/2) \sqrt{ \textrm{EB}_1(\pi^{b*})/(n - 2 m_0) }, \widehat{\textrm{ATE}}_1+ \Phi^{-1} (1-\alpha/2) \sqrt{ \textrm{EB}_1(\pi^{b*})/(n - 2 m_0) }],
$$
where $\Phi^{-1}$ is the inverse cumulative distribution function of a standard normal random variable.
It then suffices to estimate the asymptotic variance $\textrm{EB}\_1(\pi^{b*})$ to construct asymptotically valid confidence intervals. Notice that $\widehat{\textrm{ATE}}\_1$ can be represented as an average of martingale differences $\widehat{\textrm{ATE}}\_1 = \sum_{i=2m_0 +1}^n \psi^1_i /(n - 2m_0)$ where
$$
\psi\_i^1=\sum\_{a=0}^1\frac{(-1)^{a+1}}{T} \Big[ \widehat{V}\_{1,i-1}^a(O\_1^{(i)})+ \frac{\mathbb{I}(A\_1^{(i)}=a)}{\widehat{\pi}^{b*}\_{1,i-1}(a|O\_1^{(i)})}[\sum\_t R\_t^{(i)}-\widehat{V}\_{1,i-1}^a(O\_1^{(i)})]\Big].
$$
We propose using the sample variance of $\\{\psi^1\_i\\}_i$ to estimate $\textrm{EB}_1(\pi^{b*})$. Similar to Theorem 15 of Kallus and Uehara (2022) (https://dl.acm.org/doi/abs/10.5555/3455716.3455883), we can establish the consistency of the resulting sampling variance estimator.
- For TMDPs, we can similarly establish the asymptotic normality of $\widehat{\textrm{ATE}}_2$, i.e.,
$\sqrt{n - 2 m_0} ( \widehat{\textrm{ATE}}_2 - \textrm{ATE} ) \overset{d}{\rightarrow} N(0, \textrm{EB}_2(\pi^{b*}) ) $. The corresponding $1 - \alpha $ confidence interval can be constructed by
$$
[\widehat{\textrm{ATE}}_2- \Phi^{-1} (1-\alpha/2) \sqrt{ \textrm{EB}_2(\pi^{b*})/(n - 2 m_0) }, \widehat{\textrm{ATE}}_2+ \Phi^{-1} (1-\alpha/2) \sqrt{ \textrm{EB}_2(\pi^{b*})/(n - 2 m_0) }],
$$
where the unknown asymptotic variance $\textrm{EB}_2(\pi^{b*})$ can be similarly estimated via the sampling variance estimator.
We are once again immensely grateful for your dedicated attention and constructive feedback. In addition, if you have any additional questions or concerns, we would be glad to hear from you during the discussion period and provide clarification.