We greatly appreciate your valuable assessment of our work. Below, we address certain specific concerns.
--**Fisher consistency**. *Fisher consistency is described as “Roughly this requires that if the whole ‘population’ of random variables is observed, then the method of estimation should give exactly the right answer"*, see Cox and Hinkley (1974, p. 287) , Tasche (2017). Thus,
$\mathbb{E} [f_1(Q^{\pi})]=\mathbb{E} [f_2(\rho^{\pi})]=\mathbb{E} [f_3(w^{\pi})]=\mathbb{E}[f_4(Q^{\pi},w^{\pi})]=J(\pi)$
shows that the OPE methods (without abstraction) themselves are Fisher consistent. For our methods,
- $J(\pi)=\mathbb{E} [f_1(Q^{\pi})]=\mathbb{E} [f_1(Q^{\pi}_{\phi})]$ implies the Q-function-based method is Fisher consistent;
- $J(\pi)=\mathbb{E} [f_3(w^{\pi})]=\mathbb{E} [f_3(w^{\pi}_{\phi})]$ means MIS is Fisher consistent;
- $J(\pi)=\mathbb{E} [f_2(\rho^{\pi})]=\mathbb{E} [f_2(\rho^{\pi}_{\phi})]$ indicates SIS is Fisher consistent;
- $J(\pi)=\mathbb{E}[f_4(Q^{\pi},w^{\pi})]=\mathbb{E} [f_4(Q^{\pi}_{\phi},w^{\pi} _{\phi})]$ describes Fisher consistency of the double robust method.
This logic can also be found in Park and Weisberg (1998), equation (6) in page 232.
--**Two-step procedure**. We find this comment regarding the 'two-step procedure' confusing. The two-step approach refers to our earlier version submitted to NeurIPS. For this conference, we submitted a modified version, yet the term appears twice in the reviewer’s comments. A similar concern was raised by one of the reviewers in our previous submission. This reviewer, who provided the same score, had expressed a willingness to revise their score based on updates to our proof.
In response, we devoted significant effort during the rebuttal period to provide a clear and accessible proof within the word limit. Unfortunately, during the discussion period, the reviewer became reluctant to engage, only responding in the final hours. From their last response, we understood that their primary concerns were addressed. However, they did not specify which aspects of our responses were inadequate but maintained their original score, stating that "a substantial revision is needed."
Additionally, we would like to clarify that we did not perform data splitting to train abstractions and OPE estimators. Both procedures were conducted on the same dataset, ensuring no reduction in sample size that could compromise the performance of the final estimator.
--**Reference**:
- Cox, D. R., & Hinkley, D. V. (1974). Theoretical statistics. Chapman and Hall, London.
- Park, C., & Weisberg, S. (1998). Fisher consistency of GEE models under link misspecification. Computational statistics & data analysis, 27(2), 229-235.
- Tasche, D. (2017). Fisher consistency for prior probability shift. Journal of Machine Learning Research, 18(95), 1-32.