Thank you very much for the comment. Below, we will further clarify that our approach does not assume that the belief $b$ has to follow a Gaussian distribution in Eq. (10) and our theory is sound regardless of the distribution of $b$.
First, the theoretical results on the sample complexity (Lemma 3.5, line 171) and convergence point (Lemma 3.7, line 181) of VDPO only depend on the ground-truth belief function $b$ and the optimal reference policy $\pi^*$ being given. Therefore, the theory of this work is sound regardless of the specific distribution of the belief.
Moreover, MSE is a commonly used loss function for general random variable, as evidenced in [1] (P283, the MSE is the lose function for Bernoulli variables; P286, the MSE is the lose function for exponential variable). Similar to many existing works, e.g., Eq. (3) in [2] and Eq. (1) in [3], the MSE in Eq. (10) in our implementation can be validly used without prior knowledge of the distribution. The assumption of the random variable following Gaussian distribution is needed only if we want to consider the maximum likelihood estimation, and its impact on our approach is only on practical performance but not soundness. Furthermore, as mentioned in the Limitation and Future Works section (Lines 261-262), we focus on the deterministic cases in this paper, where using MSE does not affect performance.
As mentioned in the Limitation and Future Works section (Lines 262-265), we plan to investigate stochastic applications in future, in which case if the belief $b$ does not follow Gaussian distribution, using MSE loss in Eq. (10) may not achieve the best performance. Note that beyond the distribution of the belief $b$, other factors, such as the performance gap between the reference policy $\pi$ and delayed policy $\pi_\Delta$ [4], may also affect the performance in stochastic environments. It would indeed be interesting to study the impact of these factors on our approach's practical performance and explore options to improve the performance when necessary (i.e., other loss functions).
To sum up, we emphasize that our work is theoretically sound regardless of the distribution of the belief in Eq. (10). Moreover, MSE is a commonly used loss function in practical reinforcement learning [2, 3] without the Gaussian distribution assumption, and this work focuses on deterministic benchmarks, where the belief is a deterministic mapping and MSE is thus the best unbiased estimator. We acknowledge that in stochastic MDPs, using MSE under non-Gaussian belief may lead to suboptimal performance, as pointed out by the reviewer's insightful comment. We plan to investigate the stochastic scenarios and various factors that may affect the practical performance (e.g., the distribution of the belief, the choice of the loss function, and the performance gap between reference policy and delayed policy) in our future work.
[1] Lehmann, Erich L., and George Casella. Theory of point estimation. Springer Science and Business Media, 2006.
[2] De Bruin, Tim, et al. "Integrating state representation learning into deep reinforcement learning." IEEE Robotics and Automation Letters 3.3 (2018): 1394-1401.
[3] Ota, Kei, et al. "Can increasing input dimensionality improve deep reinforcement learning?." International Conference on Machine Learning. PMLR, 2020.
[4] Wu, Qingyuan, et al. "Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays." International Conference on Machine Learning. PMLR, 2024.