Thank you for your reply. We answer your further questions as follows.
---
### Q1. About the "fail-state" assumption and Nelder–Mead technique.
In theory, we would like to clarify that Theorem 4.2 does not rely on the “fail-state” assumption. The minimization over $s$ can be treated as an oracle, a common approach in theoretical RL analysis [1, 2, 3, 4]. Therefore, **our theoretical results are valid for any setting that satisfies the $d$-rectangular linear DRMDP structure**. Specifically, [1, 2, 3, 4] focus on standard linear MDPs and propose computationally efficient methods with the help of an oracle. For example, in the Bellman operator, the maximization over the action space $\mathcal{A}$, especially when continuous, is treated as an oracle. Similarly, in our work, assuming access to an oracle for minimization over the (potentially continuous) state space allows us to concentrate on the fundamental limits and challenges of $d$-rectangular linear DRMDPs with TV-divergence uncertainty sets.
In practice, the minimization/maximization can be approximated. We acknowledge that in RL, a gap often exists between theoretical guarantees and actual performance in practice. However, our focus in this work is on providing a rigorous theoretical understanding of the offline $d$-rectangular DRMDP setting, which aligns with the approach of other theoretical research and meets NeurIPS standards.
Furthermore, we plan to explore ways to address the issue of minimization over the state space in future work, which arises from the strong duality for TV distances (see Proposition G.1). As an alternative, we could consider replacing the TV-divergence uncertainty set with a Kullback-Leibler uncertainty set [6] or a $\chi^2$ uncertainty set [5], both leading to strong duality without the minimization term. We anticipate these approaches will lead to more practical algorithms for $d$-rectangular linear DRMDPs, though they are beyond the scope of this current paper.
---
### Q2. Why not consider general function approximation directly.
We would like to further elaborate on this issue from the perspectives of uncertainty sets and algorithm design.
**Panaganti et al. (2022) ([26] in our paper)**: [26] study offline $(s,a)$-rectangular DRMDPs with general function approximation. However, the $(s,a)$-rectangular uncertainty set can lead to overly conservative policies, particularly when the transition probabilities exhibit inherent structure [7,8]. To address this issue, our work focuses on structured uncertainty sets, specifically the $d$-rectangular linear DRMDP. This structured uncertainty set allows for a linear representation of robust value functions, resulting in less conservative policies compared to the general $(s,a)$-rectangular uncertainty set. This is why we do not follow [26] in studying general function approximation. In fact, our study of function approximation under structured uncertainty sets represents a significant contribution of our work, distinguishing it from [26].
**Zhou et al. (2023) ([58] in our paper)**: [58] explores the Double-Sampling uncertainty set and the Integral Probability Metric uncertainty set. They focus on scenarios with a simulator, proposing a robust Natural Actor-Critic algorithm with finite-time convergence guarantees, supported by extensive numerical experiments to demonstrate robustness. However, their method cannot be easily extended to address the $d$-rectangular uncertainty set of linear DRMDPs due to fundamental differences in uncertainty set formulations and algorithm design (value iteration-based versus policy gradient-based). We acknowledge that the method in [58] and our work are orthogonal. This distinction highlights potential avenues for complementary research in the future.
We will add the above discussion in our revision. If you have further questions, we are more than happy to discuss with you.
---
**References**
[1] Jin et al. Provably efficient reinforcement learning with linear function approximation. In COLT 2020.
[2] He et al. "Nearly minimax optimal reinforcement learning for linear markov decision processes." In ICML 2023.
[3] Xiong et al. Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Game. In ICLR 2023.
[4] Yin et al. "Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism." In ICLR 2022.
[5] Shi et al. The curious price of distributional robustness in reinforcement learning with a generative model. arXiv preprint, 2023.
[6] Zhou et al. Finite-sample regret bound for distributionally robust offline tabular reinforcement learning. In AISTATS 2021.
[7] Goyal et al. Robust markov decision processes: Beyond rectangularity. Mathematics of Operations Research, 48(1):203–226, 2023.
[8] Ma et al. Distributionally robust offline reinforcement learning with linear function approximation. arXiv preprint, 2022.