Time-Constrained Robust MDPs

Summary

This paper proposed a novel concept, time-constrained robust MDP, to address the conservativeness issue of the rectangular assumption. They assume the transition depends on an underlying parameter, and the parameter can be adversarially chosen from an uncertainty set. Several algorithms are proposed to solve the time-constrained robust MDP. Extensive numerical experiments show the effectiveness of the proposed framework and algorithms.

Strengths

Given that the field of robust MDP has limited large-scale experiments due to the intractability of the rectangular assumption, the idea in this paper is very nice. I agree that the rectangular assumption can be too general and lead to conservative policies, due to its inefficiency in leveraging the inherent structure of the uncertainty set. The proposed time constraint robust MDP is interesting, especially since the experiment results seem to be promising.

Weaknesses

My concern is that the methods proposed in this paper might be too heuristic, given the non-stationarity and possibly non-existence of the optimal policy. As I am not very familiar with the experiment side, I am not sure if the evaluation in this paper is sufficient. See my questions for details.

Questions

1. In the experiments, the oracle-TC is not always the best method. It seems that the performances are not that stable. Also, the vanilla TC outperforms in almost half cases. It is kind of anti-intuitive. 2. Compared to baseline methods, the TC-based methods show robustness to some extent, but how robust are they? What are the expected best (robust) performances? 3. In algorithm 1, what is "sample a mini-batch of transitions" for? How are the agent and adversary updated? 4. On lines 174-175, the authors claim that the rectangularity assumptions are rarely met in real-world scenarios. I think more evidence or references are required to support this claim.

Rating

5

Confidence

2

Soundness

2

Presentation

2

Contribution

2

Limitations

yes

Summary

This paper aims to develop a new framework of robust RL addressing the over-conservability issue of rectangularity and dynamic uncertainty set assumption. This framework allows for time-dependent, correlated and multifactorial disturbance to the dynamics. Three distinct algorithms are developed depending on the available information to the policy. Extensive numerical experiments are further provided to demonstrate the performance of the proposed algorithms.

Strengths

- The proposed framework addresses the major drawbacks of existing robust RL approach: overly conservative due to the use of dynamic and rectangular uncerainty sets. - Extensive experiments are provided to demonstrate the performance of the three algorithms - The approaches can be used with existing robust value iteration approaches.

Weaknesses

- Thm 2.1 only applies to the case where the policy has the exact information of \psi, which in practice is usually unavailable. - The usefulness of this framework remains questionable, as it poses a significant challenge of constructing such complex uncertainty set with time and state-action correlation. - The training algorithm has the same flavor of the adversarial training as those in the literature, e.g., RARL. The novelty seems rather limited as this is mostly an experimental paper.

Questions

See weaknesses above.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

na

Summary

The paper introduces Time-Constrained Robust MDPs (TC-RMDPs) as a novel formulation to address the overly conservative nature of traditional robust RL under sa-rectangularity assumptions. The authors propose three algorithms to handle time-dependent and correlated disturbances in different situations. Extensive evaluations on continuous control benchmarks show that these algorithms outperform traditional robust RL methods in terms of balancing performance and robustness.

Strengths

1. The introduction of TC-RMDPs addresses a significant limitation of overly pessimistic in traditional robust RL, making the approach more applicable to real-world scenarios with time-dependent disturbances, which is interesting and novel. In addition, the paper presents three distinct algorithms with varying levels of information usage, expanding its usage. 2. The authors provide formal theoretical guarantees for the proposed algorithms, which is appreciated. 3. Extensive experiments on continuous control benchmarks demonstrate the efficacy of the proposed methods.

Weaknesses

# Major 1. The complexity of the proposed algorithms, especially Oracle-TC, may limit their practical applicability in real-world scenarios where complete environmental information is unavailable. 2. How to select the Lipschitz constant $L$ in practical scenarios? Some ablations studies and analysis regarding $L$ would be beneficial. 3. It would be better if the authors could provide some visualizations of the difference between the traditional independent transitions with the time-constrained parametric transitions. 4. What is the number of worst-case episodes in the experiments? If it is 1, maybe it would be better to display the results as the average of the worst 10% of episodes for better robustness. This is because having only 1 episode might introduce stochasticity. # Minor 1. It is suggested to show the equation number for better illustration and reference. 2. There are some typos in the paper. For instance, in line 102, $B=\mathcal{B}(0_{\Psi},L)$ should be corrected to $B=\mathcal{B}({\Psi}_0,L)$, the right-most column in Table 1 should be Avg. The authors should carefully review the paper for such errors.

Questions

please see above

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

The framework assumes that the transition parameter vector $\psi$ is known during training. However, in real-world applications, it could be challenging. Also, the assumption on the uncertainty set $\Psi$ is also not practical in practice.

Summary

The paper defines a robust training method for MDPs under uncertainty in the environment dynamics. Current methods assume sa-rectangularity, where the transition dynamics from consecutive states are independent of one another. The authors argue that this assumption is unrealistic and leads to overly conservative policies. Alternatively, this paper breaks the independence assumption in the environment dynamics to use a time-step evolution approach. This paper presents a Time-Constrained TC-MDP where the Time-Constrained Parameterized MDP kernels evolve each time-step constrained to be Lipschitz continuous. The model is trained in an adversarial manner, where the adversarial policy is learned to evolve the model dynamics but is limited to be L-close to the previous dynamics by definition. This learning method is tested under different variant conditions (cosine, exponential, linear, and logarithmic) with positive results. Robustly trained agents that learn under the worst TC adversities outperform vanilla robust algorithms that assume time independence in the model dynamics.

Strengths

- The evaluation is fair. The model trained under worst-case time-constrained adversarial dynamics is then evaluated under different fixed evolving dynamics (cosine, exponential, linear, and logarithmic) that range in stochastic changes much larger than the used in learning (L=0.1 in evaluation vs L=0.001 in training). - The definition of TC-RMDP is smartly selected to preserve stationarity, defining the state as the tuple ($S \times \Psi$) - The TC strategy can be straightforwardly implemented in previous algorithms by limiting the search space of the adversarial $psi$ at every time-step.

Weaknesses

- Reporting that the worst-case TC variants outperform vanilla adversarial models as M2TD3 or RARL is expected. As the decisions of the adversary are rather limited when compared to the vanilla algorithms. Perhaps highlighting the evaluation results (fixed or static setting) in the conclusions would help in emphasizing this.

Questions

- In Algorithm 1, should is be: $b_{t+1} = \bar{\pi}(s_t, a_t, \psi_t)$ ? - Typo in Page 7 Line 276: "Appendix G and G"

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

As the authors mention, this method assumes that the environment dynamics can be correctly parametrized in a confident set $Psi$. Additionally, selecting the radius L of the proximal adversarial set of actions is not trivial. Fortunately, the authors tested the environment with a large enough L (100 times larger than the one used during training) with positive results, which validates their claim.