Model-free Low-Rank Reinforcement Learning via Leveraged Entry-wise Matrix Estimation

We consider the problem of learning an $\varepsilon$-optimal policy in controlled dynamical systems with low-rank latent structure. For this problem, we present LoRa-PI (Low-Rank Policy Iteration), a model-free learning algorithm alternating between policy improvement and policy evaluation steps. In the latter, the algorithm estimates the low-rank matrix corresponding to the (state, action) value function of the current policy using the following two-phase procedure. The entries of the matrix are first sampled uniformly at random to estimate, via a spectral method, the leverage scores of its rows and columns. These scores are then used to extract a few important rows and columns whose entries are further sampled. The algorithm exploits these new samples to complete the matrix estimation using a CUR-like method. For this leveraged matrix estimation procedure, we establish entry-wise guarantees that remarkably, do not depend on the coherence of the matrix but only on its spikiness. These guarantees imply that LoRa-PI learns an $\varepsilon$-optimal policy using $\widetilde{O}({S+A\over \mathrm{poly}(1-\gamma)\varepsilon^2})$ samples where $S$ (resp. $A$) denotes the number of states (resp. actions) and $\gamma$ the discount factor. Our algorithm achieves this order-optimal (in $S$, $A$ and $\varepsilon$) sample complexity under milder conditions than those assumed in previously proposed approaches.

Paper

References (47)

Scroll for more · 35 remaining

Similar papers

Peer review

Reviewer ff2R4/10 · confidence 3/52024-07-05

Summary

This paper proposes a leveraged matrix estimation method for low-rank policy evaluation and extends it to policy iteration as a model-free learning algorithm. The main idea is to separate the Q matrix estimation into two phases. The first step is to use half of the sample budget to estimate the leverage scores of the Q matrix. Based on the leverage scores, a skeleton of the value matrix is sampled. Then roughly one-fourth of the sample budget is used to sample the entries in the skeleton. The left sample budget is used to sample the entries outside the skeleton, and the final Q-value matrix is constructed by CUR decomposition with inverse leverage scores weighting. It's proved that the leveraged matrix estimation method relaxed eliminates the commonly used assumption of incoherence., and guarantees an entry-wise estimation error. The effectiveness of the proposed method is demonstrated via simulation in the Appendix.

Strengths

The paper is well-written and easy to follow. The theoretical investigation comes with rigorous proof. By adopting leveraged matrix estimation, the authors can relax the incoherence assumption of the low-rank Q matrix, which serves as a major advantage of the proposed method.

Weaknesses

It seems there is a lack of discussion about leveraged matrix estimation methods in matrix completion/low-rank matrix estimation literature. How does the proposed method perform when compared with other regularized policy evaluation approaches (for example, nuclear-norm/max-norm regularization)? It is crucial in assessing the significance and novelty of this work. Is that possible to include experiments in real-world datasets to enhance the significance of this work?

Questions

Is the low-rank assumption of the Q function under any deterministic policy very restrictive? How does that compare to related works? It would be better to replace shadows in the figures (experiments) with error bars, as shallows overlay each other and make it hard to read.

Rating

4

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The paper is more in a theoretical flavor and lacks real-world experiments.

Reviewer 9Ehq5/10 · confidence 3/52024-07-11

Summary

This paper considers the reinforcement learning problem with low-rank latent structure. The objective of this problem is of learning an $\epsilon$-optimal policy in a tabular setting. For this problem, they devised LoRa-PI (Low-Rank Policy Iteration), a model-free learning algorithm alternating between policy improvement and policy evaluation steps. Their main innovation lies in the latter part where the algorithm estimates the low-rank matrix corresponding to the (state, action) value function of the current policy using the following two-phase procedure. The entries of the matrix are first sampled uniformly at random to estimate, via a spectral method, the leverage scores of its rows and columns. These scores are then used to extract a few important rows and columns whose entries are further sampled. The algorithm exploits these new samples to complete the matrix estimation using a CUR-like method. For this leveraged matrix estimation procedure, they used spikiness instead of coherence (two-to-infinity norm assumption).

Strengths

1) According to the authors, their matrix estimation method is the first able to yield entry-wise guarantees that do not depend on the matrix coherence but on its spikiness instead. 2) The authors used the tabular setting where entrywise estimation is important and applied an estimation method that fits well with entry-wise estimation. As they mentioned in Lemma 1, the singular value difference could be very large, but the tabular setting lets authors focus more on entrywise estimation. I believe this is a nice strategy for writing. 3) The authors clearly state why policy iteration is much more useful than value iteration to alleviate condition number assumptions.

Weaknesses

1) Readers might think this is another application of low-rank estimation literature. Of course, this type of 'adapting results of other fields appropriately' studies is also important in the research, but I'd say this is not ground-breaking. 2) (Important) I am not sure how meaningful the transition from 'coherence' to 'spikiness' is. Maybe it is directly related to Question 1. If I understand correctly, this is the main reason for using the CUR-based matrix completion instead of others, but I believe the authors should provide a more convincing reason (such as theorems?) to make authors study CUR-based matrix completion. 3) (Important) It is not clear how much improvement they made. It would be great if there's a table for low-rank RL that compares the performance/assumptions of previous works concisely. Could you add a table of reference that includes all low-rank and many recent tabular RL results to see how much improvement you made? 4) (Important) Dependence of the threshold parameter $\beta$ with parameter $d$ or $\sigma_d(Q^\pi)$. See the questions section for details. Also, this $\beta$ seems very important, but it is written in the appendix. It feels like the authors are trying to deceive readers.

Questions

- (Important) Is spikiness always better than coherence? The authors state that in a specific case, spikiness could be finite but coherence could be very large. Is there a case where coherence is finite but spikiness blows up? To use the word 'less restrictive' I think this proof is also necessary. - (Important) Table (mentioned in the weakness section) - (Important) Could you explain how your threshold of the singular value could be free from the rank $d$? In many sparse linear or low-rank estimations, rank works as a parameter for the threshold. To alleviate the rank condition from the threshold, one must assume the minimum signal condition, such as $\sigma_{min}>\epsilon$ for some $\epsilon>0$. Seems like in your paper it was possible since you assumes 'minimum signal condition' on Theorem 2, $\epsilon < \sqrt{\frac{S+A}{SAd} \sigma_d (Q^\pi)}$. You don't know $d$ or $\sigma_d$, how could you be sure about this? One main selling point of this paper is that the learner doesn't need to know $d$ or minimum eigenvalue conditions before the game starts. (line 224-225) - Is there any great idea to extend your estimation idea to the model-based method? - This idea works well in the tabular setting. What about the non-tabular settings, such as linear RL? Let me know the difficulty of extending your result to linear RL. *** Minor points 1) Maybe using $\sigma_{min}$ instead of $\sigma_d$? Researchers frequently use $\sigma_d (Q^\pi)$ as $d$-th largest eigenvalue and $d=\max_\pi rank(Q^\pi)$. This means in the traditional notation $\sigma_d(Q^\pi)=0$. Though they've mentioned it in their related works, it would be better to change it as $\sigma_{min}$ or define $\sigma_d$ somewhere near the Notation paragraph.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

N/A

Authorsrebuttal2024-08-13

**1.** Thank you for recognizing the improvements made by our algorithm. Indeed, one key takeaway is that our adaptive, learning-based approach offers stronger guarantees than uniform random sampling and can even match the performance of settings with prior knowledge of the problem structure. **2. Regarding the condition on $T$ and its dependence on $\sigma_d$ .** First of all, we wish to clarify that by saying we do not know $\sigma_d$, we mean that our algorithm does not require as input $\sigma_d$ or any lower bound on it. For example, the number of samples per epoch in the algorithms in [34] and [35] depends on $\sigma_d$, while our algorithm does not. This allows our approach to work without prior knowledge of the parameters of $Q$-matrices which previous algorithms require. This is especially challenging in iterative settings like ours, where prior knowledge would need to include the parameters of all $Q^\pi$ matrices encountered until convergence. Next, both papers ([34] and [35]) are based on the CUR approach, and we will explain why this approach requires assumptions on the number of samples $T$. In the proof of our Theorem 5 we need to make use of the following inequality: $$ \Vert \widetilde{Q}^\pi_{\tau}(\mathcal{I},\mathcal{J})^\dagger \Vert_{\mathrm{op}} = \frac{1}{\sigma_d(\widetilde{Q}^\pi_{\tau}(\mathcal{I},\mathcal{J}))} \leq \frac{1}{\sigma_d(Q^\pi(\mathcal{I},\mathcal{J})) - \Vert \widetilde{Q}^\pi_{\tau}(\mathcal{I},\mathcal{J}) - Q^\pi(\mathcal{I},\mathcal{J}) \Vert_{\mathrm{op}}} $$ (Note that for simplicity, we set $L=R=\mathrm{Id}$.) The second term in the denominator is an error term that scales with $1/\sqrt{T}$. Therefore, to ensure the expression above is positive, we require a condition of the form $T\gtrsim 1/\sigma_d^2(Q^\pi(\mathcal{I},\mathcal{J}))$. This condition is thus necessary for the analysis of any CUR-based methods, including [34]. Now, note that in [35] (e.g., Theorem 14 or Corollary 16 for finite $S,A$), the authors use a **much stronger** assumption: the discount factor $\gamma = O\left( \frac{\sigma_d^2(Q(\mathcal{I},\mathcal{J}))}{d^2 V_{\max}} \right)$, which must hold even in the high sample regime (for large $T$). In contrast, our analysis works for any value of $\gamma$. **2.1. Regarding small $\sigma_d$.** We agree that a very small $\sigma_d$ should not drastically affect the guarantees. If $\sigma_d$ is very small and, without loss of generality, $\sigma_{d-1}$ is sufficiently large, one could set $d_{\mathrm{eff}} = d-1$ and obtain bounds that depend on $\sigma_{d-1}$ instead. However, ignoring $\sigma_d$ introduces an approximation error that depends on the incoherence of the $d$-th singular vectors. To safely disregard the recovery of the $d$-th singular value and its associated singular vectors, we must ensure that $\sigma_d$ is small and that the $d$-th singular vector is sufficiently incoherent. This contrasts with Frobenius/spectral norm guarantees, where neglecting terms corresponding to the $d$-th singular value leads only to an additive error of $\sigma_d$, regardless of the singular vectors. We are not aware of any method that can select the rank parameter $d$ to address these issues without assuming knowledge of singular vectors or the appropriate incoherence parameters. **3. Regarding incoherence and spikiness.** While our algorithm's worst-case guarantees scale with $\mu^4$, we believe this comparison is not particularly useful. As shown in the table, in a general, non-adaptive setting, the guarantees typically scale with $\alpha^2 \mu^2$ . However, in this paper (following [35]), we demonstrate that adaptive sampling allows us to eliminate the dependence on the incoherence parameter $\mu$. On the other hand, the spikiness parameter $\alpha$ quantifies the difficulty of recovering the full low-rank structure. We believe that the dependence on $\alpha$ cannot be improved through different sampling methods and instead requires a novel matrix recovery method. We will discuss these issues further in the paper and clarify our concept of incoherent-free matrix recovery. **3.1. Regarding the dependence on the rank parameter $d$.** We omit explicit mention of the rank parameter $d$ because the sample complexity scales as $d^3$ in all three CUR-based settings, including ours and those in [34,35]. We believe that the nuclear norm-based method used in Theorem 21 of [34] offers slightly better scaling, with $d^2$. We will clarify this further in our revised version of the manuscript

Reviewer Fr8E5/10 · confidence 3/52024-07-14

Summary

The works present a policy iteration algorithm that relies on supposedly low rank structure of value function (Q-function) The proposed work first estimates the low-rank matrix of a given policy before performing policy iteration step. The Leveraged Matrix Estimation (LME) algorithm learns the Q-matrix for a given policy. LME for a given sampling budget estimates Q-matrix in a 3 step process. The policy iteration algorithm also uses LME at the policy evaluation stage.

Strengths

The proposed Q-matrix estimation and policy iteration algorithm are interesting for sparse MDPs. Sample complexity bounds for the algorithm Analysis is easy to follow

Weaknesses

It is not clear where such low-rank Q-matrix appear in real-world. Is coherence and spikiness of Q generally appear in control problems? Can authors show any real-world application like gym environments? There is significant body of work that leverage rank deficit structure of Q, it is not immediately what are the additional insights the current draft has over the past work. Can you compare with other algorithms proposed.

Questions

See weakness

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

See weakness

Reviewer WY9t6/10 · confidence 2/52024-07-17

Summary

In this paper, the authors consider the problem of learning an $\epsilon$-optimal policy in systems with low-rank latent structures and is order optimal under weaker conditions. The proposed algorithm iterates between exploitation (policy evaluation) and exploration (policy improvement). For policy evaluation, the entries are sampled in two stages to complete matrix estimation. The adopted approach provides entry-wise guarantees with respect to the spikiness of the matrix.

Strengths

- The authors have done a great job of clearly defining the problem they are trying to solve and conducting a thorough literature survey showcasing the existing work as well as how their results improve upon the existing work. - The proposed algorithm is parameter-free which implies that it does not rely on spikiness/rank/coherence bounds to work. Similarly, the lack of dependence of sample complexity bounds also shows the strengths of these results.

Weaknesses

- The notation can be hard to read especially as l and L and r and R imply completely different things, while in most cases they imply smaller letters imply matrix entries for the matrix indicated by the larger letter.

Questions

- Theorem 1 shows that the Leverage Scores Estimation depends on the $d^{th}$ singular value, Wouldn't that impose constraints indirectly on the rank/spikiness of the original matrix? - In Line 285, is the subscript of $\Omega$ a typo.

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

- Minor Comment: Please elaborate on acronyms before using them. The goal is to make the paper self-contained for a first-time reader as well even though these acronyms are widely known in practice. Line 10 - CUR, Line 30 - MDP, Line 88 - ERM, MLE - Can the authors compare the simulation results in Appendix A with existing approaches akin to [35] to showcase if their algorithm indeed works in a practical setting? - Can the authors also showcase how their algorithm holds for different values of rank, spikiness, and condition number?

Reviewer 9Ehq2024-08-12

Thank you for your detailed rebuttal. 1) Seems like the table in the common rebuttal shows clear improvement compared to the previous algorithms. It looks great that this work is free from the anchor assumption, and shows the best performance. 2) About the $\sigma_d$ dependence, I understand the author's point, but I can't shake off a certain unease. In short, it seems that $beta$ nominally depends only on $T$, but $T$ comes with the condition that it must be "large enough to satisfy all the conveniences." Of course, this is a common practice in RL or bandit papers, but including $\sigma_d$ makes me question whether it is really correct to say that we don't know the minimum signal condition. The extent to which T can be assumed is subjective, but if "free from d and $\kappa$ (and eventually $\sigma_d$)" is a selling point, I personally believe that we should be able to know in advance how large $T$ needs to be before running the algorithm. Time is tight, but if possible, it would be helpful for my judgment if you could point out the cases where the $T$ conditions in the papers included in the rebuttal Table involve assumptions about variables that the agent cannot know. 2-1) Also, what happens when $\sigma_d$ is too small and you cannot satisfy the condition on $T$? In my intuition, the small singular value cannot affect the result much, but in your analysis, it feels like it will affect a lot on the result. 3) It's also good to know that coherence always includes spikiness, but according to your calculation, $\alpha^2 < \mu^4 d^2$ and your sample complexity eventually depends on $\alpha^2$ according to Theorem 2 and 3, so in some terrible case, one might say your result is depending on $\mu^4$, right? Instead of selling $\mu$-freeness, I'd rather say this paper is on a different assumption than $\mu$-based studies. Maybe they assumed they are in a setting of moderate $\mu$, while you are assuming the case of moderate $\alpha$. 3-1) It would be also great if the authors could provide a table 'include' $d$. I know $d << S, A$, but when it is multiplicative I think one should add all the parameters. Again, thanks a lot for your rebuttal.

Authorsrebuttal2024-08-14

Dear Reviewer, We thank you again for your comments and feedback. We hope that our answers address your concern and would be happy to follow up before the end of the rebuttal phase. Many thanks.

Authorsrebuttal2024-08-13

Dear reviewer, We thank you once again for your efforts. As the rebuttal period is nearing its end, we would appreciate if you could let us know whether our rebuttal addresses your concerns. We would be also happy to answer any further questions you may have while we still can. Thank you.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC