First Order Methods with Markovian Noise: from Acceleration to Variational Inequalities

This paper delves into stochastic optimization problems that involve Markovian noise. We present a unified approach for the theoretical analysis of first-order gradient methods for stochastic optimization and variational inequalities. Our approach covers scenarios for both non-convex and strongly convex minimization problems. To achieve an optimal (linear) dependence on the mixing time of the underlying noise sequence, we use the randomized batching scheme, which is based on the multilevel Monte Carlo method. Moreover, our technique allows us to eliminate the limiting assumptions of previous research on Markov noise, such as the need for a bounded domain and uniformly bounded stochastic gradients. Our extension to variational inequalities under Markovian noise is original. Additionally, we provide lower bounds that match the oracle complexity of our method in the case of strongly convex optimization problems.

Paper

References (100)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer 8XQi7/10 · confidence 4/52023-06-26

Summary

The authors study first order stochastic algorithms under a Markovian noise: the noise is no longer i.i.d. as for vanilla SGD, but follows a Markov chain, of finite mixing time $\tau$. The proposed algorithms are SGD-like algorithms, with batchsizes that can vary. In particular, they use a randomized batchsize strategy, whose aim is to trade-off between the noise variance of the stochastic gradients, and their bias (due to non-iid sampling). Batches are typically taken equal to the mixing time of the Markov chain. An accelerated SGD algorithm is first introduced, and is proved to converge in the smooth strongly convex regime, with a rate that depends on the mixing time, a maximal gradient noise (max in terms of Markov chain state space and variables), condition number and strong convexity parameter. Various provided lower bounds show that this rate, although not optimal, has an optimal noise dependency. The SGD algorithm is then studied in the smooth non-convex setting, to obtain typical rates for this algorithm under this type of noise assumption. Finally, the authors study variational inequalities under Markovian noise, by introducing Randomized Extra Gradient (with also their batchsize strategy).

Strengths

Overall, the paper is well written, and very easy to follow, on a very interesting topic (stochastic approximation/optimization, under Markovian noise), that has re-gained attention lately. In this line of work, the proposed algorithm and analysis of accelerated SGD is novel, this is the first accelerated rate for SGD under Markov chain noise. The batch size technique is also nice, and well explained. Together, these two techniques give a nice algorithm with a near-optimal rate, as shown by the lower bounds, that are more general than previous ones in this setting. Although I am not very familiar of variational inequalities, the topic of VA with MC noise seems to be not very well studied in the literature, and provide the first bound with complexity proportional to the mixing time.

Weaknesses

There are two major concerns. Questions related to these are below in the dedicated section. 1) **The noise assumption**. The noise parameter $\sigma^2$ is defined as (forgetting about the $\delta^2$): $\sigma^2\geq \sup_{z,x} \|\nabla_x F(x,z)-f(x)\|^2$. Here, taking the sup over $x\in\mathbb R^d$ is not problematic (and $\delta^2$ is usually there to compensate for this), but I worry about this sup over $z\in Z$. Indeed, for the very simple setting of gaussian additive noise, the noise term here is equal to infinity. Previous works (e.g. [28] in the paper) instead consider $\bar \sigma^2$ for noise assumption (in the smooth non-convex or smooth-PL settings), where a mean is taken over $z$ instead of a sup. $\sigma^2$ here is thus not a noise variance, but an almost sure bound, which can be confusing. Table 1 should also discuss this noise assumption. 2) **Non-convex rate, Table, and missing comparison**. *(a)* The same non-convex rate is present in [28], under milder noise assumptions. *(b)* The strongly convex rate in this same reference has no mixing time in the transient regime (i.e. in the linearly converging term), and its noise is taken at the *optimum* (and is thus often null). *(c)* *We note that the complexity 55 bounds which scale linearly with the mixing time of the underlying Markov chain are currently 56 available only for non-convex minimization problems*: this assertion is false, since rates exists for convex problems, PL problems [23,28]. These three points *(a,b,c)* lead the reviewer to have some concerns with comparison related works that are cited in this paper (and that I would not have read if not cited, so that this leads me to believe, are there some other existing works like this in the literature ?). Indeed, the non-convex rates should clearly be compared with those of [28], that seem stronger, if I understand well. Also, their strongly convex rates should be discussed: the rate of Theorem 1 can be faster in some cases but not all I think. These rates should thus be in table 1 also.

Questions

Questions related to the two major weaknesses/concerns above: 1. Is the accelerated rate in Theorem 1 obtained by adding stronger assumptions than previous works (i.e. a sup over the state space instead of a mean) ? I would like the authors to discuss this assumption, and its relation with other works. 2. Could the author provide a clearer comparison of their work to the related ones ? I think this requires adding some discussions in several parts of the paper. 3. The lower bound in [28] is in term of hitting time, and the authors of this paper provide an algorithm and analysis without noise assumption; could this be related to the current submission ? Overall, I vote for acceptance, despite the concerns noted above, due to the contributions in variational inequalities, for the clarity of the paper, and the accelerated rate with batch randomization, while waiting for the authors' answer in order to provide a final decision.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

No such limitation noted.

Reviewer RVr56/10 · confidence 4/52023-07-04

Summary

The authors design and analyze first-order methods for stochastic minimization and stochastic variational inequalities (VIs), with Markovian noise. Typically in stochastic minimization, the target function is only accessible at point x, defined as $f(x)$, and its gradient $\nabla f(x)$, through the sampling of a seed $z \in Z$ using an oracle $F(x,z)$ and its gradient $\nabla F(x,z)$. Under Markovian noise, the seed $z_t$ sampled at time $t$, is a state in a time-homogeneous Markov chain with a unique invariant distribution $\pi$. Under this assumption, the goal is to minimize the expected value over the invariant distribution, that is $\min_x f(x)=E_{z\sim \pi}[F(x,z)]$. The authors make an assumption about the mixing time of the Markov chain. Beyond this, they make no other fundamental assumptions about the Markov chain. Moreover, for every point $x$ and seed $z$, and for some positive $\sigma,\delta$, the noisy gradient $\nabla F(\cdot,\cdot)$ satisfies the following inequality: $$ \|\nabla f(x) - \nabla F(x,z) \|^2 \leq \sigma^2 + \delta^2 \|\nabla f(x)\|^2, $$ The authors prove the following results: 1. For strongly-convex minimization, they design and analyze an accelerated algorithm. This algorithm builds upon the accelerated stochastic gradient descent by Nesterov, and they further use randomized batch sizes and momentum. Additionally, they show that their convergence rate is tight in all parameters, except for $\delta$. They also show that the dependence on $\delta$ and the parameter that controls the Markov chain's mixing is necessary. 2. The authors extend their results to non-convex minimization as well as to stochastic variational inequalities (VIs) with Markovian noise. They design and analyze algorithms for both settings by modifying existing algorithms, specifically, the Stochastic Gradient Descent and Stochastic Extra Gradient. They use the randomized batching technique, previously used in their algorithm for strongly-convex minimization. After the rebuttal, the authors effectively addressed my concerns about the real-world applications of their noise model. Furthermore, in my view, they have successfully addressed the main concerns raised by the other reviewers. Consequently, I maintained my original score.

Strengths

The assumptions about the Markovian noise make sense especially in setting where the value of the noise sample is only influenced by factors exogenous of the picked action. Moreover, I think the paper is well-written, and the comparison with related work seems extensive. The authors seem to improve either in terms of assumptions, or in terms of convergence in all their settings. The results look reasonable, I checked the statements in the main body.

Weaknesses

Given that this is a theory paper, the only minor weaknesses is that for the stochastic strongly-convex minimization, the bound is not tight for parameter $\delta$, and the fact that they are missing lower bounds for the non-convex setting, as well as the stochastic variational inequality setting. I do not consider this a big weakness.

Questions

Could the authors provide more examples of stochastic minimization or stochastic VIs with Markovian noise?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

I think that the authors have addressed the limitations of their work, by listing the necessary assumptions for their theorems. Moreover, I do not believe that this work will have negative societal impact.

Reviewer 9UP46/10 · confidence 4/52023-07-05

Summary

This paper presents an optimization study focusing on stochastic optimization with Markovian noise as opposed to i.i.d. noise. The authors explore three distinct scenarios: strongly convex, non-convex, and Variational Inequalities (VI). In comparison to previous research, they relax the boundedness assumptions related to the domain or gradient magnitude for cases 1 and 2. The newly proposed convergence bounds exhibit linear dependence on the mixing time, whereas previous works relied on quadratic or exponential dependence. The authors establish a lower bound to demonstrate the optimality of this linear dependence. Notably, this work is the first to demonstrate the convergence of the stochastic EG algorithm for VI inequalities under Markovian noise.

Strengths

This paper introduces a relaxation of the restrictive assumptions found in previous analyses by adopting common and mild assumptions. It explores three distinct settings and presents a comprehensive framework for their convenient analysis. The exposition of the paper is accessible and the proofs are clearly articulated.

Weaknesses

One major limitation of the method is its lack of obliviousness to the mixing time. This paper addresses this weakness by achieving linear dependence on the mixing time; however, this is accomplished through a dependence on multilevel mini-batching, where the mini-batch sizes are contingent on the mixing time. This particular dependence was absent in some previous works, which may explain their quadratic dependence on the mixing time. Another limitation is the comparison of convergence with existing results. While the paper relaxes certain restrictive assumptions, its convergence is limited to the neighborhood. In contrast, for example, [21] achieves asymptotic convergence to the minima. Moreover, when analyzing convergence with fixed step size or momentum parameters, the feasible parameter values in this work rely on σ and other unknown values to achieve an ε error. Finally, it would be beneficial to augment the paper with experimental results that validate the upper bound. Specifically, it would be valuable to observe the linear dependence on the mixing time and also include numerical results for the Variational Inequalities (VI) setting

Questions

1- There are some typos in your proofs in supplementary materials e.g. in the proof of Lemma 2, in the equation after line 601, it should be Log M instead of log N and x^k_g instead of x^k. 2- There is no discussion in the paper about a range of \delta^2 appearing in the accelerated rate. It could be large as the condition number and in that case, the rate given in the paper is worse than SGD in terms of dependence on the condition number. Why the accelerated rate doesn’t have backward compatibility in this case? 3- Your convergence proofs are based on fixed momentum and step size parameters. And due to that, you get a convergence to the neighborhood. Could you analyze in the decaying step size regime?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes

Reviewer 11kd7/10 · confidence 4/52023-07-15

Summary

This paper studies the convergence of first-order methods with Markovian noise for solving stochastic optimization problems and variational inequalities. The authors establish finite-time complexity bounds which scale linearly with the mixing time of the underlying Markov chain.

Strengths

This paper analyzes SGD (with and without Nesterov acceleration) with Markovian noise without the bounded domain and unifomly bounded stochastic gradient estimate assumptions, for both strongly convex and nonconvex scenarios. Matching lower bounds are also provided for the strongly convex case. Finally, an analysis for variational inequalities with general stochastic Markov oracle, arbitrary optimization set and arbitrary composite term is also provided. The results are interesting and appear to be novel.

Weaknesses

Purely theoretical paper. No numerical experiments are provided.

Questions

It is suggested that the authors should provide some simple numerical experiments to demonstrate the effectiveness of the proposed algorithms.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Reviewer 9UP42023-08-13

I'd like to thank the authors for their response! I maintain my score.

Authorsrebuttal2023-08-14

Glad to hear that! Thanks again for the time spent on the review!

Reviewer RVr52023-08-14

Thank you for your comprehensive rebuttal. I found the applications you highlighted particularly interesting. I'm also grateful that you addressed my concerns regarding lower bounds for other settings. I agree with your reasoning regarding the importance of lower bounds for the minimization setting. After examining feedback and rebuttals from other reviewers, I was particularly drawn to the discussions with reviewer 9UP4 regarding mixing time and the decaying step-size, as well as the conversation with reviewer 8XQi about the related work. Your rebuttal and proof sketch have addressed these newfound concerns. Given these reasons, I've opted to maintain my score.

Authorsrebuttal2023-08-15

Thank you so much for the detailed response! We thank you again for the work and time spent on the review and the discussion process!

Reviewer 11kd2023-08-16

Response

Adding experiments is just a suggestion. Also as a more theoretical researcher, I personally think that the theoretical contribution of the paper is sufficient for me to champion its acceptance. With numerical experiments, I would say the goal is to reach a broader audience and raise the impact of the paper, especially for large conferences like NeurIPS.

Authorsrebuttal2023-08-16

Thanks so much for the reply! We understand your position (we absolutely agree!). Since another reviewer also asked for experiments, we decided to add them.

Reviewer 8XQi2023-08-17

I thank the authors for all their clarifications and suggestions that are welcomed, and keep my good opinion of this paper unchanged

Authorsrebuttal2023-08-18

Thanks so much for the reply! Thanks again for the detailed review!

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC