We consider the dynamics of a linear stochastic approximation algorithm\ndriven by Markovian noise, and derive finite-time bounds on the moments of the\nerror, i.e., deviation of the output of the algorithm from the equilibrium\npoint of an associated ordinary differential equation (ODE). We obtain\nfinite-time bounds on the mean-square error in the case of constant step-size\nalgorithms by considering the drift of an appropriately chosen Lyapunov\nfunction. The Lyapunov function can be interpreted either in terms of Stein's\nmethod to obtain bounds on steady-state performance or in terms of Lyapunov\nstability theory for linear ODEs. We also provide a comprehensive treatment of\nthe moments of the square of the 2-norm of the approximation error. Our\nanalysis yields the following results: (i) for a given step-size, we show that\nthe lower-order moments can be made small as a function of the step-size and\ncan be upper-bounded by the moments of a Gaussian random variable; (ii) we show\nthat the higher-order moments beyond a threshold may be infinite in\nsteady-state; and (iii) we characterize the number of samples needed for the\nfinite-time bounds to be of the same order as the steady-state bounds. As a\nby-product of our analysis, we also solve the open problem of obtaining\nfinite-time bounds for the performance of temporal difference learning\nalgorithms with linear function approximation and a constant step-size, without\nrequiring a projection step or an i.i.d. noise assumption.\n