We consider $d$-dimensional linear stochastic approximation algorithms (LSAs)\nwith a constant step-size and the so called Polyak-Ruppert (PR) averaging of\niterates. LSAs are widely applied in machine learning and reinforcement\nlearning (RL), where the aim is to compute an appropriate $\\theta_{*} \\in\n\\mathbb{R}^d$ (that is an optimum or a fixed point) using noisy data and $O(d)$\nupdates per iteration. In this paper, we are motivated by the problem (in RL)\nof policy evaluation from experience replay using the \\emph{temporal\ndifference} (TD) class of learning algorithms that are also LSAs. For LSAs with\na constant step-size, and PR averaging, we provide bounds for the mean squared\nerror (MSE) after $t$ iterations. We assume that data is \\iid with finite\nvariance (underlying distribution being $P$) and that the expected dynamics is\nHurwitz. For a given LSA with PR averaging, and data distribution $P$\nsatisfying the said assumptions, we show that there exists a range of constant\nstep-sizes such that its MSE decays as $O(\\frac{1}{t})$.\n We examine the conditions under which a constant step-size can be chosen\nuniformly for a class of data distributions $\\mathcal{P}$, and show that not\nall data distributions `admit' such a uniform constant step-size. We also\nsuggest a heuristic step-size tuning algorithm to choose a constant step-size\nof a given LSA for a given data distribution $P$. We compare our results with\nrelated work and also discuss the implication of our results in the context of\nTD algorithms that are LSAs.\n