We prove statistical rates of convergence for kernel-based least squares\nregression from i.i.d. data using a conjugate gradient algorithm, where\nregularization against overfitting is obtained by early stopping. This method\nis related to Kernel Partial Least Squares, a regression method that combines\nsupervised dimensionality reduction with least squares projection. Following\nthe setting introduced in earlier related literature, we study so-called "fast\nconvergence rates" depending on the regularity of the target regression\nfunction (measured by a source condition in terms of the kernel integral\noperator) and on the effective dimensionality of the data mapped into the\nkernel space. We obtain upper bounds, essentially matching known minimax lower\nbounds, for the $\\mathcal{L}^2$ (prediction) norm as well as for the stronger\nHilbert norm, if the true regression function belongs to the reproducing kernel\nHilbert space. If the latter assumption is not fulfilled, we obtain similar\nconvergence rates for appropriate norms, provided additional unlabeled data are\navailable.\n