Often used as importance sampling probabilities, leverage have become indispensable in randomized algorithms for linear algebra, optimization, graph theory, and machine learning. A major body of work seeks to adapt these to approximation problems. However, existing low-rank leverage scores can be difficult to compute, often work for just a single application, and are sensitive to matrix perturbations. We show how to avoid these issues by exploiting connections between approximation and regularization. Specifically, we employ ridge leverage scores, which are simply standard leverage computed with respect to an $\ell_2$ regularized input. Importance sampling by these gives the first unified solution to two of the most important sampling problems: $(1+\epsilon)$ error column subset selection and $(1+\epsilon)$ error projection-cost preservation. Moreover, ridge leverage satisfy a key monotonicity property that does not hold for any prior leverage scores. Their resulting robustness leads to two sought-after results in randomized linear algebra. 1) We give the first input-sparsity time approximation algorithm based on iterative column sampling, resolving an open question posed in [LMP13], [CLM+15], and [AM15]. 2) We give the first single-pass streaming column subset selection algorithm whose real-number space complexity has no dependence on stream length.