Centered kernel alignment (CKA), also known as centered kernel-target alignment, is useful as a similarity measure between kernels and as a kernel-based similarity measure between feature representations. We prove that CKA based on a Gaussian RBF kernel converges to linear CKA in the large-bandwidth limit. The result relies on mean-centering of the feature maps and on a Hilbert-Schmidt Independence Criterion (HSIC) identity. We show that convergence onset is sensitive to the geometry of the feature representations, and that a notion of representation eccentricity, <inline-formula><tex-math notation="LaTeX">$\rho$</tex-math><alternatives><mml:math><mml:mi>ρ</mml:mi></mml:math><inline-graphic xlink:href="alvarez-ieq1-3216518.gif"/></alternatives></inline-formula>, constrains the bandwidth range for which Gaussian CKA can differ noticeably from linear CKA. Our experimental results suggest that Gaussian bandwidths less than <inline-formula><tex-math notation="LaTeX">$\rho$</tex-math><alternatives><mml:math><mml:mi>ρ</mml:mi></mml:math><inline-graphic xlink:href="alvarez-ieq2-3216518.gif"/></alternatives></inline-formula> should be selected in order to enable nonlinear modeling.