In data analysis problems where we are not able to rely on distributional\nassumptions, what types of inference guarantees can still be obtained? Many\npopular methods, such as holdout methods, cross-validation methods, and\nconformal prediction, are able to provide distribution-free guarantees for\npredictive inference, but the problem of providing inference for the underlying\nregression function (for example, inference on the conditional mean\n$\\mathbb{E}[Y|X]$) is more challenging. In the setting where the features $X$\nare continuously distributed, recent work has established that any confidence\ninterval for $\\mathbb{E}[Y|X]$ must have non-vanishing width, even as sample\nsize tends to infinity. At the other extreme, if $X$ takes only a small number\nof possible values, then inference on $\\mathbb{E}[Y|X]$ is trivial to achieve.\nIn this work, we study the problem in settings in between these two extremes.\nWe find that there are several distinct regimes in between the finite setting\nand the continuous setting, where vanishing-width confidence intervals are\nachievable if and only if the effective support size of the distribution of $X$\nis smaller than the square of the sample size.\n