In a data-scarce field such as healthcare, where models often deliver\npredictions on patients with rare conditions, the ability to measure the\nuncertainty of a model's prediction could potentially lead to improved\neffectiveness of decision support tools and increased user trust. This work\nadvances the understanding of uncertainty estimation for classification and\nrisk prediction on medical tabular data, in a two-fold way. First, we expand\nand refine the set of heuristics to select an uncertainty estimation technique,\nintroducing tests for clinically-relevant scenarios such as generalization to\nuncommon pathologies, changes in clinical protocol and simulations of corrupted\ndata. We furthermore differentiate these heuristics depending on the clinical\nuse-case. Second, we observe that ensembles and related techniques perform\npoorly when it comes to detecting out-of-domain examples, a critical task which\nis carried out more successfully by auto-encoders. These remarks are enriched\nby considerations of the interplay of uncertainty estimation with class\nimbalance, post-modeling calibration and other modeling procedures. Our\nfindings are supported by an array of experiments on toy and real-world data.\n