Reliably predicting potential failure risks of machine learning (ML) systems\nwhen deployed with production data is a crucial aspect of trustworthy AI. This\npaper introduces Risk Advisor, a novel post-hoc meta-learner for estimating\nfailure risks and predictive uncertainties of any already-trained black-box\nclassification model. In addition to providing a risk score, the Risk Advisor\ndecomposes the uncertainty estimates into aleatoric and epistemic uncertainty\ncomponents, thus giving informative insights into the sources of uncertainty\ninducing the failures. Consequently, Risk Advisor can distinguish between\nfailures caused by data variability, data shifts and model limitations and\nadvise on mitigation actions (e.g., collecting more data to counter data\nshift). Extensive experiments on various families of black-box classification\nmodels and on real-world and synthetic datasets covering common ML failure\nscenarios show that the Risk Advisor reliably predicts deployment-time failure\nrisks in all the scenarios, and outperforms strong baselines.\n