To facilitate a wide-spread acceptance of AI systems guiding decision making\nin real-world applications, trustworthiness of deployed models is key. That is,\nit is crucial for predictive models to be uncertainty-aware and yield\nwell-calibrated (and thus trustworthy) predictions for both in-domain samples\nas well as under domain shift. Recent efforts to account for predictive\nuncertainty include post-processing steps for trained neural networks, Bayesian\nneural networks as well as alternative non-Bayesian approaches such as ensemble\napproaches and evidential deep learning. Here, we propose an efficient yet\ngeneral modelling approach for obtaining well-calibrated, trustworthy\nprobabilities for samples obtained after a domain shift. We introduce a new\ntraining strategy combining an entropy-encouraging loss term with an\nadversarial calibration loss term and demonstrate that this results in\nwell-calibrated and technically trustworthy predictions for a wide range of\ndomain drifts. We comprehensively evaluate previously proposed approaches on\ndifferent data modalities, a large range of data sets including sequence data,\nnetwork architectures and perturbation strategies. We observe that our\nmodelling approach substantially outperforms existing state-of-the-art\napproaches, yielding well-calibrated predictions under domain drift.\n