We present $\\Delta$-UQ -- a novel, general-purpose uncertainty estimator\nusing the concept of anchoring in predictive models. Anchoring works by first\ntransforming the input into a tuple consisting of an anchor point drawn from a\nprior distribution, and a combination of the input sample with the anchor using\na pretext encoding scheme. This encoding is such that the original input can be\nperfectly recovered from the tuple -- regardless of the choice of the anchor.\nTherefore, any predictive model should be able to predict the target response\nfrom the tuple alone (since it implicitly represents the input). Moreover, by\nvarying the anchors for a fixed sample, we can estimate uncertainty in the\nprediction even using only a single predictive model. We find this uncertainty\nis deeply connected to improper sampling of the input data, and inherent noise,\nenabling us to estimate the total uncertainty in any system. With extensive\nempirical studies on a variety of use-cases, we demonstrate that $\\Delta$-UQ\noutperforms several competitive baselines. Specifically, we study model\nfitting, sequential model optimization, model based inversion in the regression\nsetting and out of distribution detection, & calibration under distribution\nshifts for classification.\n