Models for natural language understanding (NLU) tasks often rely on the\nidiosyncratic biases of the dataset, which make them brittle against test cases\noutside the training distribution. Recently, several proposed debiasing methods\nare shown to be very effective in improving out-of-distribution performance.\nHowever, their improvements come at the expense of performance drop when models\nare evaluated on the in-distribution data, which contain examples with higher\ndiversity. This seemingly inevitable trade-off may not tell us much about the\nchanges in the reasoning and understanding capabilities of the resulting models\non broader types of examples beyond the small subset represented in the\nout-of-distribution data. In this paper, we address this trade-off by\nintroducing a novel debiasing method, called confidence regularization, which\ndiscourage models from exploiting biases while enabling them to receive enough\nincentive to learn from all the training examples. We evaluate our method on\nthree NLU tasks and show that, in contrast to its predecessors, it improves the\nperformance on out-of-distribution datasets (e.g., 7pp gain on HANS dataset)\nwhile maintaining the original in-distribution accuracy.\n