Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detection

Detecting out-of-distribution (OOD) samples is vital for developing machine\nlearning based models for critical safety systems. Common approaches for OOD\ndetection assume access to some OOD samples during training which may not be\navailable in a real-life scenario. Instead, we utilize the {\\em predictive\nnormalized maximum likelihood} (pNML) learner, in which no assumptions are made\non the tested input. We derive an explicit expression of the pNML and its\ngeneralization error, denoted as the {\\em regret}, for a single layer neural\nnetwork (NN). We show that this learner generalizes well when (i) the test\nvector resides in a subspace spanned by the eigenvectors associated with the\nlarge eigenvalues of the empirical correlation matrix of the training data, or\n(ii) the test sample is far from the decision boundary. Furthermore, we\ndescribe how to efficiently apply the derived pNML regret to any pretrained\ndeep NN, by employing the explicit pNML for the last layer, followed by the\nsoftmax function. Applying the derived regret to deep NN requires neither\nadditional tunable parameters nor extra data. We extensively evaluate our\napproach on 74 OOD detection benchmarks using DenseNet-100, ResNet-34, and\nWideResNet-40 models trained with CIFAR-100, CIFAR-10, SVHN, and ImageNet-30\nshowing a significant improvement of up to 15.6\\% over recent leading methods.\n