Many neural network speaker recognition systems model each speaker using a\nfixed-dimensional embedding vector. These embeddings are generally compared\nusing either linear or 2nd-order scoring and, until recently, do not handle\nutterance-specific uncertainty. In this work we propose scoring these\nrepresentations in a way that can capture uncertainty, enroll/test asymmetry\nand additional non-linear information. This is achieved by incorporating a\n2nd-stage neural network (known as a decision network) as part of an end-to-end\ntraining regimen. In particular, we propose the concept of decision residual\nnetworks which involves the use of a compact decision network to leverage\ncosine scores and to model the residual signal that's needed. Additionally, we\npresent a modification to the generalized end-to-end softmax loss function to\ntarget the separation of same/different speaker scores. We observed significant\nperformance gains for the two techniques.\n