Differentiable Sorting Networks for Scalable Sorting and Ranking Supervision

Sorting and ranking supervision is a method for training neural networks\nend-to-end based on ordering constraints. That is, the ground truth order of\nsets of samples is known, while their absolute values remain unsupervised. For\nthat, we propose differentiable sorting networks by relaxing their pairwise\nconditional swap operations. To address the problems of vanishing gradients and\nextensive blurring that arise with larger numbers of layers, we propose mapping\nactivations to regions with moderate gradients. We consider odd-even as well as\nbitonic sorting networks, which outperform existing relaxations of the sorting\noperation. We show that bitonic sorting networks can achieve stable training on\nlarge input sets of up to 1024 elements.\n