How to leverage cross-document interactions to improve ranking performance is\nan important topic in information retrieval (IR) research. However, this topic\nhas not been well-studied in the learning-to-rank setting and most of the\nexisting work still treats each document independently while scoring. The\nrecent development of deep learning shows strength in modeling complex\nrelationships across sequences and sets. It thus motivates us to study how to\nleverage cross-document interactions for learning-to-rank in the deep learning\nframework. In this paper, we formally define the permutation-equivariance\nrequirement for a scoring function that captures cross-document interactions.\nWe then propose a self-attention based document interaction network and show\nthat it satisfies the permutation-equivariant requirement, and can generate\nscores for document sets of varying sizes. Our proposed methods can\nautomatically learn to capture document interactions without any auxiliary\ninformation, and can scale across large document sets. We conduct experiments\non three ranking datasets: the benchmark Web30k, a Gmail search, and a Google\nDrive Quick Access dataset. Experimental results show that our proposed methods\nare both more effective and efficient than baselines.\n