Recent advances in adversarial attacks and Wasserstein GANs have advocated\nfor use of neural networks with restricted Lipschitz constants. Motivated by\nthese observations, we study the recently introduced GroupSort neural networks,\nwith constraints on the weights, and make a theoretical step towards a better\nunderstanding of their expressive power. We show in particular how these\nnetworks can represent any Lipschitz continuous piecewise linear functions. We\nalso prove that they are well-suited for approximating Lipschitz continuous\nfunctions and exhibit upper bounds on both the depth and size. To conclude, the\nefficiency of GroupSort networks compared with more standard ReLU networks is\nillustrated in a set of synthetic experiments.\n