Whole brain segmentation using deep learning (DL) is a very challenging task\nsince the number of anatomical labels is very high compared to the number of\navailable training images. To address this problem, previous DL methods\nproposed to use a global convolution neural network (CNN) or few independent\nCNNs. In this paper, we present a novel ensemble method based on a large number\nof CNNs processing different overlapping brain areas. Inspired by parliamentary\ndecision-making systems, we propose a framework called AssemblyNet, made of two\n"assemblies" of U-Nets. Such a parliamentary system is capable of dealing with\ncomplex decisions and reaching a consensus quickly. AssemblyNet introduces\nsharing of knowledge among neighboring U-Nets, an "amendment" procedure made by\nthe second assembly at higher-resolution to refine the decision taken by the\nfirst one, and a final decision obtained by majority voting. When using the\nsame 45 training images, AssemblyNet outperforms global U-Net by 28% in terms\nof the Dice metric, patch-based joint label fusion by 15% and SLANT-27 by 10%.\nFinally, AssemblyNet demonstrates high capacity to deal with limited training\ndata to achieve whole brain segmentation in practical training and testing\ntimes.\n