Whole brain segmentation using deep learning (DL) is a very challenging task\nsince the number of anatomical labels is very high compared to the number of\navailable training images. To address this problem, previous DL methods\nproposed to use a single convolution neural network (CNN) or few independent\nCNNs. In this paper, we present a novel ensemble method based on a large number\nof CNNs processing different overlapping brain areas. Inspired by parliamentary\ndecision-making systems, we propose a framework called AssemblyNet, made of two\n"assemblies" of U-Nets. Such a parliamentary system is capable of dealing with\ncomplex decisions, unseen problem and reaching a consensus quickly. AssemblyNet\nintroduces sharing of knowledge among neighboring U-Nets, an "amendment"\nprocedure made by the second assembly at higher-resolution to refine the\ndecision taken by the first one, and a final decision obtained by majority\nvoting. During our validation, AssemblyNet showed competitive performance\ncompared to state-of-the-art methods such as U-Net, Joint label fusion and\nSLANT. Moreover, we investigated the scan-rescan consistency and the robustness\nto disease effects of our method. These experiences demonstrated the\nreliability of AssemblyNet. Finally, we showed the interest of using\nsemi-supervised learning to improve the performance of our method.\n