This work explores the potency of stochastic competition-based activations,\nnamely Stochastic Local Winner-Takes-All (LWTA), against powerful\n(gradient-based) white-box and black-box adversarial attacks; we especially\nfocus on Adversarial Training settings. In our work, we replace the\nconventional ReLU-based nonlinearities with blocks comprising locally and\nstochastically competing linear units. The output of each network layer now\nyields a sparse output, depending on the outcome of winner sampling in each\nblock. We rely on the Variational Bayesian framework for training and\ninference; we incorporate conventional PGD-based adversarial training arguments\nto increase the overall adversarial robustness. As we experimentally show, the\narising networks yield state-of-the-art robustness against powerful adversarial\nattacks while retaining very high classification rate in the benign case.\n