Deep neural network based speaker recognition systems can easily be deceived\nby an adversary using minuscule imperceptible perturbations to the input speech\nsamples. These adversarial attacks pose serious security threats to the speaker\nrecognition systems that use speech biometric. To address this concern, in this\nwork, we propose a new defense mechanism based on a hybrid adversarial training\n(HAT) setup. In contrast to existing works on countermeasures against\nadversarial attacks in deep speaker recognition that only use class-boundary\ninformation by supervised cross-entropy (CE) loss, we propose to exploit\nadditional information from supervised and unsupervised cues to craft diverse\nand stronger perturbations for adversarial training. Specifically, we employ\nmulti-task objectives using CE, feature-scattering (FS), and margin losses to\ncreate adversarial perturbations and include them for adversarial training to\nenhance the robustness of the model. We conduct speaker recognition experiments\non the Librispeech dataset, and compare the performance with state-of-the-art\nprojected gradient descent (PGD)-based adversarial training which employs only\nCE objective. The proposed HAT improves adversarial accuracy by absolute 3.29%\nand 3.18% for PGD and Carlini-Wagner (CW) attacks respectively, while retaining\nhigh accuracy on benign examples.\n