Supervised learning-based segmentation methods typically require a large\nnumber of annotated training data to generalize well at test time. In medical\napplications, curating such datasets is not a favourable option because\nacquiring a large number of annotated samples from experts is time-consuming\nand expensive. Consequently, numerous methods have been proposed in the\nliterature for learning with limited annotated examples. Unfortunately, the\nproposed approaches in the literature have not yet yielded significant gains\nover random data augmentation for image segmentation, where random\naugmentations themselves do not yield high accuracy. In this work, we propose a\nnovel task-driven data augmentation method for learning with limited labeled\ndata where the synthetic data generator, is optimized for the segmentation\ntask. The generator of the proposed method models intensity and shape\nvariations using two sets of transformations, as additive intensity\ntransformations and deformation fields. Both transformations are optimized\nusing labeled as well as unlabeled examples in a semi-supervised framework. Our\nexperiments on three medical datasets, namely cardic, prostate and pancreas,\nshow that the proposed approach significantly outperforms standard augmentation\nand semi-supervised approaches for image segmentation in the limited annotation\nsetting. The code is made publicly available at\nhttps://github.com/krishnabits001/task$\\_$driven$\\_$data$\\_$augmentation.\n