Knowledge Distillation is an effective method to transfer the learning across\ndeep neural networks. Typically, the dataset originally used for training the\nTeacher model is chosen as the "Transfer Set" to conduct the knowledge transfer\nto the Student. However, this original training data may not always be freely\navailable due to privacy or sensitivity concerns. In such scenarios, existing\napproaches either iteratively compose a synthetic set representative of the\noriginal training dataset, one sample at a time or learn a generative model to\ncompose such a transfer set. However, both these approaches involve complex\noptimization (GAN training or several backpropagation steps to synthesize one\nsample) and are often computationally expensive. In this paper, as a simple\nalternative, we investigate the effectiveness of "arbitrary transfer sets" such\nas random noise, publicly available synthetic, and natural datasets, all of\nwhich are completely unrelated to the original training dataset in terms of\ntheir visual or semantic contents. Through extensive experiments on multiple\nbenchmark datasets such as MNIST, FMNIST, CIFAR-10 and CIFAR-100, we discover\nand validate surprising effectiveness of using arbitrary data to conduct\nknowledge distillation when this dataset is "target-class balanced". We believe\nthat this important observation can potentially lead to designing baselines for\nthe data-free knowledge distillation task.\n