Most existing works in few-shot learning rely on meta-learning the network on\na large base dataset which is typically from the same domain as the target\ndataset. We tackle the problem of cross-domain few-shot learning where there is\na large shift between the base and target domain. The problem of cross-domain\nfew-shot recognition with unlabeled target data is largely unaddressed in the\nliterature. STARTUP was the first method that tackles this problem using\nself-training. However, it uses a fixed teacher pretrained on a labeled base\ndataset to create soft labels for the unlabeled target samples. As the base\ndataset and unlabeled dataset are from different domains, projecting the target\nimages in the class-domain of the base dataset with a fixed pretrained model\nmight be sub-optimal. We propose a simple dynamic distillation-based approach\nto facilitate unlabeled images from the novel/base dataset. We impose\nconsistency regularization by calculating predictions from the weakly-augmented\nversions of the unlabeled images from a teacher network and matching it with\nthe strongly augmented versions of the same images from a student network. The\nparameters of the teacher network are updated as exponential moving average of\nthe parameters of the student network. We show that the proposed network learns\nrepresentation that can be easily adapted to the target domain even though it\nhas not been trained with target-specific classes during the pretraining phase.\nOur model outperforms the current state-of-the art method by 4.4% for 1-shot\nand 3.6% for 5-shot classification in the BSCD-FSL benchmark, and also shows\ncompetitive performance on traditional in-domain few-shot learning task.\n