Low-shot sketch-based image retrieval is an emerging task in computer vision,\nallowing to retrieve natural images relevant to hand-drawn sketch queries that\nare rarely seen during the training phase. Related prior works either require\naligned sketch-image pairs that are costly to obtain or inefficient memory\nfusion layer for mapping the visual information to a semantic space. In this\npaper, we address any-shot, i.e. zero-shot and few-shot, sketch-based image\nretrieval (SBIR) tasks, where we introduce the few-shot setting for SBIR. For\nsolving these tasks, we propose a semantically aligned paired cycle-consistent\ngenerative adversarial network (SEM-PCYC) for any-shot SBIR, where each branch\nof the generative adversarial network maps the visual information from sketch\nand image to a common semantic space via adversarial training. Each of these\nbranches maintains cycle consistency that only requires supervision at the\ncategory level, and avoids the need of aligned sketch-image pairs. A\nclassification criteria on the generators' outputs ensures the visual to\nsemantic space mapping to be class-specific. Furthermore, we propose to combine\ntextual and hierarchical side information via an auto-encoder that selects\ndiscriminating side information within a same end-to-end model. Our results\ndemonstrate a significant boost in any-shot SBIR performance over the\nstate-of-the-art on the extended version of the challenging Sketchy, TU-Berlin\nand QuickDraw datasets.\n