The emergence of CNNs in mainstream deployment has necessitated methods to\ndesign and train efficient architectures tailored to maximize the accuracy\nunder diverse hardware & latency constraints. To scale these resource-intensive\ntasks with an increasing number of deployment targets, Once-For-All (OFA)\nproposed an approach to jointly train several models at once with a constant\ntraining cost. However, this cost remains as high as 40-50 GPU days and also\nsuffers from a combinatorial explosion of sub-optimal model configurations. We\nseek to reduce this search space -- and hence the training budget -- by\nconstraining search to models close to the accuracy-latency Pareto frontier. We\nincorporate insights of compound relationships between model dimensions to\nbuild CompOFA, a design space smaller by several orders of magnitude. Through\nexperiments on ImageNet, we demonstrate that even with simple heuristics we can\nachieve a 2x reduction in training time and 216x speedup in model\nsearch/extraction time compared to the state of the art, without loss of Pareto\noptimality! We also show that this smaller design space is dense enough to\nsupport equally accurate models for a similar diversity of hardware and latency\ntargets, while also reducing the complexity of the training and subsequent\nextraction algorithms.\n