Deep neural networks (DNNs) have become ubiquitous thanks to their remarkable ability to model complex patterns across various domains such as computer vision, speech recognition, robotics, etc. While large DNN models are often more accurate than simpler, lightweight models, they are also resource- and energy-hungry. Hence, it is imperative to design methods to reduce reliance on such large models without significant degradation in output accuracy. The high computational cost of these models is often necessary only for a reduced set of challenging inputs, while lighter models can handle most simple ones. Thus, carefully combining properties of existing DNN models in a dynamic and input-based way opens opportunities to improve efficiency without compromising accuracy. In this work, we introduce PERTINENCE, a novel runtime method to dynamically select, from a set of pre-trained models, the most lightweight one that correctly processes a given input. To achieve this, we use an ML-based input dispatcher that chooses the model to process the current input. We explore the training space of the input dispatcher using a genetic algorithm to identify the Pareto front of training solutions that offer trade-offs between overall accuracy and computational efficiency. We showcase our approach on state-of-the-art Convolutional Neural Networks (CNNs) trained on the CIFAR-10 and CIFAR-100 datasets, as well as Vision Transformers (ViTs) trained on the TinyImageNet dataset. Furthermore, we showcase PERTINENCE in a YOLO-based road occupancy estimation application that processes real-time video feeds from cameras installed at the road’s intersection. We report results showing PERTINENCE’s ability to provide alternative solutions to existing state-of-the-art models, balancing accuracy and the number of operations. The results show that, thanks to dynamic selection of pre-trained models, PERTINENCE achieves comparable or higher accuracy compared to the best pre-trained model, with up to 36% fewer operations with equivalent or lower end-to-end inference time using tunable invocation intervals.