HYBRIDSERVE: Efficient Serving of Large AI Models with Confidence-Based Cascade Routing

Giant Deep Neural Networks (DNNs) are central to modern cloud-based AI services but are extremely energyintensive to serve - often exceeding their training cost - due to the need for large GPU clusters to host model partitions and replicas. Existing methods trade off either energy efficiency or inference accuracy. We present HYBRIDSERVE, a hybrid DNN serving system that leverages multiple model sizes - from small to giant - and routes requests based on confidence estimates. By prioritizing smaller, energy-efficient models for confident queries, HYBRIDSERVE reduces reliance on giant model replicas while preserving accuracy. A dataflow planner further optimizes model placement and replication to maximize throughput. Experiments show that HYBRIDSERVE reduces energy consumption by up to $19.8 \times$ over state-of-the-art serving systems, while achieving the same accuracy as the largest model alone.