APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs

Deploying large language models (LLMs) for online inference is often constrained by limited GPU memory, primarily due to the growing KV cache during autoregressive decoding. Hybrid GPU–CPU execution can relieve this pressure by offloading attention to the CPU, yet a key bottleneck remains: existing schedulers fail to effectively overlap CPU-offloaded work with GPU execution during the latency-critical, bandwidth-bound decode phase. This shortcoming especially penalizes real-time, decode-heavy applications (e.g., chat, Chain-of-Thought reasoning) under the memory pressure common in edge or low-cost deployments.We present APEX, a profiling-informed scheduling strategy that maximizes CPU–GPU parallelism for hybrid LLM inference. Unlike systems driven by static rules or ad-hoc heuristics, APEX predicts the execution times of CPU and GPU subtasks and dynamically dispatches them to maximize overlap while keeping runtime overheads low. Critically, APEX is the first system to enable fine-grained runtime CPU offloading without batch splitting, avoiding a common compromise that hinders performance in prior hybrid designs. APEX targets workload patterns typical of multi-turn conversation and reasoning, where prefill is brief but decode is long and performance-critical.We evaluate APEX on diverse workloads and GPUs (NVIDIA T4, A10) using LlaMa-2-7B and LlaMa-3.1-8B. Relative to GPU-only schedulers such as vLLM, APEX improves throughput by 84%–96% on T4 and 11%–89% on A10, while preserving latency. Against the strongest existing hybrid schedulers, it delivers up to 72% (T4) and 37% (A10) higher throughput in long-output settings. APEX advances hybrid LLM inference on memory-constrained hardware and offers a blueprint for scheduling on heterogeneous AI systems, filling a critical gap for efficient real-time LLM applications.