TaxBreak: Unmasking the Hidden Costs of LLM Inference Through Overhead Decomposition

Large Language Model (LLM) inference is widely used in interactive assistants and agentic systems. In latencysensitive deployments, inference time can become dominated by host-side overheads. Existing approaches typically expose this cost only as an aggregate residual or a launch/queue metric, which is often insufficient to identify which execution layer should be optimized. This work presents TaxBreak, a tracedriven methodology for decomposing host-visible orchestration overhead into three components: framework translation time, CUDA library translation time, and kernel launch-path time. We validate TaxBreak on NVIDIA H100 and H200 systems and use it to derive our proposed Host-Device Balance Index (HDBI), a boundedness summary index that relates device-active execution to host-visible orchestration. Across representative dense and mixture-of-experts workloads in both prefill and decode, we show that aggregate latency, GPU inactivity, or boundedness ratios alone can obscure the dominant optimization target. TaxBreak instead distinguishes cases where optimization should reduce software-stack overhead from cases where the primary win comes from reducing device-side work. We further show that MoE models dispatch $\mathbf{8 - 1 1} \times$ more kernels per output token than dense models, and that for such host-bound workloads, CPU singlethread performance is a first-order parameter: a faster host CPU reduces orchestration overhead by $\mathbf{1 0}-\mathbf{2 9} \boldsymbol{\%}$ and improves end-toend latency by up to $14 \%$, even when paired with a slowerclocked GPU. These results position TaxBreak as a diagnostic tool for assessing whether optimization effort should target the software stack or the device-side workload execution.