04 Deep dive

CPU-side scheduling and accelerator utilization

Kubernetes assigns a GPU to a pod exclusively, but the CPU threads that feed it are scheduled under ordinary CFS weights alongside every other pod on the node. We measure the effect of CPU-side contention on accelerator utilization using ResNet-18 training on an NVIDIA L4 (GKE g2-standard-8) and vLLM serving on the same hardware. Under CFS, 16 batch neighbors reduced training throughput by 25% and collapsed GPU utilization; under kernel-enforced tiers, throughput and GPU utilization held flat. Small-model vLLM serving showed parity in both arms; the workload is GPU-bound and the result is reported as a negative finding.

workloads: PyTorch training (ResNet-18) · vLLM serving hardware: GKE g2-standard-8 + NVIDIA L4 · c3-standard-8 records: 4 reports method: benchmark methodology

Background: scheduling asymmetry on GPU nodes

Kubernetes treats a GPU as an indivisible, exclusively assigned resource: one pod owns nvidia.com/gpu: 1 and no other pod can use it. The CPU side of the same training pod — the main loop, the DataLoader workers performing decode and augmentation, the copy threads — receives no such treatment. It competes under ordinary CFS weights with every other pod on the node. When batch neighbors outweigh the trainer, the feeder threads stall, the input pipeline drains, and the GPU idles while it remains reserved and billed.

The precondition is stated up front: this starvation mode exists when the trainer's CPU demand exceeds its request. That configuration is common in practice — DataLoader-heavy training pods are frequently provisioned with small CPU requests next to expensive GPUs — but it is a condition, not a universal property.

Results: ResNet-18 training on NVIDIA L4

A control configuration bounds the effect. With the trainer Guaranteed and demand that fit inside its 4-CPU request, both arms were flat (~616–643 samples/s, GPU ~80–85%): kubelet's QoS weighting fully defends a Guaranteed pod whose demand fits its request, and Temper added only +3–4% at high density. The control run is retained in the report because it defines the boundary condition: the starvation effect appears when demand exceeds request (measured configuration: 2-CPU request, ~7 vCPU demand), the point at which proportional weights favor the aggregate batch neighbors over the trainer. In utilization terms, at 16 neighbors CFS forfeits approximately a quarter of the accelerator's throughput; with enforcement the same node carries the batch overflow with no trainer degradation.

Comparison with quota limits and static CPU pinning

Negative result: GPU-bound serving

The serving analogue of the training experiment was run and shows no effect. vLLM serving a small model (Qwen2.5-0.5B) on the same L4, 8 concurrent request loops, in both a right-sized Guaranteed configuration and a deliberately starvation-shaped one (demand over request, weighted neighbors), measured parity in both. Right-sized, both arms were flat — p99 ~393–411 ms at idle and ~398–404 ms at bg=8, throughput within a few percent. In the starvation-shaped configuration, CFS showed only mild degradation at bg=4 (+16% vs Temper's +7%) with throughput parity, over single, noisy runs; the bg=0 rows include post-rollout warm-up effects and the record marks them unsettled.

The mechanism is consistent with the training result rather than in tension with it: tokenization and scheduling for a 0.5B model at this concurrency cost a fraction of one core, so the workload is GPU-bound and there is no CPU-side contention for a CPU scheduler to remove. The training result therefore does not generalize to this serving shape. The CPU-side effect applies to DataLoader-heavy training and preprocessing-heavy pipelines (long prompts, large tokenizers, multimodal encode) — configurations in which substantive CPU work sits between storage and the accelerator — and does not apply to workloads whose CPU side is negligible.

Limitations

• Single run per arm in all GPU experiments; 60 s measurement windows.
• The +67% arm comparison is a CPU-only node (c3-standard-8, Intel); absolute samples/s are not comparable across machine shapes — a t2d (AMD, no SMT) run of the same trainer idles at 17.2 vs c3’s 25.5.
• The training effect is conditional on trainer CPU demand exceeding its request; with a Guaranteed trainer whose demand fits its request, both schedulers hold throughput flat.
• Small-model vLLM serving on this hardware is GPU-bound and shows no benefit from CPU-side enforcement in either tested configuration.
• The L4 run also confirmed that the node agent attaches the scheduler at node startup, before any user pod is scheduled on the GPU node.