GPU Utilization vs Reality: Why You're Doing It Wrong
Your GPU dashboard says 100% busy. NVIDIA's own documentation says that number measures something else, and Meta's Llama 3 paper shows how much of a 16,000-chip cluster actually did the work. The Season 4 finale follows the missing capacity down the AI infrastructure stack, then back up to the three ways most people actually use it.
The Promise
- Batching is real capacity. On one Mac Studio running Qwen 3.8-27B, 16 requests at once produced about 51 tokens a second against 12 for a single request: four times the useful work from the same chip.
- Meta kept a 16,000-GPU training job doing useful work more than 90% of the time through 466 interruptions, by making checkpoints and restarts fast and automating diagnosis. Only three failures needed serious human intervention.
- The better metrics exist. Model FLOPs utilization, and the SM and Tensor Core activity that NVIDIA's own DCGM reports, measure work instead of presence.
- Inference on single servers doesn't need the expensive back-end fabric that lockstep training does. Knowing which one you run can change the network budget before anyone buys a switch.
The Risk
- NVIDIA's GPU-Util counts the share of time any kernel was running. A chip with a handful of its 100-plus streaming multiprocessors busy reads 100%.
- Meta's Llama 3 run used 38% to 43% of its chips' mathematical capacity. Google's PaLM landed at 46%. These are two of the best-documented training runs in the industry.
- Lockstep training waits on its slowest link. One congested port stalls thousands of GPUs, and buying another rack in response adds exchanges, adds waiting, and lowers useful work per chip.
- If you buy AI through an API, the harness writes most of your bill. A badly designed agent calls the model ten times for work a script could do once.
One column, one line of documentation
Open a terminal on any NVIDIA GPU server and run nvidia-smi. One column says GPU-Util. When it reads 100%, almost everyone reads it as a chip working flat out: the engineer, finance, and whoever approved the cluster.
NVIDIA wrote down what that number measures. Its NVML documentation defines GPU utilization as the percent of time over the sample period during which one or more kernels was executing on the GPU. A kernel is one unit of work. The sample period runs between a sixth of a second and a second, depending on the card. So the column answers one question: during that slice of time, was anything running at all?
It says nothing about how much of the chip was running it. An H100 is built from more than 100 streaming multiprocessors (SMs). Keep a handful of them busy for the whole sample window and the column reads 100% while the rest sit idle.
The number that answers the real question is model FLOPs utilization, MFU: the math your model needed, divided by the math your chips could have done in the same time. Meta published it for Llama 3, trained on more than 16,000 H100s. The result was 38% to 43%. Google’s PaLM, two years earlier, landed at 46% on its own chips. The rest of the time, the most expensive hardware on the planet was waiting.
For 15 years in security, the headline metric was agent coverage. 98% of laptops have the tool installed, the board slide goes green, and nobody asks how many attacks it stopped. A number that measures presence gets reported as if it measured performance. GPU-Util is the same number, and each point costs more.
Where the waiting goes
Big models train on thousands of chips in lockstep. Every chip computes its share of a step, then swaps results with every other chip before the next step starts. The step ends when the slowest exchange ends, so one congested link holds up the whole cluster. More chips means more exchanges and more chances for one of them to be the slow one.
Lockstep has a second cost, and Meta counted it. Over 54 days of training, the job was interrupted 466 times; 419 of those were unplanned. That is roughly one failure every three hours, with the best engineers in the industry watching. A single GPU failure can force a restart of the whole job, so every failure costs the work since the last checkpoint plus the restart. Meta still kept the run doing useful work more than 90% of the time.
Storage is next. A GPU can only compute on data that has arrived, and for inference that now includes the KV cache, the model’s working memory of a long conversation, which spills to storage when chip memory fills. WWT’s Eric Fairfield, on the AI Proving Ground podcast, put it plainly: “Storage is often forgotten until the last minute, and when it can’t keep up, you starve the GPUs.”
Then the software. A scheduler such as Slurm or Kubernetes reserves GPUs for teams. A reserved GPU nobody is using looks spoken for on every dashboard.
What batching does to one chip
When a model writes an answer, it produces one token at a time and reads its entire set of weights out of memory for each one. That step is limited by memory speed, so the math units spend much of it idle. A chip serving a chatbot can read 100% busy while its math capacity mostly waits. Batching is the fix: one trip to memory serves many users.
I measured it on a Mac Studio with an M2 Max, running Qwen 3.8-27B on MLX. One request at a time: about 12 tokens a second. 16 at once: about 51 in total. Each individual answer slowed to about three tokens a second. That is the trade every AI provider makes for you.
The curve isn’t straight. Eight requests ran slower than four, every time. A finer sweep put the change between five and six. Up to five, the chip keeps each answer quick and the total tops out around 34 tokens a second. From six on, each answer is slow, and every extra request adds output almost for free. Nothing about the chip changed between those runs; the software in front of it decided how much work it did. The method and scripts are in Labs.
If you buy it through an API
Most people never touch a GPU. They reach it through the harness: the software around a model that builds the prompt, decides which tools it can call, runs them, and decides when to stop. That layer decides whether the agent you’re buying is an agent at all, and it decides how often the model gets called.
A chatbot is the memory-speed case, and you see the provider’s batching as price and speed. A coding agent resends files, test output and session history on every turn, so it can burn more compute rereading context than writing code. That is why prompt caching matters, and why turning it on without watching the hit rate can raise the bill. An agentic system plans, calls a tool, and waits. At the bottom of the stack chips wait on the network; at the top, the agent waits on its own tools.
The verdict
The waste is real and documented, and so are the fixes. The needle leans slightly toward risk because of how the number gets used. A training job slows, every GPU reads 100%, and the team buys another rack. If the bottleneck was a congested link or storage, the rack makes it worse: more chips in lockstep, more exchanges, less useful work per chip.
In my experience, the most expensive decisions in technology get made on a metric nobody questioned. In security it was coverage. Here it is GPU-Util.
One. What does our GPU-Util number measure, and is anyone watching SM or Tensor Core activity?
Two. When a job slows down, can we tell whether the chip, the network or storage is doing the waiting?
Three. What does each useful result cost us?