Batching on Apple Silicon
How much more does one chip produce when it answers several requests at once?
About four times as much, and every answer gets slower. On a Mac Studio with an Apple M2 Max, Qwen 3.8-27B produced 12.4 tokens per second for one request and 51.5 tokens per second in total for 16 requests at once. Each of those 16 requests ran at 3.2 tokens per second.
- Measured on
- Mac Studio · Apple M2 Max · 96 GB
- Model
- Qwen 3.8-27B · MLX 8-bit
- Published
12.4 tokens per second. That is what a 27-billion-parameter model produces on a Mac Studio when one person asks it one question. Ask it 16 questions at the same time and the same chip produces 51.5.
Nothing about the hardware changed between those two numbers. The difference is software: the inference engine groups the requests into one batch and works through them together.
The result
Qwen 3.8-27B at 8-bit, running on MLX. Every request gets a 128-token prompt and writes 128 tokens back. Each figure is the median of three full runs, and no step moved by more than 0.4% between runs.
| Requests at once | Total tokens/sec | Per request tokens/sec | vs one request |
|---|---|---|---|
| 1 | 12.4 | 12.4 | 1.0x |
| 2 | 22.7 | 11.3 | 1.8x |
| 4 | 32.6 | 8.2 | 2.6x |
| 8 | 27.1 | 3.4 | 2.2x |
| 16 | 51.5 | 3.2 | 4.2x |
Two requests almost double the output for almost no cost to either user. At 16, the chip does 4.2 times the work, and each person waiting on an answer gets it at a quarter of the speed.
The cliff at six requests
Eight requests came out slower than four. In every run. So I ran every batch size from four to 12 to find where it happens.
| Requests at once | Total tokens/sec | Per request tokens/sec | Time per step |
|---|---|---|---|
| 4 | 32.7 | 8.2 | 0.12 s |
| 5 | 33.9 | 6.8 | 0.15 s |
| 6 | 20.4 | 3.4 | 0.29 s |
| 8 | 27.2 | 3.4 | 0.29 s |
| 12 | 39.6 | 3.3 | 0.30 s |
| 16 | 51.4 | 3.2 | 0.31 s |
A step is one round in which every request gets one more token. Up to five requests, a step takes about 0.15 seconds. At six it jumps to 0.29 seconds, then barely moves all the way to 16.
That gives two regimes. Below six, answers stay reasonably quick and total output flattens at about 34 tokens per second. From six up, each answer crawls at about 3.3 tokens per second, but every extra request adds almost free output, in a straight line.
The same dip shows up with Mistral Small 3.2-24B, which has no linear-attention layers, so it is not a quirk of Qwen’s hybrid design. It points at how MLX schedules the work at that batch size. I have not confirmed the cause in the MLX source, and until I do, it stays an observation.
What it means if you run AI
The throughput number and the waiting number are different numbers, and a vendor will quote the first one.
Anyone who has sized a firewall has seen this. Rated throughput gets measured with the queue full. Your users experience the queue from the inside.
So the question to ask of any AI deployment is what the work looks like. An overnight job summarising 10,000 documents wants the biggest batch the memory allows. A tool a person sits and waits on wants small batches, or more hardware, because at 3 tokens per second a 300-word answer takes more than two minutes.
Same chip. Same model. A fourfold difference in output, decided by a setting most buyers never see.
How it was measured
| Machine | Mac Studio (2023), Apple M2 Max, 38-core GPU, 96 GB unified memory |
| OS | macOS 27.0 |
| Model | Qwen 3.8-27B (lmstudio-community/Qwen3.8-27B-MLX-8bit), dense, 8-bit |
| Engine | mlx-lm 0.31.3 on MLX 0.32.2, stock mlx_lm.benchmark |
| Workload | 128-token prompt, 128 generated tokens per request |
| Runs | Three full runs, 24 September 2026, median reported |
The machine had just been upgraded to macOS 27, and the system’s photo analysis kept the processor busy for most of a night. The runs waited until the load average was under ten for two checks in a row, 20 minutes apart. Peak memory was 33 GB, with no swap.
Total tokens per second counts generated tokens across all requests, divided by generation time. Prompt processing is measured separately and is not in that figure.
Two other models ran the same workload once each, as a check on the shape:
| Model | 1 at once | 4 | 8 | 16 | Gain at 16 |
|---|---|---|---|---|---|
| Qwen 3.6-27B | 12.2 | 32.5 | 27.1 | 50.7 | 4.2x |
| Mistral Small 3.2-24B | 14.4 | 38.3 | 31.9 | 61.0 | 4.2x |
Those are single passes, not three-run medians.
This is one machine, one workload shape and one engine version. Longer prompts, longer answers, a different quantisation or a new MLX release will all move the numbers. The shape is the finding.
Run it yourself
The scripts and the unedited output of every run are in the promise-and-risk-labs repository on GitHub. On an Apple silicon Mac with enough memory:
pip install -r requirements.txt
./scripts/full-run.sh
python3 scripts/parse.pyIf your numbers come out different, I want to see them.