Labs.
Claims about AI performance are cheap. Measurements are not. Each lab takes one question, runs it on hardware you could buy, and publishes the raw output and the code alongside the result.
- Mac Studio · Apple M2 Max · 96 GB
Jais 3 on Apple Silicon
Do IFM's Jais 3 32B and 36B-A4B run on a 96 GB Mac Studio, does IFM's Arabic accuracy claim hold, and what does Arabic cost in tokens?
Two of the six Jais 3 models on one Mac Studio: memory, speed, wait and power from 1K to 128K tokens, ArabicMMLU on 100 questions scored with IFM's own code, a tokenizer count, and the two things that would not run.
- Mac Studio · Apple M2 Max · 96 GB
Clef on Apple Silicon
Does Cloudflare's open decision model run on a Mac Studio, and what does shrinking it to 4-bit change?
Seven configurations of Clef and Clef-flash on one Mac Studio: full precision, 8-bit and 4-bit, on MLX and PyTorch. What fits, how fast it decides, and the one safety decision that 4-bit changed.
- Mac Studio · Apple M2 Max · 96 GB
Batching on Apple Silicon
How much more does one chip produce when it answers several requests at once?
Same chip, same model, one to 16 requests at once. Total output rose 4.2 times, each answer slowed to a quarter of its speed, and the curve has a cliff at six requests.