How GPU AI Benchmarks Are Calculated: TFLOPS, Tokens per Second, and What the Numbers Hide

In early 2026, NVIDIA published an inference record of 2.5 million tokens per second. Impressive, until you read the footnote: the number came from 288 Blackwell Ultra GPUs working together across four full racks, and dividing it per chip is, in MLCommons' own framing, not an official metric. That is GPU benchmarking in one story: the numbers are real, and they still will not tell you what you think unless you know exactly how they were measured.

This article explains the machinery behind AI GPU benchmarks: what actually gets measured, why the TFLOPS on a spec sheet fail to predict AI performance, the two-phase physics that decides tokens per second, the four variables that swing any result several-fold, how the industry-standard MLPerf suite works, and, in full, how the AI Inference Index behind our own 78-GPU benchmark tool is calculated. By the end you will be able to sanity-check any published GPU number with one napkin formula, which is a skill worth real money when hardware is this expensive.

Strip away the branding and every serious AI benchmark measures one thing: how much useful work a GPU completes per unit of time on a defined workload. The workloads fall into three families, and a card's ranking can change completely between them.

LLM inference measures token generation: how fast a model like Llama or DeepSeek produces output, reported as tokens per second, time to first token, and latency between tokens. Image generation measures diffusion throughput, samples or images per second on models like SDXL. Vision and multimodal covers image understanding, OCR, and detection models under load. Training benchmarks form a fourth family with a different logic entirely: instead of throughput at a moment, they measure time-to-result, how long the hardware needs to train a reference model to a target quality. A GPU that dominates one family can be mid-table in another, which is why our benchmark tool scores all three inference families rather than crowning one champion from one test, and why the right card for training is a separate question from the right card for serving.

TFLOPS, trillions of floating-point operations per second, is the number everyone quotes and the worst single predictor of real AI throughput. Three reasons, in ascending order of importance.

First, peak TFLOPS is a laboratory number. It assumes perfectly shaped matrix multiplications hitting every tensor core every cycle, which production code never achieves. Engineering measurements put the maximum achievable matrix throughput of an H100 at roughly 794 TFLOPS against the 989 on the datasheet, a fifth of the headline gone before your workload even starts. Second, marketing TFLOPS often includes structured sparsity, a 2:4 pruning feature that doubles the printed figure but applies only to models specifically prepared for it, which most are not. An L40S, for example, is quoted at 1,466 FP8 TFLOPS with sparsity and 733 without; the honest number is the smaller one. Third, and decisively: most AI work is not compute-limited at all. It is limited by how fast data can move from memory to the cores, and that ceiling appears nowhere on the TFLOPS line.

The cleanest evidence is a comparison every buyer knows: the H100 delivers roughly 2.8 times the LLM inference throughput of an A100. Its paper compute advantage is over 3x, its memory bandwidth advantage is about 1.7x, and the delivered result tracks the bandwidth number plus the FP8 Transformer Engine far more closely than the FLOPS number. The same pattern holds down the stack, and it is why the following section is the single most useful mental model in GPU benchmarking.

Every LLM request runs in two phases with opposite bottlenecks, and every credible inference benchmark reports them separately.

Prefill processes your entire prompt in one parallel pass. It is compute-bound, so this is where TFLOPS genuinely matter, and it determines time to first token (TTFT), the pause before the answer starts. A 10,000-token prompt takes proportionally longer than a 100-token one on the same card. Decode then generates the answer one token at a time, and here the physics turns brutal: because language models are autoregressive, producing each token requires reading the entire set of model weights, plus the growing key-value cache, out of memory. At small batch sizes that works out to only a few floating-point operations per byte moved, which places decode deep in memory-bound territory. The metric it determines, tokens per second, therefore scales with memory bandwidth almost regardless of compute.

This split explains most benchmark surprises. It is why an RTX 5090 with 1.79 TB/s of GDDR7 outpaces cards with more TFLOPS but less bandwidth when running LLMs locally, and why single-number rankings keep misleading buyers whose workloads live in one phase.

It also explains the direction of the whole industry. The H200 and B200 lead their generations with memory upgrades rather than raw compute, as the data-center trio comparison shows, and NVIDIA built Rubin around HBM4 at 22 TB/s: nearly tripling bandwidth is worth more to inference than any amount of additional FLOPS.

The memory-bound nature of decode hands you something rare in this industry: a speed limit you can compute on a napkin. For a single stream, tokens per second cannot exceed memory bandwidth divided by the bytes the model occupies, because every token requires one full read of those bytes.

Work the examples. An 8-billion-parameter model at FP16 occupies about 16 GB, so an H100 at 3.35 TB/s tops out near 209 tokens per second for one user; an A100 at 2.0 TB/s near 125. Quantize the same model to INT4 (about 4.5 GB) and the ceilings roughly quadruple, not because the card got faster but because each token now moves fewer bytes. A 70B model at FP16 needs about 140 GB and simply does not fit an 80 GB card, which is why 70B benchmarks always involve either quantization or multiple GPUs, and why VRAM appears next to every score in our tables. Well-optimized real systems deliver 60 to 85% of the theoretical ceiling; published numbers that beat it are telling you batching, speculative decoding, or a smaller model than advertised is in play. One formula, and vendor slides can no longer surprise you.

Identical GPUs post wildly different numbers across sources, and it is rarely anyone lying. Four settings do the swinging, and a comparison is only valid when all four match.

Batch size trades individual speed for total throughput: at small batches decode is memory-bound and serving more users is nearly free, while at large batches the workload turns compute-bound and each user slows down. Continuous batching engines like vLLM live in that trade-off. Precision changes bytes moved: FP8 roughly doubles decode against FP16, INT4 quantization (GPTQ, AWQ, GGUF) can triple it, with quality costs that depend on the task. Context length feeds the KV cache, which at 32K+ tokens can consume as much VRAM as the model weights themselves and visibly drags on speed. And the software stack is the quiet giant: the same silicon produces materially different numbers under vLLM, TensorRT-LLM, llama.cpp, or Ollama, the engines behind most local LLM setups.

Training makes the stack effect measurable through one elegant metric: MFU, model FLOPs utilization, the share of a GPU's theoretical compute your training run actually uses. The published spread is startling. A naive off-the-shelf training loop has been measured at 7.7% MFU on an A100, the GPU idle more than nine seconds in every ten. Meta's Llama 65B run achieved 49.5% across 2,048 A100s; Llama 3.1 reported 38 to 43% on H100 clusters; and a well-tuned run has hit 56% of the H100 datasheet peak. Same silicon, six-fold difference, purely software. When a training benchmark reports images or tokens per second, MFU is the honesty check behind it.

MLPerf, run by the MLCommons consortium, is the closest thing AI hardware has to an independent scoreboard, and its design choices teach you what rigorous benchmarking looks like. Vendors submit results on fixed reference models under audited rules, in versioned rounds (Inference v5.1 and v6.0 are the 2026 vintage). The Closed division locks the model and quality target so hardware comparisons are clean; the Open division allows optimizations. Inference runs in scenarios: Offline measures maximum throughput with all queries available at once, while Server fires queries at random intervals and enforces tail-latency limits on first-token and per-token response, so a card only scores what it can serve while staying fast for the slowest users. Training benchmarks measure time-to-train a reference model to a defined quality, the metric that actually maps to a cloud bill.

MLPerf results still need reading skills. The 2.5-million-tokens-per-second record that opened this article was a genuine v5.1/v6.0 closed-division result, and it was also a four-rack, 288-GPU aggregate; per-chip arithmetic on it is explicitly not an official metric. Recent rounds also show how workload-dependent rankings are: the H200 roughly doubles the H100 on a detection model, the B200 doubles the H200 on image generation, yet on compute-bound detection at modest concurrency a humble L40S competes far above its price class. Whole-system results, meanwhile, are as much about the server around the GPUs and the interconnect as the chips themselves, which is why rack-scale systems are benchmarked as units.

Our GPU and AI server benchmark tool compares 78 GPUs and 48 servers on a single 100-point scale, and after everything above, you can now audit exactly how that scale is built.

The index starts from publicly published GPU-cloud benchmark data, numbers anyone can verify, rather than in-house tests nobody can audit. Each card is scored across three workload families: LLM inference from lightweight 8B assistants to 70B+ models, diffusion image generation from turbo pipelines to production renders, and vision plus document OCR under concurrent load. The three scores aggregate into one number indexed to the RTX 3090 at exactly 100 (936 GB/s, 24 GB: a baseline enough readers have owned to feel intuitively), so a card scoring 200 delivers roughly twice a 3090's combined throughput. Training throughput is deliberately kept separate, shown as an independently published single-GPU figure only where a matching benchmark exists for that exact card, because blending training into an inference index would smuggle in exactly the apples-to-invoices comparison this article warns against.

The limits are stated on the tool itself, and they matter: one aggregate number cannot capture every workload, results are orientation rather than guarantees, and your stack, drivers, and batch sizes will move your outcome. That is also why the tool pairs the index with raw VRAM, TFLOPS, and a head-to-head mode, and why it links the plain-language GPU choosing guide for readers who want the decision, not the methodology.

Everything in this article compresses into five questions. Same workload? Inference, image generation, and training rank cards differently. Same precision? FP8 versus FP16 alone can double a score. Same batch and context? Batch-32 throughput is not batch-1 speed. Per chip or per system? Headline records are rack aggregates. Peak or delivered? Even the datasheet peak of an H100 is unreachable in practice; trust measured throughput. Run any vendor slide, cloud pricing page, or yes, our own table, through those five, and the misleading comparisons identify themselves. Most fail at least two.

Three honest limits, because a methodology article that hides its own would be absurd. First, benchmarks measure reference workloads, and your model, your prompts, and your traffic pattern are not reference workloads; the only benchmark that fully predicts your throughput is your workload on rented hardware for an afternoon. Second, numbers age: driver releases and engine updates routinely move inference results by double-digit percentages between rounds, so treat anything older than a couple of quarters as history. Third, throughput is not economics. A card that wins tokens per second can lose cost per token once price, power, and utilization enter, which is the domain of hosting math and facility-scale planning covered in our AI data center pillar, and the reason the A100 vs H100 decision turns on cost per finished job rather than any single benchmark line. Benchmarks tell you what hardware can do. What it should cost you to do it is a second calculation, and both deserve real numbers.

GPU AI benchmarks are calculated by measuring delivered work on defined workloads: tokens per second and first-token latency for LLM inference, samples per second for image generation, time-to-train for training, aggregated under stated rules like MLPerf's scenarios or our RTX 3090 = 100 index. The numbers are trustworthy precisely to the degree the methodology is visible, which is why we publish ours. Remember the physics, prefill is compute, decode is bandwidth; remember the napkin formula, bandwidth over model bytes; check the four swing variables before comparing anything; and then let the benchmark tool shortlist your hardware and the GPU shop price it. Measured honestly, the numbers stop being marketing and start being engineering, and engineering is the part you can actually buy.

How are GPU AI benchmarks calculated?
By measuring delivered throughput on defined workloads rather than theoretical capability: tokens per second and time to first token for LLM inference, images or samples per second for generation and vision tasks, and time-to-train to a target quality for training. Serious benchmarks fix the model, precision, batch size, and quality target, state the software stack, and publish results in audited rounds (MLPerf) or transparent indexes so comparisons stay valid.

What is a good tokens-per-second speed?
For a single user, anything above roughly 20 to 30 tokens per second reads faster than most people can follow, so local setups target that as a floor. Serving infrastructure thinks differently: an H100 tops out near 209 tok/s per stream on an 8B FP16 model (the bandwidth ceiling) but serves thousands of aggregate tokens per second across batched users. The right target depends on which you are optimizing: one user's experience or a fleet's throughput.

What is TFLOPS and why does it not predict AI performance?
TFLOPS is trillions of floating-point operations per second, a theoretical peak assuming perfect conditions that production code never reaches (an H100's achievable matrix throughput measures about 794 TFLOPS against 989 on the datasheet, and marketing figures often double again via sparsity). More fundamentally, LLM token generation is limited by memory bandwidth, not compute, so a card's TFLOPS line can be almost irrelevant to its tokens per second.

Why is memory bandwidth so important for LLM inference?
Because language models generate autoregressively: producing every single token requires reading the entire set of model weights, plus the KV cache, out of GPU memory. That makes the decode phase memory-bound, and tokens per second scales with bandwidth divided by model size. It is why the H100's 2.8x real inference advantage over the A100 tracks its bandwidth and FP8 support rather than its raw compute, and why HBM upgrades headline every new generation.

What is the difference between prefill and decode?
Prefill is the phase that processes your prompt: all input tokens in one parallel pass, compute-bound, determining time to first token. Decode is the phase that writes the answer: one token at a time, memory-bound, determining tokens per second. A GPU can be strong in one phase and weak in the other, which is why single-number inference benchmarks hide as much as they show and rigorous ones report both metrics.

What is MLPerf and who runs it?
MLPerf is the industry-standard AI benchmark suite run by MLCommons, a consortium spanning chipmakers, clouds, and research groups. Vendors submit results on fixed reference models under audited rules in versioned rounds (Inference v5.1 and v6.0 in 2026). The Closed division locks models for clean hardware comparison, the Server scenario enforces latency limits so throughput must coexist with responsiveness, and training benchmarks measure time-to-train to a defined quality.

What is MFU in AI training?
Model FLOPs Utilization: the share of a GPU's theoretical compute that a training run actually uses, calculated from achieved token throughput against peak FLOPS. Published values span 7.7% for naive training loops to 49.5% for Meta's Llama 65B run, with Llama 3.1 reporting 38 to 43% on H100 clusters. That six-fold spread on identical silicon is why the software stack is as much a benchmark variable as the hardware.

Does quantization make a GPU faster?
It makes inference faster on the same GPU by shrinking the bytes each token must move: FP8 roughly doubles decode speed against FP16, and INT4 methods like GPTQ and AWQ can triple it, because the memory-bound decode phase scales with model size in bytes. The GPU's hardware is unchanged; the workload got lighter. Quality impact depends on the model and task, which is why honest benchmarks always state the precision used.

Why do different websites show different benchmarks for the same GPU?
Because four settings swing results several-fold: batch size (single-user speed versus batched throughput), precision (FP16 versus FP8 versus INT4), context length (KV cache load), and software stack (vLLM, TensorRT-LLM, llama.cpp, and Ollama all perform differently on identical silicon). Two honest sites testing different configurations will publish different numbers. Before trusting a comparison, confirm all four settings match.

How is the MillionMiner AI Inference Index calculated?
Four steps: collect publicly published GPU-cloud benchmark data (verifiable, no in-house guesswork); score each card on three workload families (LLM inference from 8B to 70B+, diffusion image generation, and vision plus OCR under load); aggregate onto one scale indexed to the RTX 3090 at 100, so 200 means roughly twice a 3090's combined throughput; and report training throughput separately, only where an independently published figure exists for that exact card. Assumptions and limits are stated on the tool itself.

Should I trust vendor GPU benchmarks?
Trust them as inputs, not verdicts. Vendor numbers are usually real measurements taken under the most favorable configuration: best-case precision, sparsity enabled, rack-scale aggregates, latest software. Apply the five-question test (same workload, precision, batch and context, per-chip versus per-system, peak versus delivered) and cross-check against independent sources like MLPerf rounds. When a claim survives all five questions, it is probably useful.

Sources and notes
MLPerf results and rules from MLCommons (Inference v5.1 and v6.0, 2026 rounds); the 2.5M tokens/sec figure is NVIDIA's closed-division submission on 288 Blackwell Ultra GPUs, with per-chip division not an official metric. MFU figures from published training reports: Meta's Llama 65B (49.5%) and Llama 3.1 (38 to 43% on H100), plus published engineering measurements of naive-stack baselines (7.7%) and achievable H100 matrix throughput (~794 of 989 TFLOPS). Bandwidth and capacity figures from NVIDIA specifications. Tokens-per-second ceilings are theoretical batch-1 maxima; real systems deliver 60 to 85% of them. All figures are mid-2026 snapshots; benchmark results move with software releases. Hero: Steve Jurvetson, CC BY 4.0; body: Wikimedia Commons, CC0. Diagrams: original MillionMiner graphics, free to reuse with attribution and a link to this page. Informational content, not purchasing advice