GPU Cost Per Token in 2026: the complete math for H100, B200 and MI355X
If you serve an LLM on rented GPUs, exactly one number decides whether your unit economics work: dollars per million output tokens. Not TFLOPS, not MMLU, not the hourly sticker price. This guide shows you how to compute it from first principles for any model on any GPU, why most public comparisons get it badly wrong, and what the numbers actually look like in mid-2026.
The only formula that matters
$/1M tokens = (GPU hourly price × number of GPUs) ÷ (tokens per second × 3600) × 1,000,000
Price is public. GPU count is a VRAM fit problem. The hard part is tokens per second — and this is where stale blog benchmarks lead teams to overprovision by 2–10×. You do not need to benchmark every GPU to estimate it well. You need a roofline.
Step 1: decode speed is a memory problem
Autoregressive decoding reads every active model weight once per token. So a single stream decodes at roughly memory bandwidth × utilization ÷ bytes of active weights. A 70B model at FP8 is ~70 GB of weights. An H100 SXM moves 3.35 TB/s; at a realistic 55% memory-bandwidth utilization (MBU) on a tuned vLLM or TensorRT-LLM stack, that is 3350 × 0.55 ÷ 70 ≈ 26 tokens/sec per stream. Notice compute never entered the picture — for single-stream decode, TFLOPS are almost irrelevant. This is why an MI300X (5.3 TB/s) outruns an H100 on per-user speed for large dense models despite the marketing narrative.
Step 2: batching is where margins live
Weights are read once per step regardless of how many sequences share it, so throughput scales nearly linearly with batch size until you hit the compute roofline: dense TFLOPS × MFU ÷ (2 × active params). A chat product needing 25 tok/s per user might run batch 4–8; an offline summarization pipeline can push batch 64 and get an order of magnitude better $/token from the same hardware. When someone quotes “$X per million tokens” without stating a latency floor, the number is meaningless.
Step 3: MoE breaks naive calculators
DeepSeek V3 is 671B parameters total but activates only ~37B per token. VRAM is sized by the total (you must hold all experts); speed is set by the active subset. Calculators that use one number for both either rank MoE models 18× too expensive (using 671B for speed) or claim they fit on a single 4090 (using 37B for memory). Get this wrong and every MoE-vs-dense decision you make is garbage.
Mid-2026 price reality
The market moved fast in the last year: B200 supply loosened, AMD's MI355X (288 GB, ~8 TB/s) started undercutting NVIDIA on $/GB of HBM, and marketplace pricing (Vast.ai) keeps a 30–50% discount below on-demand clouds for interruptible work. Rough on-demand anchors as of 2026-06: H100 from ~$1.9–3/hr (marketplace to Lambda), H200 ~$2.6–4/hr, B200 ~$5–6/hr, MI300X ~$1.9–2.5/hr, MI355X ~$4/hr at specialist clouds — while hyperscaler list prices for the same silicon still run 3–4× higher. Three months from now some of these will be wrong, which is precisely the point: check a live dataset, not a blog post — including this one.
A worked example: Llama 3.3 70B, FP8, 500M tokens/month
Balanced latency (≥10 tok/s per user, batch 16). One MI300X holds the 73 GB of weights plus KV cache; roofline gives ~660 tok/s aggregate. At $2.49/hr on-demand that is (2.49 ÷ (660 × 3600)) × 10⁶ ≈ $1.05 per 1M tokens, about $525/month for the volume. The same workload on 2× H100 at Lambda lands near $1.9/1M — nearly 2× the cost, invisible if you only compared hourly prices ($2.49 vs $5.98 looks like a 2.4× gap, but the throughput difference eats half of it). Marketplace pricing flips the ranking again. This is why the calculation has to be re-run, per model, per month.
The five mistakes that make estimates 10× wrong
1) Using hourly price as a proxy for cost per token. 2) Ignoring the latency floor and comparing batch-64 throughput against a realtime requirement. 3) Sizing MoE by total params for speed. 4) Assuming 100% MBU/MFU — real serving stacks sustain 40–60%. 5) Trusting benchmark numbers from a different quantization, context length, or serving framework than yours.
Do it in ten seconds instead
We maintain exactly this math — roofline model, MoE-aware sizing, per-offer price freshness stamps across Lambda, RunPod, Vast.ai, AWS, GCP, Vultr and TensorWave — as a free calculator. Run your model through it, and if you want alerts when a price shift changes your answer, Pro watches your stack monthly.