LLM Throughput Calculator

LLM/GPU · Throughput Calculator

MAX BATCHED THROUGHPUT · ROOFLINE MODEL

GPU Configuration

GPU Model

VRAM (GB)

Mem BW (GB/s)

BF16 TFLOPS

FP8 TFLOPS

FP4 TFLOPS

INT8 TOPS

⚠ A100 PCIe has no NVLink. Multi-GPU tensor parallelism uses PCIe Gen4 x16 (~64 GB/s) — expect significant TP overhead vs NVLink (600 GB/s). Single-GPU inference is unaffected.

GPU Cost ($/hr)

GPU Count (Tensor Parallel)

Model

HuggingFace Model ID

Params (B)

Layers

Hidden Dim

FFN Dim

Attn Heads

KV Heads

Head Dim

Context Len

Vocab Size

Active Params (B)

Experts (total)

Experts (active/tok)

Expert FFN dim

Precision

Expert precision

FP32

BF16

FP16

FP8

INT8

INT4

FP4

Other precision (attn + embed — usually BF16)

FP32

BF16

FP16

FP8

INT8

INT4

FP4

Batch Configuration

Batch Size

Seq Length

Throughput Scenarios

— configure GPU & model —

⚡

Select a GPU and enter model parameters
to compute throughput estimates.

Settings