hipinfer
nvcc for NVIDIA and hipcc for AMD. These are its measured results.Batch-1 decode
Generating one token at a time is memory-bound: every token reads all the weights once. The bandwidth floor
is the bytes read per token divided by the GPU's peak bandwidth, the fastest any engine could go. RTX 2080 Ti, fp16 weights,
128-token prompt. llama.cpp is llama-bench on an F16 GGUF, best of flash-attention on and off.
Data table
| model | hipinfer ms/token | llama.cpp ms/token | bandwidth floor ms | % of floor |
|---|---|---|---|---|
| TinyLlama-1.1B | 4.14 | 4.77 | 3.37 | 81% |
| Qwen2.5-0.5B | 2.34 | 3.45 | 1.61 | 69% |
| Llama-3.2-1B | 4.75 | 5.23 | 4.03 | 85% |
| Qwen2.5-1.5B | 6.21 | 7.19 | 5.02 | 81% |
| Llama-3.2-3B | 12.11 | 12.70 | 10.48 | 87% |
Data table
| model | ctx 128 | ctx 512 | ctx 1,024 | |||
|---|---|---|---|---|---|---|
| TinyLlama-1.1B | 4.13 ms (81%) | 4.17 ms (81%) | 4.22 ms (81%) | |||
| Qwen2.5-0.5B | 2.33 ms (69%) | 2.38 ms (68%) | 2.42 ms (67%) | 2.50 ms (66%) | 2.64 ms (64%) | 2.73 ms (65%) |
| Llama-3.2-1B | 4.77 ms (84%) | 4.80 ms (84%) | 4.84 ms (84%) | 4.90 ms (84%) | 5.02 ms (84%) | 5.26 ms (85%) |
| Qwen2.5-1.5B | 6.22 ms (81%) | 6.27 ms (80%) | 6.32 ms (80%) | 6.48 ms (79%) | 6.90 ms (75%) | 7.11 ms (76%) |
| Llama-3.2-3B | 12.11 ms (86%) | 12.29 ms (86%) | 12.53 ms (85%) | 12.68 ms (85%) | 13.13 ms (85%) | 13.99 ms (86%) |
Prefill
Processing a 2,000-token prompt is compute-bound, so it measures the GEMMs and attention. Measured inside the full benchmark suite, where the 2080 Ti's power cap holds it at 1.4–1.55 GHz; a short isolated run of Llama-3.2-3B gives 273.6 ms instead of 289.7.
Data table
| model | hipinfer ms | llama.cpp ms |
|---|---|---|
| TinyLlama-1.1B | 100.5 | 114.8 |
| Qwen2.5-0.5B | 43.1 | 67.3 |
| Llama-3.2-1B | 101.1 | 100.9 |
| Qwen2.5-1.5B | 140.9 | 156.7 |
| Llama-3.2-3B | 289.7 | 261.8 |
On an A100
A100-SXM4-40GB (1,555 GB/s) on Google Colab, with llama.cpp, vLLM 0.31.0 and torch.compile run in the same session and no thermal
throttling (THERMAL.md). hipinfer was re-run next to the other
engines, so every row compares runs a few minutes apart. fp16 weights, batch 1. llama.cpp is llama-bench on
an F16 GGUF, best of flash-attention on and off; vLLM uses its V1 engine with CUDA graphs; torch.compile is a
gpt-fast-style static KV cache in reduce-overhead mode. A baseline counts only if its greedy tokens match
HF transformers.
Data table
| model | hipinfer ms/token | % of floor | llama.cpp, vs hipinfer | vLLM 0.31.0, vs hipinfer | torch.compile, vs hipinfer |
|---|---|---|---|---|---|
| TinyLlama-1.1B | 2.18 | 61% | 2.57 (1.18× slower) | 2.83 (1.30× slower) | 3.14 (1.44× slower) |
| Llama-3.2-1B | 2.31 | 69% | 2.63 (1.14× slower) | 2.86 (1.23× slower) | 3.08 (1.33× slower) |
| Qwen2.5-1.5B | 3.21 | 62% | 4.08 (1.27× slower) | 4.19 (1.31× slower) | not valid |
| Llama-3.2-3B | 5.60 | 74% | 6.17 (1.10× slower) | not run | 6.79 (1.21× slower) |
generate() call, so it includes one generated tokenData table
| model | hipinfer ms | llama.cpp, vs hipinfer | vLLM 0.31.0 (incl. 1 generated token), vs hipinfer | torch.compile, vs hipinfer |
|---|---|---|---|---|
| TinyLlama-1.1B | 35.6 | 51.7 (1.45× slower) | 29.9 (1.19× faster) | 34.8 (1.02× faster) |
| Llama-3.2-1B | 33.3 | 47.7 (1.43× slower) | 30.4 (1.09× faster) | 30.2 (1.10× faster) |
| Qwen2.5-1.5B | 43.5 | 63.1 (1.45× slower) | 35.8 (1.22× faster) | not valid |
| Llama-3.2-3B | 87.0 | 100.8 (1.16× slower) | not run | 73.8 (1.18× faster) |
- torch.compile, Qwen2.5-1.5B: not valid. My torch.compile baseline left out Qwen2's QKV bias, so its greedy tokens didn't match HF (0/32) and its numbers are excluded. The bias is fixed in scripts/bench_baselines.py but hasn't been re-run.
- vLLM, Llama-3.2-3B: not run. The Colab disk filled before vLLM could load Llama-3.2-3B.
Decode: hipinfer is 1.10–1.27× faster than llama.cpp, 1.23–1.31× faster than vLLM and 1.21–1.44× faster than torch.compile on the models each one ran. It is further from the floor than on the 2080 Ti (61–74% on these 1–3B models, against 81–87%). The weights stream about 2.5× faster, but each kernel launch costs about the same, so the fixed per-launch time is a bigger share of a token. Mistral-7B-v0.3 also ran on the A100 (no other engines): 11.07 ms/token, 83% of the floor. llama.cpp and vLLM sit further below the floor still (47–67%).
Long prompts are where hipinfer loses. It is 1.16–1.45× faster than llama.cpp at 2,000 tokens, but vLLM and torch.compile are 1.02–1.22× faster than hipinfer there, and HF transformers is 1.08–1.14× faster too. All three use cuBLAS and FlashAttention-2, and on the A100 my GEMMs run at only 0.49–0.72× cuBLAS (2,048-row prefill shapes of these four models).
Two GPUs: tensor parallelism
Each layer is split across both GPUs, Megatron-style, with an all-reduce after the attention and MLP outputs. These two RTX 2080 Tis sit on different CPU sockets with no peer-to-peer path, so every byte between them crosses host memory: close to the worst case for multi-GPU inference. Llama-3.1-8B and Mistral-7B don't fit one 11 GB card and run only sharded.
Data table
| model | 1 GPU ms/token | 2 GPUs ms/token | speedup | prefill 2,000 tok: 1 GPU → 2 GPUs (ms) | 2-GPU peak batched tok/s |
|---|---|---|---|---|---|
| TinyLlama-1.1B | 4.15 | 2.71 | 1.53× | 105 → 84 | 8,464 |
| Qwen2.5-0.5B | 2.33 | 1.76 | 1.33× | 44 → 42 | 13,242 |
| Llama-3.2-1B | 4.72 | 2.81 | 1.68× | 102 → 71 | 9,454 |
| Qwen2.5-1.5B | 6.19 | 3.87 | 1.60× | 142 → 102 | 5,675 |
| Llama-3.2-3B | 11.99 | 6.89 | 1.74× | 287 → 182 | 3,298 |
| Llama-3.1-8B | doesn't fit | 14.68 | – | – → 371 | 864 |
| Mistral-7B-v0.3 | doesn't fit | 13.99 | – | – → 366 | 1,408 |
| message | size | hipinfer one-shot kernel | NCCL | speedup |
|---|---|---|---|---|
| decode B=1 | 4 KB | 8.5 µs | 12.2 µs | 1.44× |
| decode B=8 | 32 KB | 17.7 µs | 24.9 µs | 1.41× |
| decode B=64 | 256 KB | 63.6 µs | 85.4 µs | 1.34× |
| prefill T=512 | 2,048 KB | 425.0 µs | 426.0 µs | 1.00× |
| prefill T=2000 | 8,000 KB | 1574.4 µs | 1554.9 µs | 0.99× |
int8 weights (W8A16)
The projections and the lm_head are stored as int8 with one fp16 scale per output row (symmetric round-to-nearest); activations, the KV cache, norms and the embedding stay fp16. Batch-1 decode reads every weight once per token, so int8 halves the bytes the GEMVs stream. Decode converts the weights to fp32 in registers; prefill dequantizes each weight tile to fp16 on its way into shared memory and runs the existing tensor-core GEMM. RTX 2080 Ti, batch 1. hipinfer's int8 and fp16 weights and llama.cpp's Q8_0 GGUF were measured back to back in one session, so the fp16 times here differ slightly from the charts above. Llama-3.2-3B's rows come from a rerun started on a cool card, where the int8 and fp16 runs' median clocks are within 2 % (MERGED_RUN.md).
Data table
| model | context | int8 ms/token | fp16 ms/token | int8 speed-up | llama.cpp Q8_0 ms/token | int8 vs Q8_0 | int8 % of floor |
|---|---|---|---|---|---|---|---|
| TinyLlama-1.1B | 128 | 2.54 | 4.15 | 1.63× | 3.21 | 1.26× faster | 67% |
| Llama-3.2-1B | 128 | 2.73 | 4.72 | 1.73× | 3.32 | 1.22× faster | 74% |
| Llama-3.2-3B | 128 | 6.71 | 12.01 | 1.79× | 7.94 | 1.18× faster | 78% |
| TinyLlama-1.1B | 1,024 | 2.63 | 4.23 | 1.61× | 3.25 | 1.24× faster | 66% |
| Llama-3.2-1B | 1,024 | 2.81 | 4.79 | 1.71× | 3.38 | 1.20× faster | 74% |
| Llama-3.2-3B | 1,024 | 7.11 | 12.45 | 1.75× | 8.25 | 1.16× faster | 76% |
Data table
| model | tokens | int8 ms | fp16 ms | int8 vs fp16 | llama.cpp Q8_0 ms | int8 vs Q8_0 |
|---|---|---|---|---|---|---|
| TinyLlama-1.1B | 128 | 9.2 | 8.6 | 6% slower | 15.8 | 1.72× faster |
| TinyLlama-1.1B | 512 | 29.9 | 26.8 | 12% slower | 36.1 | 1.21× faster |
| TinyLlama-1.1B | 2,000 | 117.0 | 101.6 | 15% slower | 144.9 | 1.24× faster |
| Llama-3.2-1B | 128 | 9.1 | 8.4 | 9% slower | 12.3 | 1.36× faster |
| Llama-3.2-1B | 512 | 28.4 | 25.6 | 11% slower | 31.8 | 1.12× faster |
| Llama-3.2-1B | 2,000 | 114.7 | 97.5 | 18% slower | 124.3 | 1.08× faster |
| Llama-3.2-3B | 128 | 24.4 | 24.0 | 1% slower | 30.0 | 1.23× faster |
| Llama-3.2-3B | 512 | 83.1 | 73.1 | 14% slower | 92.2 | 1.11× faster |
| Llama-3.2-3B | 2,000 | 328.0 | 281.9 | 16% slower | 346.1 | 1.05× faster |
Decode is 1.63–1.79× faster than fp16, not 2×. The weight bytes halve, but the KV-cache reads and each kernel's fixed cost don't, and the conversion costs instructions: int8 decode reaches 66–78% of its bandwidth floor, against 80–87% for fp16. At a 1,024-token context it is 1.61–1.75× faster than fp16. Against llama.cpp's Q8_0 it is 1.18–1.26× faster at 128 tokens of context and 1.16–1.24× at 1,024.
Prefill is slower than fp16: 1–9% at 128 tokens and 11–18% at 512–2,000, because each weight tile is converted on its way to shared memory. It is still 1.05–1.72× faster than llama.cpp's Q8_0 prefill at every length measured.
Accuracy. Against fp16 HF, WikiText-2 perplexity rises by 0.04 % / 0.23 % / 0.20 %, the cost of rounding the weights. Against HF running the same rounded weights it is within 0.003 % (mean KL 6.6e-6 to 1.3e-5), and the strict token gate passes with 15 / 18 / 18 of 20 runs identical, so the int8 kernels add no error of their own beyond fp16 rounding.
| model | PPL HF fp16 | PPL hipinfer int8 | change | PPL HF, same int8 weights | change | mean KL vs HF int8 | greedy runs identical to HF int8 |
|---|---|---|---|---|---|---|---|
| TinyLlama-1.1B | 8.5151 | 8.5186 | +0.04% | 8.5187 | -0.0003% | 6.6e-06 | 15/20 |
| Llama-3.2-1B | 10.4631 | 10.4871 | +0.23% | 10.4874 | -0.0034% | 1.3e-05 | 18/20 |
| Llama-3.2-3B | 8.2768 | 8.2930 | +0.20% | 8.2931 | -0.0010% | 9.5e-06 | 18/20 |
- Q8_0 is not the same format. It keeps one scale per 32 weights instead of one per row, so it reads about 6 % more bytes and should round a little more accurately. llama.cpp's perplexity was not measured, and there is no 4-bit comparison.
- Scope. 3 models, one GPU type (RTX 2080 Ti), batch 1, int8 weights only (no int8 activations or KV cache).
Speculative decoding
Greedy speculative decoding with two engines on one GPU. Llama-3.2-1B drafts k tokens with its normal
graph-captured decode steps, and Llama-3.2-3B checks all of them in one pass. The longest agreeing prefix is accepted,
plus the target's own next token, and both caches roll back by resetting their lengths. The target verifies through the
batch-1 decode kernels, so every row's logits are bit-identical to the decode step they replace and the output is the
same as plain greedy decoding token for token, with no near-tie tolerance: tests/test_spec.py passes
105 of 105 on the RTX 2080 Ti and 105 of 105 on the RTX A6000 (7 prompts × 128 tokens at k = 2, 4, 6, 8, with and without CUDA graphs, plus self-drafting).
Data table
| k | tok/s | speed-up | predicted by the cost model | draft tokens accepted | tokens per round |
|---|---|---|---|---|---|
| 1 | 89.7 | 1.07× | 1.08× | 91% | 1.91 |
| 2 | 102.1 | 1.22× | 1.24× | 87% | 2.72 |
| 3 | 111.8 | 1.34× | 1.36× | 82% | 3.45 |
| 4 | 115.0 | 1.37× | 1.39× | 79% | 4.10 |
| 5 | 116.7 | 1.39× | 1.41× | 76% | 4.70 |
| 6 | 116.1 | 1.39× | 1.40× | 72% | 5.26 |
| 8 | 111.7 | 1.33× | 1.33× | 67% | 6.17 |
Up to 1.39× at k = 5 and 6. The draft is not much cheaper than the target:
both are weight-bandwidth bound, so a 1B decode step costs 39 % of a 3B one
(4.66 vs 11.85 ms), and gains level off around k = 5. Verification is cheap: checking 9
rows in one pass costs 1.31× a single 3B decode step
(15.5 vs 11.8 ms), because the weights are read once for all rows. The cost model,
k·t_draft + t_verify(k + 1) per round, is within 2 % at every k.
- Self-draft (Llama-3.2-1B drafting for itself): every draft token accepted, and 0.66× (k = 1) to 0.87× (k = 8) plain speed. This is the overhead of drafting with a model as expensive as the target; the run exists to check exactness.
- Verifying through the prefill path would have broken exactness. Its GEMMs round differently from decode: it diverged from plain greedy output in 16 of 28 self-draft runs (0 of 49 with the 3B target).
- Scope. One model pair on one GPU type (RTX 2080 Ti), greedy, batch 1, dense models only; the draft must share the target's vocabulary.
KV-cache eviction scoring (H2O, WFL)
Eviction policies decide which cached keys to drop when a sequence outgrows its cache budget. H2O keeps the keys that
have received the most attention; WFL prune, from SubCache, removes the keys another kept key can best stand in for. Both
need numbers the FlashAttention-style kernels don't produce, so with tracking on the attention kernels also record them:
decode stores each key's logit and each head's log-sum-exp and adds the attention weights to a per-key score; prefill saves
each row's log-sum-exp and a second, key-parallel pass sums each key's column. Engine.evict then compacts the
kept keys, values and scores to the front of the cache. Tracking is off by default, and turning it on changes no output:
attention outputs and greedy tokens are bitwise equal to tracking off, and scores are within 2e-6 relative of a float64
reference (475 kernel checks and an engine test on the RTX 2080 Ti and the RTX A6000). Costs below are on an RTX 2080 Ti,
with no thermal throttling.
Data table
| model | batch | keys | plain µs | with tracking µs | overhead |
|---|---|---|---|---|---|
| Llama-3.2-1B | 1 | 1,024 | 11.5 | 14.9 | +30% |
| Llama-3.2-1B | 1 | 4,096 | 25.4 | 31.4 | +23% |
| Llama-3.2-1B | 1 | 16,384 | 70.9 | 90.8 | +28% |
| Llama-3.2-1B | 1 | 32,768 | 128.7 | 168.3 | +31% |
| Llama-3.2-1B | 8 | 1,024 | 45.1 | 54.3 | +20% |
| Llama-3.2-1B | 8 | 4,096 | 137.3 | 168.8 | +23% |
| Llama-3.2-1B | 8 | 16,384 | 504.4 | 613.2 | +22% |
| Llama-3.2-1B | 8 | 32,768 | 932.8 | 1,220.7 | +31% |
| Llama-3.2-3B | 1 | 1,024 | 13.9 | 17.5 | +26% |
| Llama-3.2-3B | 1 | 4,096 | 42.0 | 48.8 | +16% |
| Llama-3.2-3B | 1 | 16,384 | 130.1 | 150.0 | +15% |
| Llama-3.2-3B | 1 | 32,768 | 244.2 | 284.9 | +17% |
| Llama-3.2-3B | 8 | 1,024 | 73.0 | 83.4 | +14% |
| Llama-3.2-3B | 8 | 4,096 | 244.3 | 280.2 | +15% |
| Llama-3.2-3B | 8 | 16,384 | 930.2 | 1,070.4 | +15% |
| Llama-3.2-3B | 8 | 32,768 | 1,845.2 | 2,140.1 | +16% |
| Qwen2.5-1.5B | 1 | 1,024 | 11.5 | 15.2 | +32% |
| Qwen2.5-1.5B | 1 | 4,096 | 27.6 | 29.1 | +5% |
| Qwen2.5-1.5B | 1 | 16,384 | 52.9 | 60.2 | +14% |
| Qwen2.5-1.5B | 1 | 32,768 | 81.9 | 95.7 | +17% |
| Qwen2.5-1.5B | 8 | 1,024 | 33.1 | 38.1 | +15% |
| Qwen2.5-1.5B | 8 | 4,096 | 87.7 | 102.9 | +17% |
| Qwen2.5-1.5B | 8 | 16,384 | 294.6 | 356.1 | +21% |
| Qwen2.5-1.5B | 8 | 32,768 | 570.8 | 697.2 | +22% |
| Llama-3.1-8B | 1 | 1,024 | 14.6 | 17.4 | +19% |
| Llama-3.1-8B | 1 | 4,096 | 43.9 | 50.6 | +15% |
| Llama-3.1-8B | 1 | 16,384 | 132.0 | 155.4 | +18% |
| Llama-3.1-8B | 1 | 32,768 | 245.8 | 291.4 | +19% |
| Llama-3.1-8B | 8 | 1,024 | 74.8 | 85.6 | +14% |
| Llama-3.1-8B | 8 | 4,096 | 246.1 | 286.6 | +16% |
| Llama-3.1-8B | 8 | 16,384 | 931.6 | 1,098.9 | +18% |
| Llama-3.1-8B | 8 | 32,768 | 1,845.3 | 2,186.4 | +18% |
Tracking slows the decode-attention kernel by 5–32 % across four models, 1k–32k keys and batch 1 and 8, but attention is a small part of a decode step: a whole Llama-3.2-1B step at a 4.1k context goes from 4.96 to 5.05 ms (+1.7 %). In prefill, the second, key-parallel pass costs 1.0–1.8× the FlashAttention kernel's own time at 2–4k tokens and 1.4–4.5× at 512 (few kv heads launch few blocks); storing the log-sum-exp in the first pass adds at most 4 %.
Choosing what to keep. Over every (layer, kv head), H2O's top-k takes 0.03–0.10 ms at 1k–4k slots. WFL prune is the expensive policy: 9–40 ms to evict 64 slots from 1,024 (8 rounds), 0.1–0.6 s at 4,096, and 0.98–4.2 s for a full prune from 4,096 to 512. Against a PyTorch matrix version of the same algorithm at 1,024 slots (Llama-3.2-1B and Qwen2.5-1.5B), the kernels are 6.4–25.0× faster.
| Llama-3.2-1B in the engine | time |
|---|---|
| decode step, 4.1k context, tracking off → on | 4.96 → 5.05 ms (+1.7%) |
| evict 4,096 → 1,024 slots, H2O | select 0.10 ms + compact 0.88 ms |
| evict 4,096 → 1,024 slots, WFL | select 885 ms + compact 0.82 ms |
| evict 4,096 → 1,024 slots, WFL with a pool of 2 × m, 8 rounds | select 63 ms + compact 0.83 ms |
| decode step after the eviction | 4.85 ms |
Prefill score pass
| model | prompt tokens | flash attention ms | with LSE stored ms | score pass ms | score pass / flash |
|---|---|---|---|---|---|
| Llama-3.2-1B | 512 | 0.045 | 0.045 | 0.105 | 2.35× |
| Llama-3.2-1B | 2,048 | 0.484 | 0.480 | 0.669 | 1.38× |
| Llama-3.2-1B | 4,096 | 1.760 | 1.763 | 2.079 | 1.18× |
| Llama-3.2-3B | 512 | 0.075 | 0.075 | 0.104 | 1.39× |
| Llama-3.2-3B | 2,048 | 0.796 | 0.801 | 0.943 | 1.18× |
| Llama-3.2-3B | 4,096 | 2.946 | 2.959 | 3.057 | 1.04× |
| Qwen2.5-1.5B | 512 | 0.045 | 0.045 | 0.201 | 4.51× |
| Qwen2.5-1.5B | 2,048 | 0.430 | 0.430 | 0.782 | 1.82× |
| Qwen2.5-1.5B | 4,096 | 1.524 | 1.526 | 1.545 | 1.01× |
| Llama-3.1-8B | 512 | 0.090 | 0.093 | 0.138 | 1.54× |
| Llama-3.1-8B | 2,048 | 1.050 | 1.046 | 1.249 | 1.19× |
| Llama-3.1-8B | 4,096 | 3.892 | 3.896 | 4.062 | 1.04× |
Selection over every (layer, kv head)
| model | (layer, kv head) problems | slots | policy | kernel ms | PyTorch matrix version |
|---|---|---|---|---|---|
| Llama-3.2-1B | 128 | 1,024 | H2O top-k | 0.03 | – |
| Llama-3.2-1B | 128 | 1,024 | WFL prune to 128 (step 8), 112 rounds | 62.5 | 1,563 ms (25.0× slower) |
| Llama-3.2-1B | 128 | 1,024 | WFL prune: evict 64 (step 8), 8 rounds | 9.73 | 128 ms (13.2× slower) |
| Llama-3.2-1B | 128 | 2,048 | H2O top-k | 0.04 | – |
| Llama-3.2-1B | 128 | 2,048 | WFL prune to 256 (step 8), 224 rounds | 248.3 | – |
| Llama-3.2-1B | 128 | 2,048 | WFL prune: evict 64 (step 8), 8 rounds | 33.5 | – |
| Llama-3.2-1B | 128 | 4,096 | H2O top-k | 0.06 | – |
| Llama-3.2-1B | 128 | 4,096 | WFL prune to 512 (step 8), 448 rounds | 1,012.0 | – |
| Llama-3.2-1B | 128 | 4,096 | WFL prune: evict 64 (step 8), 8 rounds | 118.9 | – |
| Llama-3.2-3B | 224 | 1,024 | H2O top-k | 0.06 | – |
| Llama-3.2-3B | 224 | 1,024 | WFL prune to 128 (step 8), 112 rounds | 225.1 | – |
| Llama-3.2-3B | 224 | 1,024 | WFL prune: evict 64 (step 8), 8 rounds | 34.9 | – |
| Llama-3.2-3B | 224 | 2,048 | H2O top-k | 0.06 | – |
| Llama-3.2-3B | 224 | 2,048 | WFL prune to 256 (step 8), 224 rounds | 904.8 | – |
| Llama-3.2-3B | 224 | 2,048 | WFL prune: evict 64 (step 8), 8 rounds | 115.7 | – |
| Llama-3.2-3B | 224 | 4,096 | H2O top-k | 0.09 | – |
| Llama-3.2-3B | 224 | 4,096 | WFL prune to 512 (step 8), 448 rounds | 3,653.9 | – |
| Llama-3.2-3B | 224 | 4,096 | WFL prune: evict 64 (step 8), 8 rounds | 416.2 | – |
| Qwen2.5-1.5B | 56 | 1,024 | H2O top-k | 0.10 | – |
| Qwen2.5-1.5B | 56 | 1,024 | WFL prune to 128 (step 8), 112 rounds | 61.1 | 694 ms (11.4× slower) |
| Qwen2.5-1.5B | 56 | 1,024 | WFL prune: evict 64 (step 8), 8 rounds | 9.17 | 58 ms (6.4× slower) |
| Qwen2.5-1.5B | 56 | 2,048 | H2O top-k | 0.04 | – |
| Qwen2.5-1.5B | 56 | 2,048 | WFL prune to 256 (step 8), 224 rounds | 243.2 | – |
| Qwen2.5-1.5B | 56 | 2,048 | WFL prune: evict 64 (step 8), 8 rounds | 30.0 | – |
| Qwen2.5-1.5B | 56 | 4,096 | H2O top-k | 0.04 | – |
| Qwen2.5-1.5B | 56 | 4,096 | WFL prune to 512 (step 8), 448 rounds | 980.0 | – |
| Qwen2.5-1.5B | 56 | 4,096 | WFL prune: evict 64 (step 8), 8 rounds | 106.3 | – |
| Llama-3.1-8B | 256 | 1,024 | H2O top-k | 0.05 | – |
| Llama-3.1-8B | 256 | 1,024 | WFL prune to 128 (step 8), 112 rounds | 248.5 | – |
| Llama-3.1-8B | 256 | 1,024 | WFL prune: evict 64 (step 8), 8 rounds | 40.1 | – |
| Llama-3.1-8B | 256 | 2,048 | H2O top-k | 0.06 | – |
| Llama-3.1-8B | 256 | 2,048 | WFL prune to 256 (step 8), 224 rounds | 1,000.7 | – |
| Llama-3.1-8B | 256 | 2,048 | WFL prune: evict 64 (step 8), 8 rounds | 132.7 | – |
| Llama-3.1-8B | 256 | 4,096 | H2O top-k | 0.09 | – |
| Llama-3.1-8B | 256 | 4,096 | WFL prune to 512 (step 8), 448 rounds | 4,152.2 | – |
| Llama-3.1-8B | 256 | 4,096 | WFL prune: evict 64 (step 8), 8 rounds | 605.4 | – |
- No accuracy result. These numbers show the kernels compute the policies as specified and what that costs. They say nothing about how well either policy preserves a model's answers at any cache budget.
- Scope. Costs measured on one GPU type (RTX 2080 Ti); correctness checks also ran on the RTX A6000.
Correctness
Speed only counts if the output is right. Greedy runs: 7 prompts × 128 tokens, single-sequence with and without CUDA graphs plus a batch of 6, compared token for token with HF transformers; any divergence lands where HF's own top-2 logits are tied or one fp16 step apart. Perplexity: WikiText-2, 1,024-token windows, both engines on the same tokens.
| model | GPU | setup | greedy runs token-identical to HF | PPL hipinfer | PPL HF | top-1 agreement | mean KL(HF ‖ hipinfer) |
|---|---|---|---|---|---|---|---|
| TinyLlama-1.1B | RTX 2080 Ti | 1 GPU | 20/20 | 8.5149 | 8.5151 | 99.90% | 6.6e-06 |
| Qwen2.5-0.5B | RTX 2080 Ti | 1 GPU | 18/20 | 13.4851 | 13.4851 | 99.79% | 1.6e-05 |
| Llama-3.2-1B | RTX 2080 Ti | 1 GPU | 13/20 | 10.4633 | 10.4631 | 99.81% | 1.3e-05 |
| Qwen2.5-1.5B | RTX 2080 Ti | 1 GPU | 17/20 | 9.4536 | 9.4535 | 99.86% | 1.6e-05 |
| Llama-3.2-3B | RTX 2080 Ti | 1 GPU | 20/20 | 8.2767 | 8.2768 | 99.82% | 9.5e-06 |
| TinyLlama-1.1B | A100-SXM4-40GB | 1 GPU | 20/20 | 10.7326 | 10.7326 | 99.85% | 6.3e-06 |
| Qwen2.5-0.5B | A100-SXM4-40GB | 1 GPU | 20/20 | 17.2227 | 17.2236 | 99.74% | 1.6e-05 |
| Llama-3.2-1B | A100-SXM4-40GB | 1 GPU | 13/20 | 12.5618 | 12.5619 | 99.87% | 1.3e-05 |
| Qwen2.5-1.5B | A100-SXM4-40GB | 1 GPU | 16/20 | 11.9787 | 11.9787 | 99.77% | 1.6e-05 |
| Llama-3.2-3B | A100-SXM4-40GB | 1 GPU | 20/20 | 10.2217 | 10.2214 | 99.90% | 9.5e-06 |
| Mistral-7B-v0.3 | A100-SXM4-40GB | 1 GPU | 20/20 | 7.3269 | 7.3267 | 99.88% | 4.2e-06 |
| TinyLlama-1.1B | RTX A6000 | 1 GPU | 20/20 | 8.5149 | 8.5150 | 99.88% | 6.6e-06 |
| Qwen2.5-0.5B | RTX A6000 | 1 GPU | 19/20 | 13.4846 | 13.4849 | 99.74% | 1.5e-05 |
| Llama-3.2-1B | RTX A6000 | 1 GPU | 11/20 | 10.4631 | 10.4633 | 99.86% | 1.3e-05 |
| Qwen2.5-1.5B | RTX A6000 | 1 GPU | 17/20 | 9.4534 | 9.4539 | 99.77% | 1.6e-05 |
| Llama-3.2-3B | RTX A6000 | 1 GPU | 20/20 | 8.2770 | 8.2765 | 99.88% | 9.8e-06 |
| Mistral-7B-v0.3 | RTX A6000 | 1 GPU | 20/20 | 5.7774 | 5.7774 | 99.91% | 4.6e-06 |
| Llama-3.1-8B | RTX A6000 | 1 GPU | 14/20 | 6.5467 | 6.5469 | 99.89% | 1.1e-05 |
| Llama-3.1-8B | 2× RTX 2080 Ti | sharded over 2 GPUs | – | 6.5467 | 6.5471 | 99.87% | 1.2e-05 |
| Mistral-7B-v0.3 | 2× RTX 2080 Ti | sharded over 2 GPUs | – | 5.7773 | 5.7775 | 99.88% | 6.5e-06 |
Limits of these measurements
- Thermal throttling. Due to resource constraints and degrading GPUs, runs were at times throttled due to overheating. The RTX A6000 reaches 90–93 °C under sustained load and throttles its SM clock to as low as 210 MHz: its Llama-3.2-1B, Qwen2.5-1.5B, Llama-3.2-3B and Mistral-7B-v0.3 benchmarks ran partly throttled (listed in the README and THERMAL.md), so the main charts here use the RTX 2080 Ti. The A100 ran in one Colab session with no throttling (THERMAL.md). Correctness results are unaffected.
- Launch overhead on the A100. Batch-1 decode reaches only 61–74% of the bandwidth floor on the 1–3B models: the A100 streams weights fast enough that fixed per-launch time is a large share of each token.
- Long-prompt prefill on the A100. At 2,000 tokens vLLM and torch.compile are 1.02–1.22× faster than hipinfer; hipinfer's GEMMs run at 0.49–0.72× cuBLAS on that GPU.
- int8, speculative decoding and eviction ran on one GPU type for speed. Their benchmarks are RTX 2080 Ti only, batch 1; speculative decoding is one model pair (Llama-3.2-3B with a 1B draft), and int8 prefill is 1–18 % slower than fp16. Eviction scoring has no accuracy result at any cache budget.
- AMD support. Because this project was built in HIP, it has full AMD support, and compiles properly. However, I had no available AMD GPUs and the cloud solutions available were either at capacity or too expensive.
- Peer-to-peer multi-GPU. The P2P all-reduce path is implemented but untested; only two GPUs have been tested.