hipinfer

An LLM inference engine written from scratch in HIP: tensor-core GEMMs, fused GEMV / RMSNorm / RoPE kernels, FlashAttention-style attention, a graph-captured decode loop, tensor parallelism across GPUs, int8 weights, speculative decoding and KV-cache eviction scoring. One source tree builds with nvcc for NVIDIA and hipcc for AMD. These are its measured results.
Batch-1 decode, share of the memory-bandwidth limit
81–87%
models of 1B+ parameters, RTX 2080 Ti
Batch-1 decode vs llama.cpp
1.05–1.47× faster
5 models, same GPU, fp16
Two GPUs vs one, batch-1 decode
up to 1.74× faster
tensor parallelism, 2× RTX 2080 Ti
Llama-3.1-8B on two 11 GB cards
14.7 ms/token
16 GB of weights; doesn't fit one card
Batch-1 decode on an A100 vs vLLM
1.23–1.31× faster
3 models, one session; slower on long-prompt prefill
int8 weights vs fp16, batch-1 decode
1.63–1.79× faster
and 1.18–1.26× faster than llama.cpp Q8_0; RTX 2080 Ti
Speculative decoding, Llama-3.2-3B with a 1B draft
up to 1.39× faster
output identical to greedy; one model pair, RTX 2080 Ti
WikiText-2 perplexity vs HF transformers
within 0.006%
20 model/GPU checks

Batch-1 decode

Generating one token at a time is memory-bound: every token reads all the weights once. The bandwidth floor is the bytes read per token divided by the GPU's peak bandwidth, the fastest any engine could go. RTX 2080 Ti, fp16 weights, 128-token prompt. llama.cpp is llama-bench on an F16 GGUF, best of flash-attention on and off.

Decode latency per token (lower is better)
ms/token at a 128-token context
hipinfer llama.cppbandwidth floor
Data table
modelhipinfer ms/tokenllama.cpp ms/tokenbandwidth floor ms% of floor
TinyLlama-1.1B4.144.773.3781%
Qwen2.5-0.5B2.343.451.6169%
Llama-3.2-1B4.755.234.0385%
Qwen2.5-1.5B6.217.195.0281%
Llama-3.2-3B12.1112.7010.4887%
Share of the bandwidth floor as context grows (higher is better)
batch-1 decode, % of the floor including KV-cache reads
Data table
modelctx 128ctx 512ctx 1,024
TinyLlama-1.1B4.13 ms (81%)4.17 ms (81%)4.22 ms (81%)
Qwen2.5-0.5B2.33 ms (69%)2.38 ms (68%)2.42 ms (67%)2.50 ms (66%)2.64 ms (64%)2.73 ms (65%)
Llama-3.2-1B4.77 ms (84%)4.80 ms (84%)4.84 ms (84%)4.90 ms (84%)5.02 ms (84%)5.26 ms (85%)
Qwen2.5-1.5B6.22 ms (81%)6.27 ms (80%)6.32 ms (80%)6.48 ms (79%)6.90 ms (75%)7.11 ms (76%)
Llama-3.2-3B12.11 ms (86%)12.29 ms (86%)12.53 ms (85%)12.68 ms (85%)13.13 ms (85%)13.99 ms (86%)

Prefill

Processing a 2,000-token prompt is compute-bound, so it measures the GEMMs and attention. Measured inside the full benchmark suite, where the 2080 Ti's power cap holds it at 1.4–1.55 GHz; a short isolated run of Llama-3.2-3B gives 273.6 ms instead of 289.7.

Prefill time for 2,000 tokens (lower is better)
ms, RTX 2080 Ti
hipinfer llama.cpp
Data table
modelhipinfer msllama.cpp ms
TinyLlama-1.1B100.5114.8
Qwen2.5-0.5B43.167.3
Llama-3.2-1B101.1100.9
Qwen2.5-1.5B140.9156.7
Llama-3.2-3B289.7261.8

On an A100

A100-SXM4-40GB (1,555 GB/s) on Google Colab, with llama.cpp, vLLM 0.31.0 and torch.compile run in the same session and no thermal throttling (THERMAL.md). hipinfer was re-run next to the other engines, so every row compares runs a few minutes apart. fp16 weights, batch 1. llama.cpp is llama-bench on an F16 GGUF, best of flash-attention on and off; vLLM uses its V1 engine with CUDA graphs; torch.compile is a gpt-fast-style static KV cache in reduce-overhead mode. A baseline counts only if its greedy tokens match HF transformers.

Decode latency per token on an A100 (lower is better)
ms/token at a 128-token context
hipinfer llama.cppvLLM 0.31.0 torch.compilebandwidth floor
Data table
modelhipinfer ms/token% of floorllama.cpp, vs hipinfervLLM 0.31.0, vs hipinfertorch.compile, vs hipinfer
TinyLlama-1.1B2.1861%2.57 (1.18× slower)2.83 (1.30× slower)3.14 (1.44× slower)
Llama-3.2-1B2.3169%2.63 (1.14× slower)2.86 (1.23× slower)3.08 (1.33× slower)
Qwen2.5-1.5B3.2162%4.08 (1.27× slower)4.19 (1.31× slower)not valid
Llama-3.2-3B5.6074%6.17 (1.10× slower)not run6.79 (1.21× slower)
Prefill time for 2,000 tokens on an A100 (lower is better)
ms; vLLM's time is an end-to-end generate() call, so it includes one generated token
hipinfer llama.cppvLLM 0.31.0 torch.compile
Data table
modelhipinfer msllama.cpp, vs hipinfervLLM 0.31.0 (incl. 1 generated token), vs hipinfertorch.compile, vs hipinfer
TinyLlama-1.1B35.651.7 (1.45× slower)29.9 (1.19× faster)34.8 (1.02× faster)
Llama-3.2-1B33.347.7 (1.43× slower)30.4 (1.09× faster)30.2 (1.10× faster)
Qwen2.5-1.5B43.563.1 (1.45× slower)35.8 (1.22× faster)not valid
Llama-3.2-3B87.0100.8 (1.16× slower)not run73.8 (1.18× faster)

Decode: hipinfer is 1.10–1.27× faster than llama.cpp, 1.23–1.31× faster than vLLM and 1.21–1.44× faster than torch.compile on the models each one ran. It is further from the floor than on the 2080 Ti (61–74% on these 1–3B models, against 81–87%). The weights stream about 2.5× faster, but each kernel launch costs about the same, so the fixed per-launch time is a bigger share of a token. Mistral-7B-v0.3 also ran on the A100 (no other engines): 11.07 ms/token, 83% of the floor. llama.cpp and vLLM sit further below the floor still (47–67%).

Long prompts are where hipinfer loses. It is 1.16–1.45× faster than llama.cpp at 2,000 tokens, but vLLM and torch.compile are 1.02–1.22× faster than hipinfer there, and HF transformers is 1.08–1.14× faster too. All three use cuBLAS and FlashAttention-2, and on the A100 my GEMMs run at only 0.49–0.72× cuBLAS (2,048-row prefill shapes of these four models).

Two GPUs: tensor parallelism

Each layer is split across both GPUs, Megatron-style, with an all-reduce after the attention and MLP outputs. These two RTX 2080 Tis sit on different CPU sockets with no peer-to-peer path, so every byte between them crosses host memory: close to the worst case for multi-GPU inference. Llama-3.1-8B and Mistral-7B don't fit one 11 GB card and run only sharded.

Decode latency per token, one GPU vs two (lower is better)
ms/token at a 128-token context
1 GPU 2 GPUs
Data table
model1 GPU ms/token2 GPUs ms/tokenspeedupprefill 2,000 tok: 1 GPU → 2 GPUs (ms)2-GPU peak batched tok/s
TinyLlama-1.1B4.152.711.53×105 → 848,464
Qwen2.5-0.5B2.331.761.33×44 → 4213,242
Llama-3.2-1B4.722.811.68×102 → 719,454
Qwen2.5-1.5B6.193.871.60×142 → 1025,675
Llama-3.2-3B11.996.891.74×287 → 1823,298
Llama-3.1-8Bdoesn't fit14.68–– → 371864
Mistral-7B-v0.3doesn't fit13.99–– → 3661,408
All-reduce latency: hipinfer's one-shot kernel vs NCCL
fp16, in place, two GPUs. Decode-sized messages use the one-shot kernel; large prefill messages use NCCL.
messagesizehipinfer one-shot kernelNCCLspeedup
decode B=14 KB8.5 µs12.2 µs1.44×
decode B=832 KB17.7 µs24.9 µs1.41×
decode B=64256 KB63.6 µs85.4 µs1.34×
prefill T=5122,048 KB425.0 µs426.0 µs1.00×
prefill T=20008,000 KB1574.4 µs1554.9 µs0.99×

int8 weights (W8A16)

The projections and the lm_head are stored as int8 with one fp16 scale per output row (symmetric round-to-nearest); activations, the KV cache, norms and the embedding stay fp16. Batch-1 decode reads every weight once per token, so int8 halves the bytes the GEMVs stream. Decode converts the weights to fp32 in registers; prefill dequantizes each weight tile to fp16 on its way into shared memory and runs the existing tensor-core GEMM. RTX 2080 Ti, batch 1. hipinfer's int8 and fp16 weights and llama.cpp's Q8_0 GGUF were measured back to back in one session, so the fp16 times here differ slightly from the charts above. Llama-3.2-3B's rows come from a rerun started on a cool card, where the int8 and fp16 runs' median clocks are within 2 % (MERGED_RUN.md).

Decode latency per token with int8 weights (lower is better)
ms/token at a 128-token context; the marker is int8's own bandwidth floor
hipinfer int8 hipinfer fp16 llama.cpp Q8_0int8 bandwidth floor
Data table
modelcontextint8 ms/tokenfp16 ms/tokenint8 speed-upllama.cpp Q8_0 ms/tokenint8 vs Q8_0int8 % of floor
TinyLlama-1.1B1282.544.151.63×3.211.26× faster67%
Llama-3.2-1B1282.734.721.73×3.321.22× faster74%
Llama-3.2-3B1286.7112.011.79×7.941.18× faster78%
TinyLlama-1.1B1,0242.634.231.61×3.251.24× faster66%
Llama-3.2-1B1,0242.814.791.71×3.381.20× faster74%
Llama-3.2-3B1,0247.1112.451.75×8.251.16× faster76%
Prefill time for 2,000 tokens with int8 weights (lower is better)
ms, same session
hipinfer int8 hipinfer fp16 llama.cpp Q8_0
Data table
modeltokensint8 msfp16 msint8 vs fp16llama.cpp Q8_0 msint8 vs Q8_0
TinyLlama-1.1B1289.28.66% slower15.81.72× faster
TinyLlama-1.1B51229.926.812% slower36.11.21× faster
TinyLlama-1.1B2,000117.0101.615% slower144.91.24× faster
Llama-3.2-1B1289.18.49% slower12.31.36× faster
Llama-3.2-1B51228.425.611% slower31.81.12× faster
Llama-3.2-1B2,000114.797.518% slower124.31.08× faster
Llama-3.2-3B12824.424.01% slower30.01.23× faster
Llama-3.2-3B51283.173.114% slower92.21.11× faster
Llama-3.2-3B2,000328.0281.916% slower346.11.05× faster

Decode is 1.63–1.79× faster than fp16, not 2×. The weight bytes halve, but the KV-cache reads and each kernel's fixed cost don't, and the conversion costs instructions: int8 decode reaches 66–78% of its bandwidth floor, against 80–87% for fp16. At a 1,024-token context it is 1.61–1.75× faster than fp16. Against llama.cpp's Q8_0 it is 1.18–1.26× faster at 128 tokens of context and 1.16–1.24× at 1,024.

Prefill is slower than fp16: 1–9% at 128 tokens and 11–18% at 512–2,000, because each weight tile is converted on its way to shared memory. It is still 1.05–1.72× faster than llama.cpp's Q8_0 prefill at every length measured.

Accuracy. Against fp16 HF, WikiText-2 perplexity rises by 0.04 % / 0.23 % / 0.20 %, the cost of rounding the weights. Against HF running the same rounded weights it is within 0.003 % (mean KL 6.6e-6 to 1.3e-5), and the strict token gate passes with 15 / 18 / 18 of 20 runs identical, so the int8 kernels add no error of their own beyond fp16 rounding.

int8 perplexity and token gate
WikiText-2, 16 × 1,024-token windows; the gate is 7 prompts × 128 tokens, 20 runs, against HF on the same int8 weights
modelPPL HF fp16PPL hipinfer int8changePPL HF, same int8 weightschangemean KL vs HF int8greedy runs identical to HF int8
TinyLlama-1.1B8.51518.5186+0.04%8.5187-0.0003%6.6e-0615/20
Llama-3.2-1B10.463110.4871+0.23%10.4874-0.0034%1.3e-0518/20
Llama-3.2-3B8.27688.2930+0.20%8.2931-0.0010%9.5e-0618/20

Speculative decoding

Greedy speculative decoding with two engines on one GPU. Llama-3.2-1B drafts k tokens with its normal graph-captured decode steps, and Llama-3.2-3B checks all of them in one pass. The longest agreeing prefix is accepted, plus the target's own next token, and both caches roll back by resetting their lengths. The target verifies through the batch-1 decode kernels, so every row's logits are bit-identical to the decode step they replace and the output is the same as plain greedy decoding token for token, with no near-tie tolerance: tests/test_spec.py passes 105 of 105 on the RTX 2080 Ti and 105 of 105 on the RTX A6000 (7 prompts × 128 tokens at k = 2, 4, 6, 8, with and without CUDA graphs, plus self-drafting).

Speed-up over plain greedy decoding (higher is better)
Llama-3.2-3B with a Llama-3.2-1B draft, RTX 2080 Ti, 7 prompts × 128 new tokens; plain greedy decoding gives 83.7 tok/s
measured cost modelplain greedy decoding
Data table
ktok/sspeed-uppredicted by the cost modeldraft tokens acceptedtokens per round
189.71.07×1.08×91%1.91
2102.11.22×1.24×87%2.72
3111.81.34×1.36×82%3.45
4115.01.37×1.39×79%4.10
5116.71.39×1.41×76%4.70
6116.11.39×1.40×72%5.26
8111.71.33×1.33×67%6.17

Up to 1.39× at k = 5 and 6. The draft is not much cheaper than the target: both are weight-bandwidth bound, so a 1B decode step costs 39 % of a 3B one (4.66 vs 11.85 ms), and gains level off around k = 5. Verification is cheap: checking 9 rows in one pass costs 1.31× a single 3B decode step (15.5 vs 11.8 ms), because the weights are read once for all rows. The cost model, k·t_draft + t_verify(k + 1) per round, is within 2 % at every k.

KV-cache eviction scoring (H2O, WFL)

Eviction policies decide which cached keys to drop when a sequence outgrows its cache budget. H2O keeps the keys that have received the most attention; WFL prune, from SubCache, removes the keys another kept key can best stand in for. Both need numbers the FlashAttention-style kernels don't produce, so with tracking on the attention kernels also record them: decode stores each key's logit and each head's log-sum-exp and adds the attention weights to a per-key score; prefill saves each row's log-sum-exp and a second, key-parallel pass sums each key's column. Engine.evict then compacts the kept keys, values and scores to the front of the cache. Tracking is off by default, and turning it on changes no output: attention outputs and greedy tokens are bitwise equal to tracking off, and scores are within 2e-6 relative of a float64 reference (475 kernel checks and an engine test on the RTX 2080 Ti and the RTX A6000). Costs below are on an RTX 2080 Ti, with no thermal throttling.

Decode-attention time added by score tracking (lower is better)
the attention kernel alone, batch 1, % over the plain kernel; batch 8 is in the table
Data table
modelbatchkeysplain µswith tracking µsoverhead
Llama-3.2-1B11,02411.514.9+30%
Llama-3.2-1B14,09625.431.4+23%
Llama-3.2-1B116,38470.990.8+28%
Llama-3.2-1B132,768128.7168.3+31%
Llama-3.2-1B81,02445.154.3+20%
Llama-3.2-1B84,096137.3168.8+23%
Llama-3.2-1B816,384504.4613.2+22%
Llama-3.2-1B832,768932.81,220.7+31%
Llama-3.2-3B11,02413.917.5+26%
Llama-3.2-3B14,09642.048.8+16%
Llama-3.2-3B116,384130.1150.0+15%
Llama-3.2-3B132,768244.2284.9+17%
Llama-3.2-3B81,02473.083.4+14%
Llama-3.2-3B84,096244.3280.2+15%
Llama-3.2-3B816,384930.21,070.4+15%
Llama-3.2-3B832,7681,845.22,140.1+16%
Qwen2.5-1.5B11,02411.515.2+32%
Qwen2.5-1.5B14,09627.629.1+5%
Qwen2.5-1.5B116,38452.960.2+14%
Qwen2.5-1.5B132,76881.995.7+17%
Qwen2.5-1.5B81,02433.138.1+15%
Qwen2.5-1.5B84,09687.7102.9+17%
Qwen2.5-1.5B816,384294.6356.1+21%
Qwen2.5-1.5B832,768570.8697.2+22%
Llama-3.1-8B11,02414.617.4+19%
Llama-3.1-8B14,09643.950.6+15%
Llama-3.1-8B116,384132.0155.4+18%
Llama-3.1-8B132,768245.8291.4+19%
Llama-3.1-8B81,02474.885.6+14%
Llama-3.1-8B84,096246.1286.6+16%
Llama-3.1-8B816,384931.61,098.9+18%
Llama-3.1-8B832,7681,845.32,186.4+18%

Tracking slows the decode-attention kernel by 5–32 % across four models, 1k–32k keys and batch 1 and 8, but attention is a small part of a decode step: a whole Llama-3.2-1B step at a 4.1k context goes from 4.96 to 5.05 ms (+1.7 %). In prefill, the second, key-parallel pass costs 1.0–1.8× the FlashAttention kernel's own time at 2–4k tokens and 1.4–4.5× at 512 (few kv heads launch few blocks); storing the log-sum-exp in the first pass adds at most 4 %.

Choosing what to keep. Over every (layer, kv head), H2O's top-k takes 0.03–0.10 ms at 1k–4k slots. WFL prune is the expensive policy: 9–40 ms to evict 64 slots from 1,024 (8 rounds), 0.1–0.6 s at 4,096, and 0.98–4.2 s for a full prune from 4,096 to 512. Against a PyTorch matrix version of the same algorithm at 1,024 slots (Llama-3.2-1B and Qwen2.5-1.5B), the kernels are 6.4–25.0× faster.

Prefill score pass, selection and eviction times
RTX 2080 Ti, from results/2080ti/eviction/bench_eviction.json
Llama-3.2-1B in the enginetime
decode step, 4.1k context, tracking off → on4.96 → 5.05 ms (+1.7%)
evict 4,096 → 1,024 slots, H2Oselect 0.10 ms + compact 0.88 ms
evict 4,096 → 1,024 slots, WFLselect 885 ms + compact 0.82 ms
evict 4,096 → 1,024 slots, WFL with a pool of 2 × m, 8 roundsselect 63 ms + compact 0.83 ms
decode step after the eviction4.85 ms
Prefill score pass
modelprompt tokensflash attention mswith LSE stored msscore pass msscore pass / flash
Llama-3.2-1B5120.0450.0450.1052.35×
Llama-3.2-1B2,0480.4840.4800.6691.38×
Llama-3.2-1B4,0961.7601.7632.0791.18×
Llama-3.2-3B5120.0750.0750.1041.39×
Llama-3.2-3B2,0480.7960.8010.9431.18×
Llama-3.2-3B4,0962.9462.9593.0571.04×
Qwen2.5-1.5B5120.0450.0450.2014.51×
Qwen2.5-1.5B2,0480.4300.4300.7821.82×
Qwen2.5-1.5B4,0961.5241.5261.5451.01×
Llama-3.1-8B5120.0900.0930.1381.54×
Llama-3.1-8B2,0481.0501.0461.2491.19×
Llama-3.1-8B4,0963.8923.8964.0621.04×
Selection over every (layer, kv head)
model(layer, kv head) problemsslotspolicykernel msPyTorch matrix version
Llama-3.2-1B1281,024H2O top-k0.03–
Llama-3.2-1B1281,024WFL prune to 128 (step 8), 112 rounds62.51,563 ms (25.0× slower)
Llama-3.2-1B1281,024WFL prune: evict 64 (step 8), 8 rounds9.73128 ms (13.2× slower)
Llama-3.2-1B1282,048H2O top-k0.04–
Llama-3.2-1B1282,048WFL prune to 256 (step 8), 224 rounds248.3–
Llama-3.2-1B1282,048WFL prune: evict 64 (step 8), 8 rounds33.5–
Llama-3.2-1B1284,096H2O top-k0.06–
Llama-3.2-1B1284,096WFL prune to 512 (step 8), 448 rounds1,012.0–
Llama-3.2-1B1284,096WFL prune: evict 64 (step 8), 8 rounds118.9–
Llama-3.2-3B2241,024H2O top-k0.06–
Llama-3.2-3B2241,024WFL prune to 128 (step 8), 112 rounds225.1–
Llama-3.2-3B2241,024WFL prune: evict 64 (step 8), 8 rounds34.9–
Llama-3.2-3B2242,048H2O top-k0.06–
Llama-3.2-3B2242,048WFL prune to 256 (step 8), 224 rounds904.8–
Llama-3.2-3B2242,048WFL prune: evict 64 (step 8), 8 rounds115.7–
Llama-3.2-3B2244,096H2O top-k0.09–
Llama-3.2-3B2244,096WFL prune to 512 (step 8), 448 rounds3,653.9–
Llama-3.2-3B2244,096WFL prune: evict 64 (step 8), 8 rounds416.2–
Qwen2.5-1.5B561,024H2O top-k0.10–
Qwen2.5-1.5B561,024WFL prune to 128 (step 8), 112 rounds61.1694 ms (11.4× slower)
Qwen2.5-1.5B561,024WFL prune: evict 64 (step 8), 8 rounds9.1758 ms (6.4× slower)
Qwen2.5-1.5B562,048H2O top-k0.04–
Qwen2.5-1.5B562,048WFL prune to 256 (step 8), 224 rounds243.2–
Qwen2.5-1.5B562,048WFL prune: evict 64 (step 8), 8 rounds30.0–
Qwen2.5-1.5B564,096H2O top-k0.04–
Qwen2.5-1.5B564,096WFL prune to 512 (step 8), 448 rounds980.0–
Qwen2.5-1.5B564,096WFL prune: evict 64 (step 8), 8 rounds106.3–
Llama-3.1-8B2561,024H2O top-k0.05–
Llama-3.1-8B2561,024WFL prune to 128 (step 8), 112 rounds248.5–
Llama-3.1-8B2561,024WFL prune: evict 64 (step 8), 8 rounds40.1–
Llama-3.1-8B2562,048H2O top-k0.06–
Llama-3.1-8B2562,048WFL prune to 256 (step 8), 224 rounds1,000.7–
Llama-3.1-8B2562,048WFL prune: evict 64 (step 8), 8 rounds132.7–
Llama-3.1-8B2564,096H2O top-k0.09–
Llama-3.1-8B2564,096WFL prune to 512 (step 8), 448 rounds4,152.2–
Llama-3.1-8B2564,096WFL prune: evict 64 (step 8), 8 rounds605.4–

Correctness

Speed only counts if the output is right. Greedy runs: 7 prompts × 128 tokens, single-sequence with and without CUDA graphs plus a batch of 6, compared token for token with HF transformers; any divergence lands where HF's own top-2 logits are tied or one fp16 step apart. Perplexity: WikiText-2, 1,024-token windows, both engines on the same tokens.

modelGPUsetupgreedy runs token-identical to HFPPL hipinferPPL HFtop-1 agreementmean KL(HF ‖ hipinfer)
TinyLlama-1.1BRTX 2080 Ti1 GPU20/208.51498.515199.90%6.6e-06
Qwen2.5-0.5BRTX 2080 Ti1 GPU18/2013.485113.485199.79%1.6e-05
Llama-3.2-1BRTX 2080 Ti1 GPU13/2010.463310.463199.81%1.3e-05
Qwen2.5-1.5BRTX 2080 Ti1 GPU17/209.45369.453599.86%1.6e-05
Llama-3.2-3BRTX 2080 Ti1 GPU20/208.27678.276899.82%9.5e-06
TinyLlama-1.1BA100-SXM4-40GB1 GPU20/2010.732610.732699.85%6.3e-06
Qwen2.5-0.5BA100-SXM4-40GB1 GPU20/2017.222717.223699.74%1.6e-05
Llama-3.2-1BA100-SXM4-40GB1 GPU13/2012.561812.561999.87%1.3e-05
Qwen2.5-1.5BA100-SXM4-40GB1 GPU16/2011.978711.978799.77%1.6e-05
Llama-3.2-3BA100-SXM4-40GB1 GPU20/2010.221710.221499.90%9.5e-06
Mistral-7B-v0.3A100-SXM4-40GB1 GPU20/207.32697.326799.88%4.2e-06
TinyLlama-1.1BRTX A60001 GPU20/208.51498.515099.88%6.6e-06
Qwen2.5-0.5BRTX A60001 GPU19/2013.484613.484999.74%1.5e-05
Llama-3.2-1BRTX A60001 GPU11/2010.463110.463399.86%1.3e-05
Qwen2.5-1.5BRTX A60001 GPU17/209.45349.453999.77%1.6e-05
Llama-3.2-3BRTX A60001 GPU20/208.27708.276599.88%9.8e-06
Mistral-7B-v0.3RTX A60001 GPU20/205.77745.777499.91%4.6e-06
Llama-3.1-8BRTX A60001 GPU14/206.54676.546999.89%1.1e-05
Llama-3.1-8B2× RTX 2080 Tisharded over 2 GPUs–6.54676.547199.87%1.2e-05
Mistral-7B-v0.32× RTX 2080 Tisharded over 2 GPUs–5.77735.777599.88%6.5e-06

Limits of these measurements