Getting more tokens/sec out of epsilon's llama.cpp

TL;DR: what matters, after testing

HyperQwen (patched vLLM) vs llama.cpp: measured 23 Sep

syv-ai/HyperQwen is a 38-patch series on vLLM 0.29 tuned for Qwen3.8-27B on 24 GB cards: requantized heads, a calibrated MTP draft vocabulary, and DFlash2. It cannot load GGUF, and HauhauCS publishes only GGUF, so the test used Ar4ikov/Qwen3.8-27B-Uncensored-AWQ-W4A16-ASYM-HyperQwen-fast. That is the same architecture and parameter count with a different uncensoring (abliteration, not Hauhau's fine-tune), so speed transfers but behaviour may differ.

Method: haufastmtp stopped for 23 min, each config benched alone, and haufastmtp auto-restored. 4 prompts (code, explanation, math, story) × greedy and sampled (T=0.7) × 2 reps, 512-token answers, thinking off, one request at a time, after an untimed warm-up. Decode = tokens ÷ (last − first token time). Both llama.cpp configs ran on a private port so agent traffic couldn't touch them.

• 2.05× on one GPU, 1.7× on two. The gain comes mostly from speculative decoding: vLLM's MTP accepts 3.3–3.6 tokens per step on these prompts versus about 2.0 for llama.cpp FastMTP. Faster int4 Marlin kernels and CUDA graphs add the rest. It also frees GPU1 entirely in the 1-GPU config.
• TP=2 is slower than TP=1 here (112 vs 136). vLLM logged "Custom allreduce is disabled because your platform lacks GPU P2P capability", so each layer's all-reduce goes over GPU1's Gen2 x4 chipset link through host RAM. The repo's own 2×3090 test on PCIe 4.0 x8 got +27% from TP=2; this board gets −18%. That is direct evidence for the x8/x8 board recommendation.
• DFlash2 ≈ MTP (111 vs 112): longer accepted drafts (4+ tokens) but a costlier drafter. It wins on math/code, loses on prose.
• Quality spot checks passed: correct 3-point science answer, 17×23 = 391, tool call with valid JSON arguments, image colours and positions correct, and a needle found at 90k tokens.
• Costs: boot takes about 4–5 min (vs ~20 s for llama.cpp), and it uses almost all VRAM on the cards it runs on. Prefill on the 90k-token prompt at TP=2 took 167 s (~540 tok/s), in the same range as llama.cpp's 641 tok/s at 122k. The repo claims ~1,440–1,940 tok/s prefill at 1k on one card; I did not measure short-prompt prefill.

Nova dashboard: one-click drop-in on :8094

New entry "HyperQwen vLLM" (bolt icon), next to "HauhauCS FastMTP". Clicking Start stops haufastmtp (same port and GPUs) and boots HyperQwen on :8094 under the same model name, qwen38-hauhau, so clients need no changes. To switch back: Stop it, then Start "HauhauCS FastMTP". Wiring follows the existing pattern: ~/hyperqwen-run.sh + conf/hyperqwen.conf, editable in the dashboard; use recreate after edits. Both entries now show status by their own container rather than by port, so only the active one lights up.

Profile choice (in the conf): the default is TP=2 / 262k, which matches today's endpoint (agents send 66–122k prompts) at 112 tok/s. For about 136 tok/s set GPUS="0" TP=1 MAX_LEN=65536 and recreate; that also frees GPU1 for Gemma. Requests over 64k would then be rejected.

Current state (what's actually running)

Hardware

Live llama.cpp server (container haufastmtp)

llama-server -m Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-Q4_K_P.gguf \
  -ngl 999 -c 262144 -fa on --jinja --tools all --mmproj …BF16.gguf \
  --spec-draft-model …FastMTP-32K.gguf --spec-draft-ngl 99 \
  --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0 \
  -ub 256 -sm layer          # 4 slots, unified KV, f16 KV cache

Red flags found

Config & software changes (no hardware, no money)

Ranked by expected gain on epsilon. All of these need a container restart later. Test them on a spare port or while the model is idle; nothing here has been applied.

1 · "Fast" profile: one GPU, bigger batches tested: −20% TG

Measured 23 Sep: this exact profile gave 53.1 tok/s vs 66.1 for the deployed 2-GPU config, so it is not recommended as written. Most likely cause: the q8_0 KV cache (dequant cost in flash-attention). If you retry, keep f16 KV and change one knob at a time. The text below is the original first-pass reasoning, kept for the record.

The Q4_K_P model (17.9 GB), the FastMTP sidecar (0.9 GB) and the mmproj fit on one 3090. Earlier on this box, 64k context with q8 KV fit at 23.6 GB. Use -sm none (or --device CUDA0) so there's no cross-GPU hop and the MTP prefill penalty disappears:

CUDA_VISIBLE_DEVICES=0 GGML_CUDA_GRAPH_OPT=1 \
llama-server -m …Q4_K_P.gguf -ngl 999 -sm none -c 65536 -np 1 \
  -fa on -ub 1024 -b 2048 --jinja --tools all --mmproj …BF16.gguf \
  --spec-draft-model …FastMTP-32K.gguf --spec-draft-ngl 99 \
  --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0

• Trade-off: context drops from 262k to about 64k, or about 40k with f16 KV. Keep the current 2-GPU config as a "long-context" profile, and switch between them from the Nova dashboard.
• Bonus: GPU1 becomes free for Gemma4-26B-A4B (~200+ tok/s), a second Qwen instance, or ComfyUI. That doubles total throughput even when single-stream speed is the same.
• -np 1 when there's a single user: all KV goes to one stream and scheduling overhead drops. Keep 2–4 slots only if agents really call it concurrently.
• GGML_CUDA_GRAPH_OPT=1 enables concurrent CUDA streams, single-GPU only.

2 · Batch sizes for prefill +50–100% PP

Even if you stay on two GPUs: -ub 1024 -b 2048, or -ub 2048 -b 4096 if VRAM allows. The compute buffer grows by roughly 1–2 GB, and there is 4–5 GB free per card now. Large-batch prefill on a 3090 is compute-bound, so this is where the power limit (below) starts to matter.

3 · GPU clocks (live, no restart) +5–10% TG

# GPU0 values proven on this box in July (Gemma decode 145→158 tok/s)
sudo nvtool -i 0 --setmemoffset 1500 --setcoreoffset 200
sudo nvidia-smi -i 0 -pl 330        # helps prefill; TG is bandwidth-bound
# GPU1: only after its cooling is fixed; start at +800 mem (it hung with OC before)
# P2-state memory clock: driver ≥580 honours CUDA_DISABLE_PERF_BOOST=1 in the container env

• These can be applied without stopping anything, but they are not persistent. Put them in a boot hook (HiveOS nvidia-oc.conf with the right 2-GPU arrays, or a systemd one-shot).
• Optionally lock the core clock (nvtool --setcore 1700) to cut heat and power while memory stays high. Token generation doesn't use the extra core MHz.

4 · Speculative-decoding knobs ±5%

• Keep --spec-draft-n-max 2. Independent 3090 tests find n=2 beats n=3 for Qwen3.8 MTP (66.4 vs 63.3 tok/s). The August FastMTP test here used n=3, so an A/B on your real prompts is worth doing.
• For long outputs (over ~900 tokens), --spec-draft-p-min 0.75 --spec-draft-n-max 6 reportedly gave +68% on a 3090 Ti. The gain comes from drafting deeper only when the drafter is confident.

5 · KV cache & context avoid regressions

• Qwen3.8 is a hybrid model: only 16 of its 64 layers keep a KV cache, so the KV cache is small and quantizing it saves little. There is an open report of generation collapsing about 25× past 80k context with a quantized KV cache on hybrid models. For long-context work keep f16 KV. Use q8 only for the 64k single-GPU profile, and measure it.

6 · Rebuild on a newer base enabler

• The image is pinned to commit 4df29be (Aug) on CUDA 12.6. Rebasing the FastMTP patch onto current master picks up DFlash2 (#27342), -sm tensor (#19378), and ongoing MTP + multi-GPU fixes. Build in the existing nvidia/cuda devel container with -DCMAKE_CUDA_ARCHITECTURES=86 (add -DGGML_CUDA_NCCL=ON only if you'll try tensor mode).
• Keep the current image tagged so you can roll back.

7 · Minor host tweaks 1–3%

• CPU governor ondemand → performance. It trims token-launch latency in the fully-offloaded, CUDA-graph path.
• Free space on the NVMe (93% full). Unused model folders (qwen36-27b, qwen3vl-32b-heretic, qwen38-27b, dangling 7.5 GB images) cost load-time headroom and wear.

DFlash & other drafters

Why DFlash doesn't win here yet: it drafts a whole block in one diffusion step, so acceptance length is higher (~5 vs ~2.7). But Qwen3.8's linear-attention layers must replay their recurrent state for every rejected draft token, which eats the gain in llama.cpp. Engines that handle this better (ExLlamaV3 fork, vLLM) show much larger numbers, though mostly on greedy, short-context benchmarks.

Other engines & community mods

Hardware options (ranked by value)

The honest ceiling: one 3090 generates at ≤ 936 GB/s ÷ ~17 GB of weights ≈ 55 tok/s before speculation. Spec decoding multiplies that by roughly the accepted draft length. Two 3090s only add speed with true tensor parallelism, and that needs a fast GPU-to-GPU path, which the chipset x4 slot can't provide. So the cheapest real gains are: one GPU per model, prefill batch tuning, clocks and cooling.

Live tests (23 Sep): core-clock cap & GPU0 memory offset

Both tests ran on the live server under real agent traffic, one setting at a time, in 3-minute phases with a baseline between each. Safety measures: automatic revert on exit, a detached watchdog that forced defaults after 15–20 min, and a dmesg Xid + health check every 10 s. Both reverted cleanly (offsets 0, clocks unlocked) with zero Xid errors.

A · Core-clock cap, both GPUs (nvidia-smi -lgc 210,<cap>)

• Cap 1700: about −20 W per card (−9%) and GPU1 −4 °C, with no measurable generation loss. Average clocks barely moved; the cap mainly removes the 1900+ MHz boost spikes that cost power for no gain in memory-bound decode. free win
• Cap 1400: about −40 W per card (−17%), GPU1 −12 °C and GPU0 −5 °C. Generation stayed within the baseline noise, but prompt processing fell by roughly 45% (few samples), as expected, because prefill is compute-bound.
• *TG/PP are aggregated from the server's own per-request logs in each phase. With 1–4 concurrent agent streams they are noisy: the three baselines alone spread 25–35 tok/s. The power, clock and temperature columns are clean.

B · GPU0 memory offset (nvtool -i 0 --setmemoffset)

• Bandwidth scales linearly with the offset: +2.5% / +5.2% / +7.8%. Decode speed tracks bandwidth, so expect about +7% generation when the model runs only on GPU0. In the current 2-GPU split, where GPU0 holds about half the layers, expect about +3.5–4%.
• Stability: 0 Xid errors, and the server stayed healthy at every step. Caveat: live traffic dropped off midway, so +1000 and +1500 were validated under the probe's load, not full agent load. Soak-test +1500 under real load for 30+ min before persisting it. GPU0 ran +1500 for Gemma in July without issues.
• The offset also moves the CUDA P2 memory clock (9501 → 10251 MHz). The earlier concern that P2 ignores the offset does not hold on driver 580 with nvtool.

Recommended persistent settings: nvidia-smi -lgc 210,1700 on both cards, plus nvtool -i 0 --setmemoffset 1500 on GPU0 after a soak test. Leave GPU1's memory at stock until its cooling is fixed. Put them in a boot one-shot, and fix the stale 6-GPU arrays in HiveOS nvidia-oc.conf so HiveOS doesn't overwrite them.

Suggested order

Sources

• llama.cpp #22105 DFlash
• llama.cpp #27342 DFlash2
• llama.cpp #25173 DSpark
• llama.cpp #19378 -sm tensor
• llama.cpp #27858 MTP + tensor (draft)
• llama.cpp #27428 MTP halves PP on split
• llama.cpp #27623 quant-KV collapse >80k
• llama.cpp multi-gpu.md
• Qwen3.8-27B spec-decoding on RTX 3090 (hackmd)
• z-lab/Qwen3.8-27B-DFlash2-GGUF
• z-lab/dflash
• ik_llama.cpp · -sm graph #1022 · Qwen3.8 MTP #2369
• club-3090 · PCIe P2P guide
• aikitoria P2P driver
• VeroFess 2×3090 Qwen3.8 fork
• r0b0tlab EXL3 + DFlash2
• HyperQwen vLLM single 3090
• Dual-3090 batch tuning (sanj.dev)
• p-min / n-max MTP tuning (neoteric)
• NVIDIA: P2 forced state / CUDA_DISABLE_PERF_BOOST
• GGML_CUDA_GRAPH_OPT