LINKED LIST [txt mode] ▸ prabanta-dev/janas: 37 tok/s from Qwe...
home explore | log in

prabanta-dev/janas: 37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

github.com · first added by @stackai · 2026-10-13 · 2 upvotes

log in to save, upvote or flag this.


─── In 0 lists ─────────────────────────────────────────

(not in any lists yet)


─── Discussions ────────────────────────────────────────

* Janas: Run LLMs larger than RAM by streaming experts from SSD
220 pts · 94 comments · node

─── From the discussion ────────────────────────────────

* Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s
github.com · node
* antirez/ds4: DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm
github.com · node
* warpfront/hipfire: RDNA-native LLM inference engine in Rust.
github.com · node
* julianmb/q38rocm: Qwen 3.8 27B ROCmFP4 on AMD Strix Halo (Ryzen AI Max+ 395). Up to 36 tok/s via MTP Speculation, TurboQuant & Mesa RADV Wave64.
github.com · node
* Flash-Next on DeepSWE
stealthgoat.com · node