llama.cpp Just Got an Expert Cache. Do You Still Need Strata?
Running Local
9,645 views
llama.cpp merged an expert cache on October 7. Does it make Strata unnecessary for Qwen 3.8 Flash-Next on one GPU with a lot of system RAM?
I went through the posted numbers: the pull request's own runs, owners from an 8 GB laptop to the RTX 5090, the PCIe math, and the one test of both engines on the same PC.
If Flash-Next is your one daily model, you have 64 GB of RAM and the card sits in a full x16 slot, keep Strata. On the only same-PC test it wrote about 70 % faster than llama.cpp with the cache and read prompts over ten times faster. Set your sampling yourself, and if you installed before October 6, check that updates still work.
If you switch between models, or the history rewrite put you off, go back to llama.cpp b11480 or newer and try --moe-cache-mib. For Flash-Next on 16 GB or more in an x16 slot, the reports land around 1.5 to 2 times your old offload speed. Start the cache near 10 % of the expert size and read the hit rate in the log with -lv 4.
On an 8 or 12 GB card, a second slot wired at x4, or older PCIe, leave the cache off until your own before-and-after test shows a gain. If the whole model fits in VRAM, none of this applies.
0:00 One flag: double the speed, or half
0:19 Three options for one GPU and lots of RAM
0:34 How the expert cache works
1:14 The flag, the release and the hit rate
1:50 Big cards: about double
2:22 Small cards: where the trouble starts
3:18 Too small or too big: the U curve
3:43 The PCIe math: full slot vs second slot
4:29 How big a cache, and when it loses
5:20 Strata vs llama.cpp on the same PC
6:16 Caveats and where Strata's lead comes from
6:56 Strata's history rewrite
7:32 One model, Ollama, LM Studio, output quality
8:09 Who keeps Strata, who turns the cache on
9:23 Test it on your own box
Sources
The cache, its flag and the RTX 4090/5090 runs, plus owner reports (R9700, RTX 5090, GLM 5.3 Flash, RTX 3090, top-k loop): https://github.com/ggml-org/llama.cpp...
First release with the cache (b11480): https://github.com/ggml-org/llama.cpp...
Multi-GPU support and the two-GPU prompt numbers: https://github.com/ggml-org/llama.cpp...
RTX 4060 laptop runs: Reddit: probably_im_doing_something_wrong_using_pr29887
RTX 5090, Arc B580, RTX 3060 and RX 9070 XT runs: Reddit: 1x03xkc
Same-PC test, Strata vs llama.cpp, and the MTP finding: Reddit: 125b_moe_qwen38flashnext_at_55_toks_on_a_16_gb
Flash-Next layers and experts: https://huggingface.co/Qwen/Qwen3.8-F... and https://huggingface.co/ggml-org/Qwen3...
PCIe bandwidth: https://en.wikipedia.org/wiki/PCI_Exp... , measured 26 GB/s and x4 second slots: https://github.com/ggml-org/llama.cpp...
Second slot at x4 on a B650 board: https://www.asus.com/motherboards-com...
Strata history rewrite and the broken update: https://github.com/Niko1221/Strata/is... and https://github.com/Niko1221/Strata/re...
Strata runs one model and decodes greedy by default: https://github.com/Niko1221/Strata/bl... , https://github.com/Niko1221/Strata/bl... and https://github.com/Niko1221/Strata/pu...
Ollama and LM Studio builds: https://github.com/ollama/ollama/blob... and https://lmstudio.ai/changelog
Credits
Thumbnail llama.cpp logo: ggml-org llama.cpp (MIT licence repo), via the svgl.app logo collection, recoloured flat.
Your hardware, your rules.
7 Comments
llama.cpp Just Got an Expert Cache. Do You Still Need Strata?
Running Local
9,645 views
llama.cpp merged an expert cache on October 7. Does it make Strata unnecessary for Qwen 3.8 Flash-Next on one GPU with a lot of system RAM?
I went through the posted numbers: the pull request's own runs, owners from an 8 GB laptop to the RTX 5090, the PCIe math, and the one test of both engines on the same PC.
If Flash-Next is your one daily model, you have 64 GB of RAM and the card sits in a full x16 slot, keep Strata. On the only same-PC test it wrote about 70 % faster than llama.cpp with the cache and read prompts over ten times faster. Set your sampling yourself, and if you installed before October 6, check that updates still work.
If you switch between models, or the history rewrite put you off, go back to llama.cpp b11480 or newer and try --moe-cache-mib. For Flash-Next on 16 GB or more in an x16 slot, the reports land around 1.5 to 2 times your old offload speed. Start the cache near 10 % of the expert size and read the hit rate in the log with -lv 4.
On an 8 or 12 GB card, a second slot wired at x4, or older PCIe, leave the cache off until your own before-and-after test shows a gain. If the whole model fits in VRAM, none of this applies.
0:00 One flag: double the speed, or half
0:19 Three options for one GPU and lots of RAM
0:34 How the expert cache works
1:14 The flag, the release and the hit rate
1:50 Big cards: about double
2:22 Small cards: where the trouble starts
3:18 Too small or too big: the U curve
3:43 The PCIe math: full slot vs second slot
4:29 How big a cache, and when it loses
5:20 Strata vs llama.cpp on the same PC
6:16 Caveats and where Strata's lead comes from
6:56 Strata's history rewrite
7:32 One model, Ollama, LM Studio, output quality
8:09 Who keeps Strata, who turns the cache on
9:23 Test it on your own box
Sources
The cache, its flag and the RTX 4090/5090 runs, plus owner reports (R9700, RTX 5090, GLM 5.3 Flash, RTX 3090, top-k loop): https://github.com/ggml-org/llama.cpp...
First release with the cache (b11480): https://github.com/ggml-org/llama.cpp...
Multi-GPU support and the two-GPU prompt numbers: https://github.com/ggml-org/llama.cpp...
RTX 4060 laptop runs: Reddit: probably_im_doing_something_wrong_using_pr29887
RTX 5090, Arc B580, RTX 3060 and RX 9070 XT runs: Reddit: 1x03xkc
Same-PC test, Strata vs llama.cpp, and the MTP finding: Reddit: 125b_moe_qwen38flashnext_at_55_toks_on_a_16_gb
Flash-Next layers and experts: https://huggingface.co/Qwen/Qwen3.8-F... and https://huggingface.co/ggml-org/Qwen3...
PCIe bandwidth: https://en.wikipedia.org/wiki/PCI_Exp... , measured 26 GB/s and x4 second slots: https://github.com/ggml-org/llama.cpp...
Second slot at x4 on a B650 board: https://www.asus.com/motherboards-com...
Strata history rewrite and the broken update: https://github.com/Niko1221/Strata/is... and https://github.com/Niko1221/Strata/re...
Strata runs one model and decodes greedy by default: https://github.com/Niko1221/Strata/bl... , https://github.com/Niko1221/Strata/bl... and https://github.com/Niko1221/Strata/pu...
Ollama and LM Studio builds: https://github.com/ollama/ollama/blob... and https://lmstudio.ai/changelog
Credits
Thumbnail llama.cpp logo: ggml-org llama.cpp (MIT licence repo), via the svgl.app logo collection, recoloured flat.
Your hardware, your rules.