2026-05-15: Four Candles For The Cathedral

The cathedral was fast. It was not done being fast.

John Carmack has a habit worth stealing: when a system already runs quick, he asks where the last cycles hide, not whether any are left. So after the fusion work took the coder to 50 tok/s and ported clean onto Yoda’s 35B, we put that same question to Qui-Gon, who never met a warm cycle he didn’t want to reclaim. Where do the cycles actually go now?

Not into matmul — matmul was already fused. They were leaking into KV cache reads, kernel-launch overhead, and the dead weight of a fallback path nobody used in production. Four answers came back. Four candles for the cathedral, lit one at a time, in the order they earned their light.

#1 — Drop the fallback weights after FusedXxxProj builds

The fusion pattern kept the canonical q_proj, k_proj, v_proj (and gate_proj, up_proj) alive next to the concat’d fused weight. That fallback existed for the env-var-dynamic toggle: if someone disabled fusion mid-flight, the original 3-matmul path had to still work.

The dynamic toggle is theoretical hygiene. The memory is real. With the canonicals retained, every fused layer held about 3x the projection weight pool — on the 35B, roughly 6 GB of resident memory that inference never touched.

sanctum-rs 0adb32b (PR #20) makes the surgical move: after try_build succeeds, force-eval the concat’d weights to materialize them into their own buffers, then replace q_proj/k_proj/v_proj with 1x1 placeholder Linears. MLX refcount drops the originals immediately. The fallback path is dead in production; the env-var toggle survives for build-time-only A/B testing.

Bisects still need the canonicals alive for their parity comparison, so SANCTUM_MLX_FUSION_KEEP_FALLBACK=1 is a one-line escape hatch set at the top of main. Production unset; bisects set. All three regression bisects (qwen2 QKV, qwen2 gate+up, qwen3_5_moe QKV+MLP) stay byte-exact across 20 decode steps with populated cache.

#2 — Prewarm the prompt cache at boot

The prompt cache pool’s LCP matching means a second request that shares a system-prompt prefix with the first gets that prefix for free. The first request still pays the prefill. What if it didn’t have to?

sanctum-rs 42de932 (PR #21) adds SANCTUM_MLX_PREWARM_FILE. At AppState construction the cathedral reads the file, tokenizes it, runs forward_last_logit to fill a fresh KV cache, and drops the resulting (tokens, cache) pair into slot 0 of the prompt cache pool. Future requests LCP-match the slot and skip the prefix prefill.

Validated live: a request whose system prompt matched the prewarm file logged cache_hit_ratio=0.73 on its first cold request — 45 of 62 prompt tokens already in cache at boot. Output correct; the remaining 17 tokens are the only prefill cost.

The catch is subtle. The prewarm tokens must be a strict token prefix of typical requests. Tokenize-then-truncate-mid-template breaks the LCP match, because the extra cached tokens become “ghost context” the model was never supposed to see. Concretely: include <|im_start|>system\n plus the system content, but not the closing <|im_end|> — the closing tag and what follows are request-specific. That caveat is the first thing in the source a future author who edits the prewarm file will read.

#3 — KV cache compression audit

Memory said CompressedKVCache was somewhere in the tree. Was it actually on the hot path, or a good idea that never got wired in?

Audit complete: the --turboquant flag on the Yoda plist selects CacheKind::TurboQuant, which routes the full sampling stack through crate::turboquant::CompressedKVCache. Yoda is already running with its cache reads halved by 4-bit compression. The 14B coder runs an Fp16 cache (full precision); extending TurboQuant there would need a calibration adapter — a separate workstream.

The audit took 30 seconds and the result was “nothing to do here.” That is still a hack worth running, because the cost of not running it is rebuilding what was already shipped.

#4 — PLD on Yoda 35B

Cathedral memory had prompt-lookup decoding on the 14B coder, logging pld_max_lookup=5 pld_ngram_size=3. The Yoda plist set neither env var, so PLD was effectively off for the 35B — even though decode_with_pld_fp16 is generic over LoadedModel and would route through qwen3_5_moe just fine.

sanctum-rs 0adb32b (PR #20, same commit as #1) adds the two env vars to the Yoda plist. The log line AppState: prompt-lookup decoding ENABLED pld_max_lookup=5 pld_ngram_size=3 now fires on Yoda startup. The win shows on repetitive prose where n-gram lookups land; freeform creative output is unchanged.

What’s live

Surface	Phase 1 fusion	PLD	Carmack #1 drop	Carmack #2 prewarm
Coder :1338 (14B)	QKV + gate+up	yes	yes	yes
Yoda :1337 (35B MoE)	QKV + gate+up	yes	yes	n/a (TurboQuant path has no prompt cache pool)

The 14B coder ships at 35–50 tok/s on freeform completions, with cache_hit_ratio=0.73 on the first cold request when the matching system prompt is configured. The 35B Yoda ships at 43–55 tok/s on short freeform, PLD ENABLED in the log.

What’s not in this batch

Memory-budget-aware routed-experts fusion is the thread left dangling. PR #19 explored fusing the 256 routed experts in the qwen3_5_moe MLP path; it produced byte-exact math via the moe bisect and a 6x perf regression on the live 35B. Root cause: concatenating the per-expert gate+up weights across 40 MoE layers added roughly 10 GB of duplicated weight memory while the originals were kept around for the fallback path.

Carmack #1 — drop the fallback — is exactly what unblocks it. The next session inherits a clean surface: build the routed-experts fused weight, force-eval it, then drop switch_mlp.gate_proj and switch_mlp.up_proj. The FusedSwitchGateUp struct and its try_build are already checked in as documented dead code with the why-not comment, waiting for someone to wire them back into forward.

Note

The “ask Carmack for other hacks” prompt produced four answers in five minutes: three one-PR ships and one audit that found the work already done. That is the whole point of the exercise. Brainstorming with the right voice in the room costs nothing and consistently beats drawing a roadmap from a blank page. Four candles, and only two of them cost us any wax.