Eval Harness — Apple-Grade

A benchmark can lie to you. The polite ones lie in a clean table with two decimal places, right up until you promote the model and production quietly disagrees.

Ours did exactly that. The Carmack Olympics started life as a leaderboard — a pretty report card for the local models — and then it caught itself counting "windu" as a correct answer because those five letters sit inside the word "windows". A report card that grades itself is not evidence. So we stopped, and rebuilt it into something that could stand up in front of Windu with a memorization gap it had actually earned.

A benchmark that scores models in a pretty table is a report card. A benchmark that gates promotion is infrastructure. This page is the record of the crossing: how the Olympics grew a skeleton, a nervous system, and a conscience. As of 2026-04-18 it runs green — 17/17 end-to-end checks passing — and it is active.

The goal isn’t to produce nicer dashboards. It’s to make every routing decision defensible with a number you can audit six months from now, on a clean machine, with git checkout as the only setup step. Those decisions feed dynamic model routing and the champion gate; neither deserves to rest on a scoreboard that flatters itself.

The Twelve Layers

Each layer below is independent. Every one has a rollback. Every one is exercised by scripts/test_eval_harness.sh.

#	Layer	What it removes	Where it lives
1	Word-boundary scoring	False positives like "windu" matching inside "windows"	eval_common.score_test
2	Plumbed seeds (--seed)	“N=5 seeds” that were actually N=5 identical runs	seed_mlx() + CLI flag on all three evals
3	Warmup (--warmup-n)	First-task penalty from cold KV cache / Metal pipelines	eval_common.warmup
4	Repro bundle	”I can’t reproduce this result” two months later	git commit, mlx_lm version, model SHA256, full env snapshot (chip, RAM, macOS, pip hash)
5	Smoke canary (--smoke)	40 minutes wasted before noticing a broken config	3 tasks across 3 categories
6	Resumability (--resume)	Starting over when the eval crashes at task 23 of 30	Atomic per-task writes + completed_keys() skip
7	Latency + per-agent view	”Won accuracy but lost 3× in speed” going unnoticed	latency_ms per task, aggregated in eval_stats.py
8	Determinism self-test (--check-determinism)	Seed plumbing silently regressing	Two-run byte-identical assertion
9	Budget guardrails (--budget-sec, --max-tokens-total)	A runaway generator turning an hour into ten	Abort with exit 3 + partial-save
10	Refusal classifier	Score 0 meaning four different things	WRONG / REFUSED / TOO_SHORT / OFF_LANGUAGE / HALLUCINATED / UNKNOWN per row
11	Regression gate + task diff	Silent quality regressions landing in prod routing	eval_stats.py --gate + eval_diff.py (both exit 1 on failure)
12	Atomic writes + SHA256 trailer	Truncated JSON after a mid-run crash	tmp + rename + <file>.sha256

Tip

Three of these twelve (word-boundary scoring, plumbed seeds, warmup) silently invalidate any previous eval. If you have numbers from before 2026-04-18, either re-run them or caveat the claim — otherwise you’re comparing two different measurement systems.

Twelve layers, each independently rollbackable, each closing a way a benchmark can lie to you. The point of evidence isn’t to flatter the decision you already made — it’s to survive being wrong in front of yourself six months from now, on a clean machine, with somebody else reading the same JSON.

Reproducibility bundle

Every output JSON now carries enough metadata to exactly reproduce the run. Example from a recent carmack_eval.py --output eval/foo.json:

{
  "repro": {
    "git_commit": "452139ef61",
    "git_dirty": false,
    "mlx_lm_version": "0.31.2",
    "mlx_version": "0.19.3",
    "model_path": "/Users/neo/.cache/huggingface/.../Qwen3.6-35B-A3B-4bit/...",
    "model_sha256": "45797d2985a12c55",
    "env": {
      "python_version": "3.12.13",
      "macos_version": "26.4.1",
      "chip": "Apple M4 Max",
      "memory": "128 GB",
      "pip_freeze_sha256": "03ecb1d1cbb58d1d"
    }
  }
}

A .sha256 trailer file sits alongside every output so truncation is detectable:

$ shasum -a 256 -c eval/coder14b-seed42.json.sha256
eval/coder14b-seed42.json: OK

Statistical rigor in eval_stats.py

Aggregation over N seed runs now reports bootstrap CI, effect size, and Benjamini-Hochberg FDR correction across categories:

## Paired comparison: gemma4-lora (B) minus coder14b-trained (A)

**N paired tasks:** 34  |  **Mean delta:** -0.4250  (95% CI [-0.5912, -0.2603], p ≈ 0.000)
**Effect size (Cohen's d):** -1.12 (large)
**Verdict:** A > B (significant at 95%)

### Per-category delta

| Category | Mean Δ | 95% CI | Cohen's d | p | p (BH-FDR) | Sig | N |
|---|---|---|---|---|---|---|---|
| satellite | -1.000 | [-1.000, -1.000] | -3.16 (large) | 0.000 | 0.000 | ✓ | 3 |
| topology | -0.700 | [-1.000, -0.300] | -1.55 (large) | 0.000 | 0.000 | ✓ | 5 |
| home_automation | -0.500 | [-0.900, -0.100] | -1.12 (large) | 0.010 | 0.027 | ✓ | 5 |
| ... | | | | | | | |

The Cohen’s d column prevents the trap of calling a +0.02 delta “significant” when the effect is tiny. The BH-FDR-adjusted p prevents the trap of calling one cherry-picked category significant at 95% when you tested eight.

Regression gate — the CI/CD bit

eval_stats.py --gate turns the harness from a reporting tool into a promotion gate:

# Fails with exit 1 if any gated category drops more than 5 points
.venv/bin/python scripts/eval_stats.py \
  --inputs eval/coder14b-seed*.json --label candidate \
  --gate eval/last-week-baseline.json \
  --gate-max-drop 0.05 \
  --gate-categories identity,jailbreak,tool_calling \
  --json-output eval/gate-report.json

A separate scripts/eval_diff.py surfaces which specific tasks regressed — not just aggregates. Also exits 1 on any regression.

Anti-contamination

scripts/check_holdout_contamination.py runs a 3-gram Jaccard sweep of each holdout task against the training corpus. Windu’s bar for trusting a memorization gap > 0.10 starts here:

.venv/bin/python scripts/check_holdout_contamination.py \
  --holdouts data/splits-carmack/valid.jsonl \
  --training data/splits-carmack/train.jsonl \
  --threshold 0.30 \
  --output eval/contamination-report.json

Exit 1 if any holdout overlaps training content above threshold. No external deps — just stdlib re and set operations.

Real-traffic replay

scripts/replay_real_traffic.py complements synthetic benchmarks by replaying anonymized production turns through a candidate endpoint. PII scrubber (phone/email/IP) runs before anything leaves the local machine.

Caution

The extract_turns() loader ships with a placeholder schema — point --log-dir at your actual conversation logs and adapt extract_turns() to match. The harness does the heavy lifting; you wire the source.

Authenticating against the Tailscale-exposed council

The cathedral fork serves the council seat on :1337 over mTLS-only, launched with --no-plain — the plain socket is gone, not deprecated. A plain probe doesn’t get a 401; it gets Received HTTP/0.9 when not allowed, because there’s no HTTP server on the bare port at all. The TLS listener binds loopback, the Tailscale IP, and the 10.0.0.1 vmnet bridge; the cert’s SAN carries the stable MagicDNS name so it survives Tailscale IP drift. The former coder :1338 seat retired 2026-06-07; the CODER role now runs Codestral-22B (Codestral-22B-v0.1-4bit) as a separate plain-loopback service on :3301.

The PKI lives at ~/.sanctum/certs/ on every Sanctum machine. Per-client cert and key pairs are issued for each named consumer — guardian, canary, drift, sanctum-server, parity-smoke, council-offbox. A curl probe with a client identity reaches the seat:

curl --cacert ~/.sanctum/certs/ca.crt \
     --cert ~/.sanctum/certs/clients/sanctum-server.crt \
     --key  ~/.sanctum/certs/clients/sanctum-server.key \
     https://manoir.local:1337/v1/models
# -> {"data":[{"id":"38740b847e4cb78f...","object":"model"}], ...}

Caution

The Rust sanctum-mlx backend returns its model ID as a raw snapshot hash (e.g. 38740b847e4cb78f...) rather than the friendly mlx-community/Qwen3.6-35B-A3B-4bit. That hash is the model snapshot directory name — it’s stable, just not pretty. The harness’s model-ID parser handles both, but if you’re scripting around /v1/models directly, don’t assume the friendly name.

Caution

The eval CLI hasn’t caught up to the server. carmack_eval_http.py still authenticates with a Bearer token (--api-key-env COUNCIL_API_KEY) over plain HTTP — it has no --ca-cert/--client-cert/--client-key, so it physically cannot present a client cert. Against a --no-plain mTLS seat that means the connection dies before the first byte. This is the exact overclaim this page preaches against, so here it is named: the server is mTLS-only today; the client is the migration debt. Until the eval grows mTLS flags, remote council runs go through the curl probe above, or against a seat still speaking plain Bearer. See the mTLS migration for the server side that landed first.

Weekend workflow

# 1. Pre-flight: smoke canary + determinism
.venv/bin/python scripts/carmack_eval.py \
  --model ... --seed 42 --check-determinism --smoke \
  --output eval/smoke.json

# 2. Full N=5 with budget and notify
for s in 42 1138 66 2187 501; do
  .venv/bin/python scripts/carmack_eval.py \
    --model ... --adapter-path ... --seed $s \
    --budget-sec 3600 --max-tokens-total 120000 \
    --notify 'osascript -e "display notification \"eval done\""' \
    --output eval/coder14b-seed${s}.json
done

# 3. Aggregate + gate
.venv/bin/python scripts/eval_stats.py \
  --inputs eval/coder14b-seed*.json --label coder14b \
  --gate eval/last-week-baseline.json --gate-max-drop 0.05 \
  --output eval/coder14b.md --json-output eval/coder14b.json

# 4. Diff for task-level detail
.venv/bin/python scripts/eval_diff.py \
  --baseline eval/last-week-baseline.json \
  --current  eval/coder14b.json \
  --min-delta 0.1 --output eval/coder14b-diff.md

Exit code contract

Code	Meaning	Who emits it
0	All green	Any eval or stats script on success
1	Test/gate failure	test_eval_harness.sh with any failed check; --gate with regressions; eval_diff.py with any task regression; --check-determinism mismatch
2	Input/config error	Missing seed for --check-determinism; unparseable --inputs glob
3	Budget exceeded	--budget-sec or --max-tokens-total breach; partial results flushed before exit

CI pipelines can distinguish “benchmark broken” (1) from “budget exhausted, try again” (3).

What isn’t in here

Explicitly out of scope, honestly flagged:

• LLM-as-judge disagreement — Carmack uses programmatic keyword rubrics, not an LLM judge. This lives in the Standard Olympics track if needed.
• Power analysis — bootstrap CI + Cohen’s d cover 95% of what you’d ask a power calculation.
• Secret scanning on response logs — assumed low risk since prompts are authored locally; add scanning if you start committing response logs to git.

Verification

The harness lives in the private Ogilthorp3/mlx-finetune repo — so the “git checkout as the only setup step” promise starts with the clone:

git clone [email protected]:Ogilthorp3/mlx-finetune.git ~/Projects/mlx-finetune
cd ~/Projects/mlx-finetune
bash scripts/test_eval_harness.sh

Current baseline: 17 pass, 0 fail, 1 skip. The skip is the --check-determinism HTTP test against :1337 — and worth being honest about why. It’s filed as “depends on server seed support,” but the nearer cause is the same client-side debt above: the test reaches for the seat over plain Bearer, and the --no-plain mTLS server has nothing listening on the bare port. A skip that needs two sentences to explain is a skip worth fixing.