2026-07-03: The flakes that only failed under load

A flaky test is a small betrayal. It passes a hundred times, fails once, and teaches the whole team to re-run instead of read. A green suite you don’t trust is worse than an honest red one. So before the first beta, the test suites got the same treatment every service did: find the flake, find the reason, fix the reason. Never the symptom.

Rule zero: root cause before any fix

Every flake got the four-phase treatment — reproduce, find the mechanism, form one hypothesis, fix and verify. “Re-run and it passed” is not a diagnosis. It’s a shrug. Two of the three flakes turned out to have precise, provable mechanisms. The third didn’t, and that changed what the right fix was. This is the same engineering discipline the services run on, pointed at the tests that guard them.

Flake one — the unit test that read the live haus

test_gather_then_voice_flattens_all_findings_into_voice_prompt checked that a two-tool council gather flattens both findings into the voice prompt. It mocked the model’s tool decision but let the tool execution run for real. So agent_list and logs_tail actually queried the live box.

Here is the trap. The voice prompt is byte-capped — 8 000 bytes, so a chatty instrument can’t blow the budget. Measured live, agent_list alone was 9 991 bytes. Over the cap by itself. The flattener truncated mid-first-tool, the second tool’s name never made it in, and the both-names assertion failed. The “flakiness” was just the running-agent count drifting across the 8 KB line over the hours of a working day.

The fix is hermeticity: stub the tool-execution boundary with small deterministic results, so the test exercises the flatten-and-accumulate logic it’s about — not the size of the live process table. One subtlety is worth keeping. The sibling tests that assert on the audit trail keep the real execution, because they test the audit write. You mock the boundary a test isn’t about. Never the one it is.

Flake two — the readiness wait that gave up quietly

The sanctum-server end-to-end tests spin up mock backends and the real router binary on ephemeral ports, then poll for readiness. Both polls had the same bug. They gave up silently:

for _ in 0..60 {                 // 6-second cap
    if health.mode == "routed" { break; }
    sleep(100ms).await;
}
// ...falls through here whether or not the server ever came up

Under load — the lib-test binary running beside the e2e binary, each e2e case spawning its own server subprocess — six seconds occasionally wasn’t enough. The loop fell through. The test ran against a half-started router, and the routing assertions failed for a reason that had nothing to do with routing.

The fix is the condition-based-waiting discipline: poll to a generous deadline and fail loudly on timeout with the last state you saw.

let deadline = Instant::now() + Duration::from_secs(30);
loop {
    if health.mode == "routed" { break; }
    if Instant::now() >= deadline {
        panic!("server never reached routed mode in 30s (last: {last})");
    }
    sleep(100ms).await;
}

A slow-but-fine start now waits instead of flaking. A genuinely broken start reports the real problem instead of hiding behind a downstream assertion.

Flake three — the ghost we hardened instead of chased

One test flaked exactly once and never again. _run_first_hello resolves the installed script through Path.home(), and in a single full-suite run home resolved to a script-less directory, so the script never ran. The mechanism was provable by elimination — a with patch(...) always intercepts, so the failure could only be the home-resolution early-return. But nine clean full-suite runs and a grep of every test and source file turned up no stray Path.home/HOME patch to blame. The ghost stayed a ghost.

Note

When a bug won’t reproduce after honest investigation, you have two moves: keep digging, or make the code immune to the whole class. Here the class is “depends on ambient home resolution,” so the fix is to pin Path.home per-test rather than trust the environment any other test could perturb. That’s not papering over a mystery — a unit test that resolves paths through Path.home() should control home directly. The ghost can’t come back through a door that’s no longer there.

The one that was a safety decision, not a typo

The workspace sweep also surfaced a test that fails every time, not sometimes: sanctum-chitti’s attention_quiet_overrides_alert. That’s not flakiness. It’s a deterministic disagreement between a test and the code, and its fix was a safety decision, not a mechanical one.

classify_posture checked the Fever breaker (acute pressure ≥ 0.95) before the “be quiet” override. So a quiet request at Fever-level pressure still returned Fever. The test expected Conserving, and a comment one layer up said quiet “wins regardless of what the lower koshas say.” Those genuinely conflicted. The two fixes were opposites: correct the test’s input to the Alert band (quiet overrides Alert, Fever keeps paging), or reorder the code so a “be quiet” can silence even a Fever breaker on a live, thermally-stressed box. One of those changes a running safety behavior in chitti’s felt-state layer. So it went to the domain owner as a question, not a commit. That’s the guess a machine shouldn’t make on its own.

The owner’s call: quiet wins over everything, Fever included. An explicit “be quiet” — recording, on-call, asleep — is the human’s context, and it outranks even the saturation breaker. So the attention_quiet check moved above the Fever check (the non-quiet path is byte-for-byte unchanged), a dedicated attention_quiet_overrides_fever test now pins the precedence, and the live chittid was rebuilt and restarted to carry it. The interim test-only fix that kept the suite green while the question was open got reverted into the real one. Ask first. Then execute the answer exactly — including the part that changes a running service.

Where it landed

The council and server flakes are fixed and merged. The First Hello test is hardened. The chitti posture rule is decided and shipped to the live daemon. The full CLI suite is green across ten runs, and every Rust crate is green. This is the same standard behind honest green: a passing suite has to mean something.

A test that passes on the second attempt has not passed. It has told you the truth once and then talked you out of it. The estate’s tests now fail for reasons, or they do not fail at all.