2026-04-20: The A+ Roadmap Closes

One day, five sittings, and by the last one the first six Living Force principles had all finished becoming code. The morning shipped the integrity probe. Noon shipped the Rust router. An afternoon push closed four more A+ gaps. A second afternoon killed the primary for real and watched the request land elsewhere. Late at night, a routine reboot revealed the Python fallback had been quietly winning a race nobody had noticed.

Each sitting is below. None was remarkable alone; together they are the day the week-old doctrine became code.

Morning — Rule 1 Becomes Code

Yesterday’s entry ended with three rules and a roadmap pointer. This morning the top one became a running LaunchAgent: com.sanctum.council-integrity, every five minutes on the Mini. Four checks, any one fails the whole thing, and it alerts Force Flow at error severity on its own channel — not buried in the composite drift report that hid a dead binary for sixteen hours.

Check	How	Catches
Binary exists + executable	stat on target/release/sanctum-mlx	the thing that was gone for sixteen hours
Signature verifies	codesign --verify --strict (Gatekeeper’s own first-run quarantine check)	revoked cert, modified bytes, truncation
Metallib colocated	sibling file next to the binary	Metal load panic — dead before it serves a byte
Running-process inode == on-disk inode	compare the two	the stale “rebuilt but not restarted” process; next kickstart -k flips behaviour

Smoke-tested on the MBP first, it flagged a missing metallib and a stale process inode — both legitimate on a machine just resigned — and returned clean on the Mini’s production binary. It runs on its own track, separate from guardian’s 60-second HTTP probe, canary’s 10-minute chat probe, and drift’s hourly SHA compare — one probe per failure mode. The blind spot lasted sixteen hours because binary_missing rode inside a composite drift_detected event beside expected noise — guardian.sh drift (Mini ahead of the repo), repo_dirty (turboquant in flight). Bundled alerts get filtered. Dedicated channels don’t.

Rule

An alert you bundle with expected noise is an alert you’ve already silenced. Every serious failure mode gets its own probe, its own cooldown, its own channel. One probe checking four things is strictly worse than four probes each emitting a specific alert.

Post-Entry State

Mini: Apple-notarized (submission 00000000-0000-0000-0000-000000000001, Accepted)
Channels live: guardian + canary + drift + parity-smoke + integrity + 2 off-box = 7, each its own cooldown
Cadence: council-integrity every 5 min
Commit: ec5867b   Roadmap: item #10 closed

Noon — Last Two Doors Closed

Two items had sat on the roadmap overnight. The ASC API key was still Admin-scoped — we’d asked for Developer at creation but got Admin, unnoticed until the next day. And sanctum-server, the Rust router that fronts every council request and holds the HA fallback_urls wiring, was a 44-hour-old orphan running a binary cargo clean had deleted. Both closed in one session.

ASC API Key — Developer Scope

The old key (PLACEHLDR0, Admin) was revoked from appstoreconnect.apple.com/access/integrations/api; a new one minted with Developer role (PLACEHLDR1, same issuer 00000000-0000-0000-0000-000000000002). The .p8 downloads once — Apple’s rule — so it went to two homes at mode 0600, and xcrun notarytool store-credentials sanctum re-ran on both machines. A smoke submission came back status: Accepted in about 90 seconds.

MBP:  ~/.appstoreconnect/AuthKey_PLACEHLDR1.p8       (0600)
Mini: ~/.keys/holocron-notary/AuthKey_PLACEHLDR1.p8  (0600)
Archived revoked key, both machines: *.revoked-2026-04-20  (257 bytes each)

Keeping the revoked .p8 is deliberate: revocation is authoritative, but the file records what shipped under which identity. If the new key leaks, the blast radius is now Developer-only — submit to notary, read team names — where Admin could once alter team membership and agreements.

sanctum-server, Actually Live

The sanctum-server Rust binary — the smart router, not the Python sanctum-server-mlx it replaced a lifetime ago — had run on the Mini since April 17 against a model path that no longer existed. Nobody was talking to it; it kept a socket open out of habit. Four steps put a real one in place:

1. Build   PATH=/opt/homebrew/bin:... cargo build --release -p sanctum-server
             6.8s — deps cached from the sanctum-mlx build earlier
2. Sign    Developer-ID codesign, hardened runtime + secure timestamp
             identity Bertrand Nepveu (GJ994MN2YF), same as sanctum-mlx
3. Install com.sanctum.server.plist in ~/Library/LaunchAgents/
             --router-config /Users/neo/.sanctum/instance.yaml   (engages the smart router)
             --host 127.0.0.1 --port 8900                         (loopback only)
4. Wrap    sanctum-server-launch reads ~/.sanctum/secrets/council-mlx.token (0600),
             exports COUNCIL_API_KEY, then exec's the binary — ProgramArguments point
             at the wrapper, so the token never sits in the mode-644 plist

Local clients reach it at http://127.0.0.1:8900/v1/chat/completions; cross-machine traffic still hits sanctum-mlx directly, or crosses Tailscale to the MBP shadow. The HttpProxyBackend reads api_key_env: COUNCIL_API_KEY from instance.yaml and forwards Authorization: Bearer <token> upstream — the reason the env gets wired at all.

[Client] → 127.0.0.1:8900 (sanctum-server, smart router)
           ├─ council-secure ──► 127.0.0.1:1337 (sanctum-mlx, primary)
           │                     └─ fallback ──► 100.0.0.55:8902 (MBP shadow, bearer'd)
           ├─ council-ops   ──► 127.0.0.1:1234 (LM Studio)
           ├─ coder         ──► 127.0.0.1:1234 (LM Studio)
           └─ cloud         ──► https://openrouter.ai

Startup logged Registered backend backend=council-secure url=http://127.0.0.1:1337/v1 fallback_urls=["http://100.0.0.55:8902/v1"], and model: "council-secure" returned "2 + 2 equals" in 8.7 s — through the primary, the token ignored on the localhost side. The HA failover path is now wired end to end: code (P7.2), config (instance.yaml), upstream auth (the wrapper), router live. Killing sanctum-mlx to watch the request land on the shadow is a one-liner away; we left it for the next sitting, since doing it while auto-recovery is armed risks a promotion cascade.

Rule

Secrets move in environment variables loaded from 0600 files, not in plists. A LaunchAgent plist is world-readable; a bearer token in an EnvironmentVariables block is plaintext in a public file — security theater. A launcher that reads the mode-0600 secret at startup keeps the surface to one file and one process.

Post-Entry State

Notary: MBP + Mini both on Developer-scoped PLACEHLDR1; Admin key revoked, .p8 archived
Router: sanctum-server live at 127.0.0.1:8900 on the Mini, HA failover wired end-to-end
Monitors: 6 channels green (5 sidecars + sanctum-server health); mTLS clients 2/6 (canary, guardian)
Commits: sanctum-rs 205c5cb, sanctum-docs this entry

Afternoon — Four A+ Gaps Closed

The morning had made rule 1 code. This push closed four more roadmap items at once — metrics, transport, remediation, wiring.

Streaming-Path Metrics

The P7.1 Prometheus instrumentation only covered the non-streaming path; streaming ran through tokio::spawn’d futures that never called record_inference. Now it does, and the histograms combine across both paths:

stream_started       captured at spawn
prompt_tokens_count  threaded from the initial tokenizer encode
finish_reason        threaded from the sampling result
record_inference     fired at the end of the streaming task
verified: sanctum_mlx_inference_completion_tokens_total 0→8, requests_total{stop_reason="stop"}=1

First Client on mTLS

council-canary now auto-detects the CA cert and its client cert at ~/.sanctum/certs/clients/canary.{crt,key} and switches to https://127.0.0.1:1338 when they exist. Its log carries a transport field — "mtls" or "plain-loopback" — so future Prometheus can watch the migration per probe. Delete the certs and the next tick reverts to bearer, no code change. The pattern is there for the other probes to copy.

Integrity Auto-Failover

The morning’s probe alerted on a missing binary but didn’t fix it. Now the two “next respawn WILL fail” classes trigger a promotion:

binary-missing / codesign-verify-failed:
  write FALLBACK_LOCK → launchctl bootout the Rust agent → bootstrap Python com.sanctum.server-mlx → notify
  (same recipe as council-guardian, duplicated here so it works even if guardian is hung)
metallib-missing / inode-mismatch:
  alert only — a rebuild or kickstart recovers those without promoting to Python

The zombie-listener now recovers within five minutes instead of going dark.

HA Failover Wiring

instance.yaml gained fallback_urls: [http://100.0.0.55:8902/v1] for council-secure, and the MBP shadow now enforces bearer auth matching the Mini’s token, so the router’s api_key_env: COUNCIL_API_KEY works against it. Exercising it live still needs sanctum-server rebuilt — the running one is that 44-hour-old orphan, cleaned up at noon. The config is ready for the next deploy.

Rule

Ship code that remediates, not just code that reports. An alert is an IOU to a human; an auto-remediation is a service that stays up without one. A probe that names a failure class should, where it can, know that class’s canonical recovery and run it — then alert about the recovery, a far calmer sentence than “we’re broken.”

Post-Entry State

Mini: 5 channels green (guardian, canary, drift, parity-smoke, integrity)
Canary on mTLS: transport:"mtls", 501 ms probe (down from ~2s bearer — loopback bypass
                still took the HTTP path; mTLS stays at the TLS layer)
Shipped: roadmap #2 (canary), #5 (streaming metrics); #10 (integrity) reinforced with auto-recovery
Orphan sanctum-server cleaned up. Commit 60b2154 on feat/proxy-hardening

Afternoon — HA Failover Under Real Failure

The last unchecked box was exercising the failover under an actually-dead primary — not mocking a URL, not reading the code, but running launchctl bootout on the Rust sanctum-mlx agent, sending a real request, and watching where it lands.

The Test

Guardian and canary were paused first so their restart logic wouldn’t interfere. A baseline chat through sanctum-server (model: "council-secure", seven words in) returned "Understood" in 12 s — router to primary http://127.0.0.1:1337/v1 to Rust mlx and back. Then launchctl bootout gui/$(id -u)/com.sanctum.mlx, :1337 and :1338 gone per lsof, and the request re-sent.

INFO  Direct backend match model=council-secure backend=council-secure
WARN  backend connect failed, trying next
      backend=council-secure
      url=http://127.0.0.1:1337/v1
      error=error sending request for url (http://127.0.0.1:1337/v1/chat/completions)
INFO  failed over from primary
      backend=council-secure
      winning_url=http://100.0.0.55:8902/v1

"Understood" came back in 0.34 s. The whole chain — Mini router, Tailscale hop to the MBP, the Authorization: Bearer from the 0600 token file through the sanctum-server-launch wrapper’s env, the shadow accepting it, the MBP’s Rust mlx already warm from the morning’s probes — worked first try, exactly as P7.2 drew it up. Bootstrap the Mini agent back, wait ~45 s for the Metal context to reload, resend: no “failed over” line, router back on the primary. Nothing stale, nothing sticky.

What Fought Back

The Python fallback plist we’d disabled two entries ago kept winning the :1337 race — boot out Rust, Python appears; force-kill it, KeepAlive respawns it (exit code -15 logged). Disabled=true in a file is read at bootstrap; an already-bootstrapped agent keeps its old settings until re-bootstrapped. Three moves fixed it:

launchctl disable gui/<uid>/com.sanctum.server-mlx   # state-level, not a file edit
launchctl bootout <label>                            # clear the stale bootstrap
pkill -9 -f mlx_lm.server                            # the live process KeepAlive kept resurrecting
# then launchctl print-disabled shows it disabled, and :1337 finally goes to nothing

One thing broke us back: launchctl bootstrap gui/$UID ~/Library/LaunchAgents/com.sanctum.mlx.plist returned Bootstrap failed: 5: Input/output error twice — the label already registered from a previous lifetime. Recovery: bootout the label, bootstrap fresh, kickstart -k.

Rule

Disabled=true in a plist is a hint; launchctl disable is enforcement. A plist edit doesn’t touch an already-bootstrapped agent — reboot, fully bootout the label, or disable it against the running launchd state.

Corollary: launchctl bootstrap failing with “Input/output error” means the label survives in launchd’s state from a previous lifetime. bootout first; don’t keep retrying.

Post-Entry State

Failover: exercised under live failure — 0.34s from "primary down" to "fallback responding"
Serving: Mini + MBP on canonical ports; 5 Mini sidecars + 2 MBP off-box watchers green
Python fallback: disabled in launchd state AND in the plist — nothing respawning uninvited
Binary: Dev-ID-signed, notarized under Developer-scoped PLACEHLDR1
=> every A+ roadmap item shippable this week is shipped. Commit: sanctum-docs this entry

Late Night — The Reboot That Almost Wasn’t

The Mini rebooted itself around midnight — pending update, kernel panic, or gravity; cause unknown. It came back in two minutes at a load average of 95, every LaunchAgent firing at once, the Rust sanctum-mlx loading a 27-billion-parameter model into Metal while thirty-seven other services raced it for the GPU. All five monitor channels re-bootstrapped and Rust came up on :1337 and :1338 untouched; the off-box canary logged canary_ok throughout, off-box drift one ssh timeout while the Mini was unreachable, then recovered. The Living Force as advertised — except Python won the :1337 race.

A disabled plist is still a bootstrap race participant

Disabled=true in a file does nothing if the process was already bootstrapped. A reboot preserves the plist directory and the launchctl state database — not your intent — and the two drift. The plist wins startup; the state database only later decides whether to keep it running.

What Was Wrong

com.sanctum.server-mlx.plist — the Python mlx_lm.server fallback — still lived in ~/Library/LaunchAgents/, which launchd bootstraps at login. It had KeepAlive=true and no RunAtLoad; KeepAlive=true alone starts the process on bootstrap. Python bound *:1337 and answered requests long before the Rust binary finished its ed25519-signed manifest (10 s), a SHA-check of 16 GB of weights (10 s), and the Metal load (60 s). On every reboot, Python would have won.

What Also Went Sideways

Something had rebuilt the Rust binary at 23:54 the night before — cargo-clean or a parallel feature branch, cause unknown. The rebuild was (adhoc,linker-signed): no hardened runtime, no Developer ID. council-integrity caught it on codesign --verify --strict, though the adhoc binary started well enough that Rust reached :1338 and the fallback-lock never fired. Right to worry; the worry just wasn’t needed.

What Changed

The plist now ships RunAtLoad=false and Disabled=true: a reboot bootstraps it, but launchd refuses to start it until something explicitly enables it. activate_fallback() in both council-guardian.sh and council-integrity-check.sh now runs launchctl enable gui/<uid>/com.sanctum.server-mlx before its bootstrap + kickstart.

<key>KeepAlive</key>
<true/>
<key>RunAtLoad</key>
<false/>
<key>KeepAlive</key>
<true/>
<key>Disabled</key>
<true/>

KeepAlive=true stays, so once Python is kicked on it persists across crashes. Two changes rode along: council-guardian became the second mTLS client, and council-drift-check stopped alerting on repo-dirty — usually just an uncommitted parallel session, while the SHA compare still catches real drift.

Rule

A plist in ~/Library/LaunchAgents/ is a startup-race participant whether you meant it to be or not. The fallback-only pattern needs an explicit opt-out — RunAtLoad=false, Disabled=true, launchctl disable — or a plist launchd doesn’t scan. Intent isn’t configuration; launchd reads it.

Corollary: an alert that fires every hour isn’t an alert, it’s wallpaper. Repo-dirty fired at the wrong severity on an active branch and drowned the drift events beside it — which is how 2026-04-19 went sixteen hours unread.

Post-Entry State

Mini post-reboot: Rust sanctum-mlx on :1337 plain + :1338 mTLS, Dev-ID signed,
                  notarized (submission 00000000-0000-0000-0000-000000000003, Accepted)
mTLS clients: 2/6 (canary, guardian → https://127.0.0.1:1338/v1/models, cert CN=guardian
              at ~/.sanctum/certs/clients/guardian.{crt,key}; delete to revert)
Python fallback: Disabled=true — dormant across reboots until guardian/integrity promotes it
Monitors: 5 channels green. Commits: sanctum-rs 6d27666, sanctum-docs this entry

Day Close

Six principles finished becoming code today: the integrity probe, the auto-remediation wiring, the mTLS migration, the failover exercise, the fallback-race fix, and the alert-fatigue split. Not one is new; each had been doctrine for at least a week. What changed is that the doctrine shipped. If the A+ roadmap had an honest way to close itself, this was it — exercise the failover under real failure, sleep, wake to a reboot that proves the plist discipline was never quite tight enough, and ship the fix before the next sitting.