MBP Reboot Runbook

The MacBook Pro just came back up. Somewhere in the last ninety seconds, four launch agents woke on their own, a Rust server claimed two ports, and two watchers started poking the Mac Mini across Tailscale to confirm it was still breathing. You did none of it. That is the whole point.

A technical pencil sketch of a sentinel workstation waking from reboot, twin listener arrays pulsing with teal authentication glow, paper schematics pinned overhead charting the recovery choreography, clean hand-drawn lines, no text.

The MBP runs the Sanctum shadow. Its sanctum-mlx binds :8902 (plain + bearer) and :8903 (mTLS), and it stands as the high-availability fallback for when the Mini’s :1337/:1338 goes dark. It also carries two off-box watchers that probe the Mini over Tailscale — because if the Mini panics, nothing Mini-local is left alive to tell anyone. This is Qui-Gon’s kind of paranoia, wired into the council’s resilience doctrine: the watcher must never share a fate with the thing it watches.

None of it needs a human after a reboot. Every agent is RunAtLoad=true or StartInterval-scheduled, and the plists on disk are the only source of truth. So the checklist below is not a recovery procedure — it is a five-minute question: did the machine come back exactly as it left?

Agent	Binds / schedule	Role
`com.sanctum.shadow-mlx`	`:8902` (plain+bearer) and `:8903` (mTLS)	The MBP’s Rust sanctum-mlx serving as HA fallback for the Mini
`com.sanctum.council-canary-offbox`	10 min `StartInterval`	Probes the Mini’s council over Tailscale with a chat request
`com.sanctum.council-drift-offbox`	1 h `StartInterval`	Runs `deploy-sanctum-mlx.sh verify` against the Mini
`com.sanctum.agent-markdown-sync`	`RunAtLoad=true`	Syncs agent docs between repos

The shadow plist carries LimitLoadToSessionType = Aqua, so it waits for the user session before binding Metal. Expect 60–70 seconds from login to the first successful probe. The 27 billion parameters still need to load, and Metal does not hurry.

Five-minute sanity check

Five checks, in order. Each one confirms a layer above the last — agents, then listeners, then the mTLS handshake, then the two watchers that live off the box.

Agents are loaded.
Terminal window
```
launchctl list | grep com.sanctum
```
The four agents above should appear. shadow-mlx should have a numeric PID in column 1; the others run on intervals and show - when idle.
Shadow listeners are up.
Terminal window
```
lsof -nP -i :8902,8903 | grep sanctum-m
```
Two lines, both with the same sanctum-mlx PID. One on *:8902, one on *:8903.

mTLS path works.

curl -sf --cacert ~/.sanctum/certs/ca.crt \
         --cert ~/.sanctum/certs/clients/sanctum-server.crt \
         --key  ~/.sanctum/certs/clients/sanctum-server.key \
         https://127.0.0.1:8903/v1/models | head -c 100

Returns a JSON {"data":[{...}]}. If it hangs, the model is still loading; wait 30 seconds and retry.

Off-box canary has started probing.
Terminal window
```
tail -3 ~/.openclaw/logs/council-canary-offbox.log
```
Expect event:"canary_ok" with transport:"mtls" within 10 minutes of login. The first probe on a cold start often logs a single canary_fail if the Mini is simultaneously cold — this is benign and auto-recovers on the next tick.
Off-box drift is clean.
Terminal window
```
tail -3 ~/.openclaw/logs/council-drift-offbox.log
```
Expect event:"drift_ok" on the most recent hourly run. A single drift_detected immediately after a reboot is normal (the Mini’s model may still be loading); look for recovery on the next tick.

If something didn’t come back

Rare, but it happens. A session dies dirty, a plist gets disabled by hand, Tailscale sulks. Here is how we put each piece back.

Shadow didn’t bind

launchctl bootout  gui/$(id -u)/com.sanctum.shadow-mlx 2>/dev/null
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.sanctum.shadow-mlx.plist
launchctl kickstart -k gui/$(id -u)/com.sanctum.shadow-mlx

If bootstrap returns Input/output error, the label is still bootstrapped from a previous session. bootout always runs first; never retry bootstrap in a loop. (This is an actual, documented launchd gotcha — see the 2026-04-20 Living Force entry.)

Off-box watchers disappeared

for a in council-canary-offbox council-drift-offbox; do
  launchctl bootout  gui/$(id -u)/com.sanctum.$a 2>/dev/null
  launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.sanctum.$a.plist
done

They’ll fire on their own cadence afterward (every 10 min and every 1 h respectively).

mTLS probe hangs or fails handshake

Either the Mini is unreachable over Tailscale (the common case — check tailscale status) or the cert files moved. The cert bundle lives at ~/.sanctum/certs/ and the auto-detect in every probe script looks for exactly these paths:

~/.sanctum/certs/ca.crt
~/.sanctum/certs/clients/<probe-name>.crt
~/.sanctum/certs/clients/<probe-name>.key

If the files are gone, the probes all fall back to bearer over :1337 with no code change — the token at ~/.sanctum/secrets/council-mlx.token is the failsafe. If the files are present and TLS still fails, the certs expired (check with openssl x509 -in <file> -noout -dates) — see the mTLS migration for the cert-rotation path.

What’s in /tmp

Nothing important. The mTLS test scripts write to /tmp/sanctum-mtls-test/, and they clean up after themselves. A reboot clears /tmp anyway. If you spot anything sanctum-related in there after a reboot, file a bug against the script that put it there — the directory is supposed to be empty.

What survives the reboot

Keychain unlocked state. Does not survive — a reboot locks the login keychain. If you need to codesign on the Mini over SSH after the reboot, you’ll need to unlock it again (security unlock-keychain).
notarytool credentials. Stored in the keychain; survive the reboot but require the keychain to be unlocked when used.
Launch agent state. Re-bootstraps from the plists on disk. Any manual launchctl disable will persist.
sanctum-mlx’s Metal model cache. Does not survive — cold-start load is ~60 s after every reboot. Same for the MBP shadow and the Mini’s primary.
Bearer tokens, certs, the .p8 for notarization. All live in ~/.sanctum/ or ~/.appstoreconnect/, persist.
Prometheus metrics counters. Do not survive; they restart at zero per process lifetime. Current values are at http://127.0.0.1:1337/metrics while the process lives.

One thing the reboot can’t fix

For a stretch, the Mini’s :1337 was served by a non-canonical sanctum-mlx invocation without TLS args — a quiet hole in the trust topology that no reboot would ever close, because rebooting only re-ran the wrong thing faithfully. That was resolved 2026-05-03. The signed com.sanctum.mlx LaunchAgent is now the only thing binding :1337, mTLS-only, with the signed manifest gate in place. The MBP shadow on :8903 is symmetric — signed, notarized, mTLS — and wired into sanctum-server’s council-secure fallback chain.

That is the reassuring part. A reboot can only bring back what the plists describe, so the plists had to become the whole truth. Get that right once and the machine can wake up alone at 3 a.m., rebuild itself from disk, and never phone you. The best runbook is the one you never have to open.