On 2026-04-29 the agents on the VM stopped working. The signs were subtle — claude-code claiming the API key was invalid, the Firewalla bridge returning 401 every fourth request — and the silence had been compounding for twenty-four hours. The cause was a git merge from the day before that left literal <<<<<<< HEAD markers inside the SOPS-encrypted secrets file. Every agent had been failing to load secrets at startup. None of them had a way to say so.
That afternoon we built the Secrets Trifecta. Not a new architecture — a named one, with clean rules, a status indicator, and a small set of automated processes meant to make that failure mode impossible to repeat.
The trifecta has three layers. Calling them three stores is a useful approximation that turns out to be wrong.
1Password Sanctum vault — Source of Truth
The human’s daily driver. Touch ID, audit log, search, multi-device. When a provider rotates a key, the new value lands here first. Every other layer is derived from this one. The items live in triptyq.1password.com / Sanctum — moved there from the personal Private vault on 2026-05-16 so a 1Password Service Account could authenticate headlessly. Item titles are still Manoir - <Service>, so the catalog stays grep-able no matter which vault holds it.
VM SOPS — Agent-Authoritative Copy
~/.openclaw/secrets.enc.yaml on the airgapped VM. Agents read it once at startup via sops-start.sh. They cannot reach 1Password from inside the airgap, so SOPS is not optional for them — it is the runtime surface. Encrypted at rest with an age key whose master copy lives in the Mini’s keychain and the 1Password vault.
Mini Keychain — Hot Cache
~/Library/Keychains/login.keychain-db, account sanctum. Mini-side tools (the openrouter CLI, the domain-ops handler, the Force Flow daemon) read it for speed: a keychain lookup is around forty milliseconds; an op read is closer to one second. The Keychain is not a store — it is a derivative cache that auto-rebuilds from 1Password on demand.
The reframe matters. Before this pass, the Mini Keychain felt like a third independent store, and the mental load was real: which copy wins on conflict? Once you accept the keychain as a cache, the answer is routine. 1Password wins. The cache rebuilds. The architecture stops humming with anxiety.
┌─────────────────────┐ │ 1Password / Sanctum │ ← rotations happen here │ (Touch ID + audit) │ SOURCE OF TRUTH └──────────┬──────────┘ │ sync.py auto-mirrors ┌────────────┴────────────┐ ▼ ▼ ┌──────────────┐ ┌──────────────────┐ │ VM SOPS │ │ Mini Keychain │ │ (agents) │ │ (Mini-tool │ │ │ │ hot cache) │ └──────────────┘ └──────────────────┘ STORE CACHETwo LaunchAgents are written to keep this honest. Both plists live in tools/secret-rotator/ and load under an Aqua session — staged, but not currently bootstrapped on the host (launchctl list | grep secrets- comes back empty):
| Agent | Cadence | Purpose |
|---|---|---|
com.sanctum.secrets-sync-drift-check | Daily 09:30 | Runs sync.py apply --yes --notify. Reads 1Password for each cross-tier secret, compares against SOPS and Keychain, writes any drift, posts a Force Flow alert if changes were made or any write failed. The intent is auto-heal, not just auto-detect — but the keychain-write leg only succeeds with an unlocked login keychain, and under a headless session it does not. See Seventeen Days of Silent Failure for the run that lied about this. |
com.sanctum.secrets-doctor | Every 4h | Runs doctor.py --notify. Probes the operational dependencies — Tailscale reach to the Mini, the Aqua-context keychain bridge, VM SOPS readability, 1Password CLI integration — and is meant to alert BEFORE the next sync hits the same wall. Catches operational rot ahead of data drift. |
A SwiftBar plugin (tools/secret-rotator/swiftbar/sanctum-sync.5m.py) is written to surface live status in the menu bar: green when all nine manifest entries are in sync, yellow when a check found drift mid-flux, red when something errored, white when the daily check is more than thirty hours stale. One click reveals the last check timestamp, today’s writes, and four actions: run the check now, force a resync, open the audit log, open this handbook. SwiftBar itself is not installed on the host yet, so the glyph is potential, not present — but the plugin reads its counts straight from the audit log, so the day SwiftBar lands it lights up with no further wiring.
The goal is that the user does not need to remember the system exists. The code is most of the way there. The install isn’t yet.
Nine entries, declared in tools/secret-rotator/providers.yaml under sync_mirrors:
| Tier | Count | Examples |
|---|---|---|
| Cross-tier (all three layers) | 5 | anthropic, gemini, openrouter, xai, gateway |
| SOPS-only (no keychain mirror) | 3 | firewalla_bridge_token, ionos_api_prefix, ionos_api_secret |
| Keychain-only (computed) | 1 | ionos_api_key_combined — concatenated prefix.secret for tools that consume IONOS_API_KEY as one env var |
Each entry has a name, an op field naming the 1Password item, an optional sops key, an optional kc keychain service, and an optional combine format string for values computed from multiple items. Adding a new cross-tier secret is a YAML edit, not a code change.
Two credentials deliberately left this manifest. firewalla_bridge_token dropped its keychain leg on 2026-05-31; cloudflare_tunnel_token left sync_mirrors entirely on 2026-05-16. Both have an operational source of truth that is not 1Password — the bridge token reads from a mode-600 file (a system LaunchDaemon can’t reach the keychain at all), the tunnel token auto-rotates straight into the keychain. Pushing 1Password into those slots too just made two healers fight over one value. The manifest holds the secrets where 1Password genuinely wins, and declines the ones where it doesn’t.
Lost Mini Keychain
Severity: low. The cache rebuilds itself on the next sync.py apply. The wrapper at ~/.sanctum/scripts/sanctum-secret falls back to op read if a tool requests a value that is not yet warmed.
Lost SOPS file or age key
Severity: high but recoverable. SOPS backups exist as .bak-sync-* next to the live file (sync.py writes one before every change). The age key has two backup locations: Mini keychain entry age-openclaw-key, and 1Password item SanctumBridge — age private key (master).
Lost 1Password access
Severity: high. The runtime keeps working — agents have SOPS, Mini tools have Keychain — until rotations are needed. Recover via 1Password’s emergency kit. Do not wipe the vault: its values are still authoritative even when locked out.
Lost Mini hardware entirely
Severity: medium. Time Machine restores the keychain. New Mini installs sanctum-secret + age key; sync.py apply rebuilds the keychain from 1Password. The runbook lives in docs/secrets-handbook.md for the line-by-line steps.
Six scenarios are documented in full in the handbook. The worst — losing 1Password and the age key at once — is the only one that forces re-minting every provider credential, and it takes losing both the Mac’s secure-enclave biometric chain and the printed emergency kit simultaneously. The design stops short of impossible. It stops well short of casual.
Four rules that earn the trifecta its name:
apply still confirms; the daily plist does not.firewalla-bridge-token that lived only in the locked login Keychain and put the bridge in a fatal-restart loop (the full nine-hour audit). The three tiers are not interchangeable: 1Password is unreachable from any service runtime; SOPS needs the VM, the sops binary, and the age key; Keychain is gated on a user session that may not exist. Filesystem (~/.sanctum/secrets/<name>, mode 600) is the only tier with no ambient prerequisites. So: if a service is P0, its secret writes through to filesystem on rotation, or it isn’t allowed to be P0.The trifecta is small on purpose. Every secret that passes through it passes through this discipline. Every secret that does not is a secret we have decided is rare enough to handle by hand — and a future cleanup pass will probably absorb most of those into the manifest too.
The architecture and code are written, council-reviewed, and unit-tested. The runtime is honest but not yet hands-off: the two scheduled agents and the menu-bar indicator are staged on the host, not loaded, and the keychain-write leg of apply still fails under a headless session — the gap dissected in Seventeen Days of Silent Failure. Interactive works; unattended is the open work. Three improvements are documented but not yet built:
Install the automation — bootstrap both LaunchAgents and SwiftBar, after wiring an unlocked-keychain path so the daily apply finishes its write leg instead of logging six changes and six fails. The code is ready; the keychain session is the blocker.
Phase 6: extend the manifest to all Mini-only secrets — the deepgram keys, the SanctumBridge integrations, the Cloudflare account tokens. They’re in 1Password and the Keychain today, but synchronized by hand. Cilghal voted for this on 2026-04-29; the cost is mostly tedium, not invention.
Phase 7: phased migration toward eliminating the Keychain layer — Mini tools could read 1Password directly via op read. The wrapper at ~/.sanctum/scripts/sanctum-secret already implements this fallback path. The path is mechanical; the trigger is the moment the Keychain’s speed advantage stops outweighing its conceptual cost.
None of the three is urgent. The trifecta as built moves toward the bar set on 2026-04-29: the user does not have to remember it exists. The doctrine, the manifest, and the recovery story already clear it; the unattended daemon is the last mile. That is the only test that matters, and it is the one still being earned.