Skip to content
Le Sanctum

2026-09-14: The bridge that swallowed its own exec

Pencil sketch of an ancient mechanical stone bridge with clockwork gears and glowing cyan circuit conduits spanning an abyss, where an intricate parchment script scroll feeds through a brass gate, dark background, subtle cyan and amber accent lines.

A silent exit is the deadliest failure mode. When a critical daemon exits zero, launchd assumes the work finished cleanly. Neo spotted the flashing amber badge during morning inspection. The Firewalla bridge on 127.0.0.1:1984 had gone dark. Force Flow could no longer enforce curfews or screen-time rules. Every self-healing guard had fired in turn and collapsed. Tommy stood watch as we traced the cascade to a single escaped quote that caused a shell script to swallow its own execution.

The silent exit zero

Section titled “The silent exit zero”

The Firewalla bridge on the Mac Mini is wrapped by ~/.sanctum/scripts/firewalla-bridge.sh. It runs under the system LaunchDaemon haus.sanctum.firewalla, detailed in services and the port summary. In a provenance commit intended to stamp running binary hashes before execution, an inline Python snippet was embedded:

Terminal window
# The offending line in scripts/firewalla-bridge.sh:
python3 - "$BRIDGE" "~/.sanctum/state/firewalla-bridge-running.json" <<'"'"'STAMP'"'"' 2>/dev/null || true
...
STAMP


exec /usr/local/bin/node "$BRIDGE"

The bug was subtle. Because the heredoc token was written as <<'"'"'STAMP'"'"', bash interpreted the delimiter tag as containing literal embedded single quotes: 'STAMP'.

When the script reached the unquoted STAMP line, bash refused to terminate the here-document. It kept reading. It fed every subsequent line directly into Python’s standard input. That swallowed both STAMP and exec /usr/local/bin/node "$BRIDGE".

Python parsed the stream and crashed on a syntax error. Because stderr was discarded with 2>/dev/null || true, the crash was invisible. Bash reached the end of the script and exited zero. The daemon was dead, but launchd believed it had succeeded.

The cascade of failed healers

Section titled “The cascade of failed healers”

Once the bridge process died, every automated guard in the haus attempted recovery and stumbled, echoing the resilience principles in The zero-plaintext citadel:

  1. sanctum-firewalla-watch.sh Unbound Variable Bug: In its alerting branches, variable declarations ended with trailing backslashes (MSG="..." \ curl ...). Under set -uo pipefail, bash attempted command substitution before setting the variable, aborting with MSG: unbound variable.
  2. firewalla-watch Auth Confusion: The watcher only scanned for log lines containing "Token refreshed". When the daemon died, token refreshes stopped. After 120 minutes, the watcher classified the state as AUTH-EXPIRED—a state where it intentionally refuses to restart the bridge. It lacked any basic check for whether the bridge process was simply dead.
  3. force-flow-bridge-sentinel.py Domain Mismatch: When the sentinel detected BRIDGE-DOWN, it delegated healing to haus.sanctum.firewalla-watch rather than restarting system/haus.sanctum.firewalla directly, believing user LaunchAgents lacked permission.
  4. R2D2 Recipe Domain Mismatch: R2D2’s heal-force-flow-bridge-down.sh, documented in agents, attempted launchctl kickstart -k "gui/$UID_NUM/haus.sanctum.firewalla". Because the bridge is a system daemon, launchd rejected the command. Furthermore, passing -k to a system domain target without root privileges resulted in Operation not permitted.

The multi-tier repair

Section titled “The multi-tier repair”

We resolved all layers across the stack:

  • Heredoc Delimiter Fix: Corrected <<'"'"'STAMP'"'"' to clean <<'STAMP' in scripts/firewalla-bridge.sh. The provenance stamp now executes cleanly and passes control directly to exec /usr/local/bin/node "$BRIDGE".
  • sanctum-firewalla-watch.sh Hardening: Added explicit BRIDGE-DOWN process detection via pgrep. When the process is dead, the watcher immediately kickstarts the system daemon. The trailing backslash syntax errors on MSG were eliminated.
  • R2D2 Self-Heal Repair: Updated heal-force-flow-bridge-down.sh to target system/haus.sanctum.firewalla and use passwordless kickstarts.
  • Sentinel Defense-in-Depth: force-flow-bridge-sentinel.py now directly issues sudo -n /bin/launchctl kickstart -k system/haus.sanctum.firewalla in parallel with delegating to the watch agent.
  • Permanent Sudoers Envelope: Added bert ALL=(root) NOPASSWD: /bin/launchctl kickstart [-k] system/haus.sanctum.firewalla and /usr/bin/xcodebuild -license accept to sudoers-sanctum-kickstart (installed to /etc/sudoers.d/sanctum-kickstart), closing the privilege gap permanently.
  • Xcode License & Git Resilience: Background macOS updates bump Xcode / Command Line Tools and break /usr/bin/git with rc=69. We installed standalone Homebrew git (/opt/homebrew/bin/git), updated screen-time-sentinel.sh to prefer it and auto-accept licenses on rc=69, and embedded auto-acceptance in both sanctum-sunday-upgrade.sh and apple-post-boot.sh.

Following the repair, system/haus.sanctum.firewalla resumed listening on 127.0.0.1:1984, authenticated with Firewalla cloud, and Force Flow reported enforcement_ready: true. All sentinels and commit validations pass cleanly.