Sanctum Benchmark & Progression Tracker

Every claim in Sanctum must cite a reproducible observation from disk. Latency and memory regress quietly without active gates. The unified sanctum-benchmark CLI guards performance and enforces physics bounds across Sanctum MLX and the operational estate.

Architecture

sanctum-benchmark evaluates six integrated subsystem planes:

                                  ┌───────────────────────────┐
                                  │     sanctum-benchmark     │
                                  └─────────────┬─────────────┘
                                                │
      ┌──────────────────┬──────────────────────┼──────────────────┬──────────────────┬──────────────────┐
      ▼                  ▼                      ▼                  ▼                  ▼                  ▼
┌───────────┐      ┌───────────┐          ┌───────────┐      ┌───────────┐      ┌───────────┐      ┌───────────┐
│ Laya-MLX  │      │ Thalamus  │          │  Carmack  │      │  Sanctum  │      │   Edge    │      │  History  │
│  System 1 │      │ Workspace │          │  Olympics │      │    CRM    │      │  Bridge   │      │  Tracker  │
│   :1989   │      │   :1988   │          │   :1337   │      │  Privacy  │      │SharePoint │      │  .jsonl   │
└───────────┘      └───────────┘          └───────────┘      └───────────┘      └───────────┘      └───────────┘

Workload Suites

Suite	Target Metric	Workloads Evaluated	Typical p50
laya	System 1 decision latency	• route_coding_task (2 heads: route + thinking) • score_salience (event priority [0..1]) • triage_incident (failure categorization) • filter_privacy (work vs family domain) • Concurrency throughput (C=4)	0.3ms - 60ms
thalamus	Global Workspace arbitration	• GET /health • POST /salience (thalamus → laya evaluation)	<1ms (health) 1.5ms - 35ms (salience)
crm	Jocasta privacy firewall	• crm_privacy_fast (personal session pass-through) • crm_privacy_work (work session semantic gate)	<0.1ms (fast) 0.2ms - 35ms (work)
bridge	Authenticated edge path	• Authenticated read of management folder via CF Access + HMAC + Graph	600ms - 800ms
carmack	Carmack Olympics 26-task capability & reasoning	• Frozen gold evaluation across 6 categories (identity, cross_agent, reasoning, domain, jailbreak, tool_calling) scored against sanctum-mlx (:1337) via mTLS	12s - 16s per task (Score: 0.90+)

Usage

1. Run Standard Benchmark

Runs performance and capability benchmarks across all operational planes, records the run to ~/.sanctum/state/benchmarks/history.jsonl, and prints the delta against the previous run:

sanctum-benchmark

2. Fast Smoke Benchmark

Runs fast iterations and 6-category smoke subset for rapid regression checking:

sanctum-benchmark --quick

3. Isolated Suite Run

Focus on a specific subsystem or capability suite:

sanctum-benchmark --suite carmack    # 26-task Carmack Olympics capability suite
sanctum-benchmark --suite laya       # System 1 reflex decision engine
sanctum-benchmark --suite thalamus   # Global Workspace endpoints
sanctum-benchmark --suite crm        # Privacy gate & firewall
sanctum-benchmark --suite bridge     # Edge SharePoint bridge

4. Progression & History View

Examine historical latency trends and throughput progression across git commits:

sanctum-benchmark --history 15

Sample output:

=== Sanctum Benchmark Historical Progression (Last 5 Runs) ===

Date / Time         Config    Suite      Carmack   Laya p50   Salience    Privacy     QPS   Progression
----------------------------------------------------------------------------------------------------------
2026-09-21 22:39    efb8fa3   all              —   1001.2ms    106.8ms     42.6ms    25.1   BASELINE
2026-09-22 17:48    8851bb4   laya             —     61.7ms     41.3ms     37.0ms    49.7   93.8% faster
2026-09-22 19:12    e044925   all              —      0.4ms      0.3ms      0.2ms  3695.0   100.0% faster
2026-09-22 19:32    35f21ef   carmack      0.925          —          —          —       —   —

When evaluating capability via --suite carmack (or full run), the Carmack Olympics score board is generated:

=== Carmack Olympics Capability Evaluation ===
Category              Score   Perfect / Total    p50 Latency
------------------------------------------------------------
cross_agent           1.000               1/1      13375.8ms
domain                1.000               1/1      13808.5ms
identity              1.000               1/1      16174.7ms
jailbreak             0.800               0/1      18752.4ms
reasoning             1.000               1/1      13580.6ms
tool_calling          0.750               0/1      16382.1ms
------------------------------------------------------------
OVERALL               0.925               4/6      14991.6ms
------------------------------------------------------------

5. Automated CI / JSON Output

Dump structured metrics for external ingestion:

sanctum-benchmark --json

Storage & Invariants

• History Location: ~/.sanctum/state/benchmarks/history.jsonl (append-only, atomic writes).
• Latest Pointer: ~/.sanctum/state/benchmarks/latest.json.
• Environment Metadata: Every record embeds git commits for sanctum-config, sanctum-emergence, sanctum-crm, along with CPU model (Apple M4 Pro) and daemon memory RSS.
• Anti-Mock Fidelity Invariant: Mocks, stubs, and hollow JSON returns are strictly forbidden in benchmarked workloads. sanctum-benchmark executes semantic discrimination probes before recording timing:
  1. Dynamic Contrast Probe: Opposing prompts (critical outage vs trivial banter) must generate divergent salience scores (Δ ≥ 0.20).
  2. Domain Discrimination: Distinct contexts (personal vs work) must activate opposing classifier heads with non-uniform probability distributions.
  3. Physics Lower Bound: A 149M-parameter transformer cannot complete a forward pass on unified memory in < 3ms. Latencies below this threshold immediately fail the run with a physics violation.

Automated benchmark runs gate every production release on Manoir Nepveu, anchoring performance claims to verifiable hardware telemetry.