4 — Operations

The daily read

Open the board. The footer states when the prober last completed a round — if that timestamp is stale, the prober itself is down and everything below is the last known state, said so. The vantage strip on each card shows the split; a silent relay reads UNKNOWN and the page names the gap.

The secrets inventory

Secret	Where	What it gates
OPS_TOKEN	the status worker	the admin + /ops/drafts (unset = both closed, 404)
STATUS_RELAY_TOKEN	BOTH workers, same value	/api/vantage-report (unset = the relay never lands, its vantage reads unknown)
MAIL_PROVIDER_KEY	the status worker	the HTTPS mail posture with the mail: block (unset = the honest console no-op; subscribe answers 503)
STATUS_PROBE_TOKEN	dormant	the exercised auth-probe labeling, while that leg is retired

The instance config never carries secret VALUES — secretHeaders NAME them; the loader refuses inline values.

The watchdogs

• The heartbeat (.github/workflows/heartbeat.yml, every 30 minutes): the off-platform probe of your own page — board reachable
  • prober fresh within ten probe intervals, and (TODO 81) the TLS-certificate expiry of every unique HTTPS host in the registry via openssl (the engine cannot see peer certificates from Workers; docs/manual/09). A cert inside CERT_WARN_DAYS (default 14; CERT_WATCH=off disables the watch) opens the same canonical tracking issue; an unmeasurable host is a named note, never an assumed expiry. It opens (or re-asserts on, at most every 6 hours) ONE canonical tracking issue while unhealthy and closes it on recovery.
• The deploy pipeline: gates on every main merge; the deploy steps run once the repo secrets are declared, and skip honestly without them.
• The audit register (TODO 74, the console's Audit card): the operator acts land append-only KV rows (audit:), bounded at 500 with the oldest aging out — who (the session's identity posture or the machine Bearer's kind), the act, one honest line of detail. It answers "who published this, who deleted that, which token failed a login" without touching platform logs. Rides in the backup with a row-shape check; details carry ids and codes, never payloads.
• The platform logs: wrangler tail (or the dashboard's Workers Logs) is the engine's own voice — the probe round's unset-secret warnings, the restore's cross-instance notice. What the KV rows deliberately do not keep (per-leg probe chatter beyond the rings) is a platform-log question, not a store question (the read-scaling path: docs/manual/09).

Troubleshooting playbooks

The page says the prober has never run / is stale — and the worker's log shows KV put() limit exceeded for the day. (RESOLVED on this account 2026-09-08: the upgrade to Workers Paid. The playbook stays for whitelabel instances.) The account is on the Workers FREE plan: 1,000 KV writes/day per namespace. This prober writes ~17 rows/minute (every service x every vantage + the metas + the heartbeat) ≈ 24,000/day — the free quota dies ~70 minutes after midnight UTC, and EVERY write path fails for the rest of the day (the cron runs; the writes throw; the page honestly goes stale). The real 2026-09-07/08 case looked like "crons not executing" — the tail showed no errors only because the exceptions landed in ctx.waitUntil; the KV tail told the truth. The fix is the owner's billing act: upgrade the account to Workers Paid (1M writes/day; a per-minute multi-vantage prober is a paid-tier workload by design). Interim honesty: the page shows its stale state, the heartbeat opens its tracking issue at the next check — both correct.

The page says the prober has never run, with NO quota error in the log. Now suspect the schedule binding: registered via the API but never executing (compare a sibling worker's fresh schedule; recreate the worker — KV state survives). Rare; the quota case above is the common one.

A relay vantage reads unknown. Its reports are not landing: wrong STATUS_RELAY_TOKEN on either side (the surface 401s/404s), the relay worker down, or its cron not executing (the same playbook as above). npx wrangler tail ribose-status-relay shows the round and its report outcome.

A service reads degraded but the site looks fine. Read the card's honest line: it names the failing vantage and the reason — a content assertion failing (the page changed its copy: update the marker in the YAML) or latency over budget. npm run probe:live reproduces it from your machine.

Inspecting state. npx wrangler kv key get --namespace-id <id> "svc:<serviceId>" reads any row; the full layout is in the API reference. The backup document (next chapter) is the read-everything view.

The attachment check (before any deletion or redeploy theory)

scripts/attach-custom-domain.sh --check prints which service (and account) the public hostname currently attaches to. Run it BEFORE acting on any theory about which worker is "the real one" — the 2026-09-12 incident deleted production workers on an unverified shadow theory and took the domain down. The check is read-only and takes one command; there is never a reason to skip it.