Multi-vantage probing: the design

The 2026-09-01 incident (the public record) proved the status service's second false-green class: the identity sign-in's write path hung for users routed through Europe (the D1 primary sits away from the users; reads were already fine), while the prober's single vantage answered healthy the whole time. The board read 13/13 operational through a real outage. The incident's own follow-up list names the fix: probe from more than one network vantage.

This document is the design: the options map, the chosen shape, and what the result honestly proves. The implementation notes live in the README; this file is the reasoning.

The constraints

• No external paid uptime services. The estate self-hosts this service precisely so its honesty is ours to keep.
• The vantage points available are the estate's own Cloudflare workers. Nothing else exists under our control with geographic spread.
• Cloudflare colos are anycast-driven. A worker runs where its inbound trigger lands: a fetch-triggered worker runs at (near) the caller's colo, a cron-triggered worker runs at its scheduling colo. The scheduling colo is roughly stable per worker but is not choosable: Workers has no jurisdiction or region pinning (that knob exists for D1 databases and Durable Objects, not for worker placement). Any design must treat a vantage's geography as observed, never claimed.
• Honesty above all. A declared two-vantage posture with named, observed locations beats a fake "global" claim. A vantage that stops reporting is UNKNOWN, never green.

The options map

A. A second cron worker placed via a jurisdiction hint

Dead end as stated: Workers offers no jurisdiction or region hint to place (the estate verified this against the platform docs while scoping; D1's jurisdiction and read replication are database knobs, not worker placement). A second cron worker still yields a second uncontrolled vantage — its scheduling colo is whatever the scheduler picks — so the idea survives, minus the "hint": see the chosen shape.

B. A probe-relay on the sibling estate workers

The identity OP (id.ribose.com), the platform instances (demo/nmi/tl/platform), and the rag worker already run where their users are — a relay module in each would give user-geography-driven vantages, the best signal available. The cost: every sibling is a separate repo with its own tag-gated deploy pipeline, so v1 becomes a four-repo coordinated wave; and most siblings have no cron trigger today, so the "vantage" would still need a scheduled runner added per worker. This is the right expansion path, not the right v1. The chosen shape's report endpoint is deliberately sibling-shaped so these relays can join later without another status-repo change.

C. Cloudflare's own per-colo facilities

None exist for this: there is no API to run "a probe from colo X" on demand, and the analytics/observability surfaces report traffic, they do not probe. Nothing to build on.

D. The chosen shape: a same-repo relay cron worker + an authenticated report endpoint

A second worker deployed from this repository (ribose-status-relay, wrangler.relay.toml) runs the same probe registry on its own cron, from its own scheduling colo, and POSTs the round's results to the status worker's guarded POST /api/vantage-report. The status worker folds each report into per-vantage KV rows and computes the verdict across vantages.

Why this one:

• One repo, one wave. The relay shares the registry, the prober, and the test kit with the status worker; no sibling repo moves.
• Real extra vantages. One worker per relay means one scheduling colo each. Whether they differ is observed after deploy (each report carries the relay's self-observed colo) and shown on the page — never asserted in advance. If both land in the same colo the page says so, and the honest answer is "one effective vantage today", not a rebrand.
• The report endpoint is the validation boundary. The relay holds no KV binding: a shared KV namespace would let a misconfigured or compromised relay overwrite any row (the home rows, the drafts, the subscriber ledger). The endpoint constrains a relay to writing its own vantage's rows, validates every field, and rejects undeclared vantage ids. Sibling-worker relays (option B) later authenticate against the same endpoint with the same posture.
• Fail-closed by construction. The relay declares itself with a shared secret; the status worker tracks declared vantages from config, so a relay that goes silent turns its vantage UNKNOWN (never green) instead of vanishing.

The shape in detail

ribose-status (this worker, vantage "home")
  cron * * * * * → probe every service → svc:<id> rows (as today)
                 → compute the verdict across vantages → draft sweep
  POST /api/vantage-report (Bearer STATUS_RELAY_TOKEN)
                 → validate → fold into vantage:<relay>/svc:<id> rows
                 → vantage:<relay>/meta (last report, observed colo)
                 → recompute the verdict → draft sweep

ribose-status-relay (the v1 relay, vantage "relay-1")
  cron * * * * * → probe every service (the same registry and
                   assertions; the probe-token header only rides the
                   exercised auth legs — retired 2026-09-07 until the
                   deployed OP carries the route again)
                 → observe its own colo (best effort, see below)
                 → POST the round to the report endpoint
  No KV, no page, no state: a stateless prober with a report duty.

• Vantages are declared config (the instance YAML's vantages block): home (this worker) plus each expected relay. A report naming an undeclared vantage is refused; a declared relay that stops reporting goes UNKNOWN after the staleness window (5 minutes, the prober's own stale doctrine).
• Every probe row carries its vantage label. svc:<id> remains the home row (the existing layout, untouched); relay rows live at vantage:<vantageId>/svc:<id> with the same shape, so all rollup math is reused unchanged.
• The verdict is a worst-of, never an average. A service is DOWN when it is down from ANY reporting vantage (three consecutive failed probes from that vantage, the existing threshold); DEGRADED when any reporting vantage flaps. When the vantages disagree the card says so in plain words ("failing from vantage relay-1, healthy from vantage home: a region-scoped signal") — that sentence is the whole point of the exercise. A vantage with no fresh report contributes nothing to the verdict and shows UNKNOWN on the card.
• The board and /api/summary.json show the per-vantage split. Each card gains a vantage strip (one line per declared vantage: state, last probe, observed colo when known); the summary carries a per-service vantages array. No merged fake green anywhere.
• Uptime bars and percentages pool all vantage evidence and say so ("across every reporting vantage"): a failure from any vantage counts against the day. Percentages still truncate, never round up.
• The draft automation runs on the verdict. A region-scoped outage (one vantage down, another green) opens a draft whose evidence names the vantage. Recovery closes the draft only when every vantage that went down in it has passed again — a vantage that goes silent while down NEVER closes its own outage (silence is not recovery); the draft waits for a human.

The relay report's security

• POST /api/vantage-report takes Authorization: Bearer STATUS_RELAY_TOKEN, the same shared-secret posture as the estate's STATUS_PROBE_TOKEN precedent (a Worker secret on both workers, wrangler secret put per side, never in a repo) and the same hashed-compare check as OPS_TOKEN on /ops/drafts.
• The secret unset on the status worker closes the surface entirely (404, indistinguishable from an unknown route — the /ops/drafts posture). Unset on the relay, the relay logs the misconfiguration and skips the report; the home side then marks the vantage stale, which is the fail-closed path doing its job.
• A wrong token answers 401 and writes nothing. An undeclared vantage id answers 403 and writes nothing. A malformed payload answers 400 and writes nothing. Validation rejects unknown service ids, absurd timestamps (beyond clock-skew bounds), and non-finite latencies per entry.
• The report path's geography is irrelevant (anycast lands the status worker wherever the relay is; that is fine — only the probe's geography carries signal), so the canonical HTTPS URL is the report transport. No service binding needed.

The colo honesty rule

A vantage's location is observed, never claimed: each relay best-effort reads its own colo once per round (the colo= field of the estate zone's /cdn-cgi/trace, answered by the edge that served the subrequest) and reports it with the round. The page shows it as "observed colo FRA at report time". When the observation fails, the vantage shows no colo rather than a guessed one. Cloudflare may move a worker's scheduling colo at any time; the label follows the latest observation and the docs never promise a geography.

What this honestly proves (and what it does not)

• Proven: every service is exercised from two independent worker scheduling locations once a minute; a region-scoped outage on either vantage's path (the 2026-09-01 class) now reads down or degraded-with-the-split-note instead of green; a dead relay reads UNKNOWN, not green.
• Not proven: "global" coverage. Two vantages are two points. A region-scoped outage whose failing path misses both vantages still reads green, and the page's footing says the vantage count out loud. More relays (more vantage: ids in config, sibling-worker relays per option B) widen the net without changing the contract.
• Also not hidden: if both vantages land in the same colo, the effective vantage count is one and the page's observed-colo labels will show it. The doctrine forbids renaming that as two.

Later waves (named, not built)

• Sibling-worker relays (option B): the identity OP and the platform instances reporting from their user-driven colos against the same endpoint. The report contract is already sibling-shaped.
• Per-vantage latency sparklines on the card (v1 shows the home vantage's line, labeled, plus per-vantage last latencies in the strip).
• A third vantage in a deliberately different failure domain. The off-platform heartbeat (the GitHub-action probe this repo now runs against its own page) covers the page's own reachability; a third PROBE vantage would widen the estate coverage further.