How Uptimepage is built
Open-source uptime monitoring and public status pages, one Rust binary. This is the live map of how a request and a check move through it, from the edge to the databases. Pick a flow to trace its path.
Pick a flow on the right to light up the path through the system and read what is passed at each hop, then press Play to watch the signal travel it one hop at a time. Click any node to filter the flow list to the ones that touch it.
Actors / ingress
Edge + routing
HTTP handlers
Core services
Probe engine
Stores
External
How it fits together
Uptimepage is one Rust binary, and it keeps two databases. The split is not about which one is faster. A monitor is a row you edit, so it lives in Postgres with its constraints and transactions. A check result is written once and never touched again, so it goes to ClickHouse. Most of the rest of this map follows from that single line.
The probes run from outside your infrastructure. A monitor that sits next to the thing it watches will miss the outage that takes down both, and that is the outage your customers notice. Regional agents are stateless: they pull their config, run their checks, and post the results back. An agent holds no database. Its region and its identity come from its token and never from the payload it sends, so an agent cannot claim to be somewhere it is not.
The scheduler is a min-heap rather than a cron. Each target pops at its interval plus a jitter derived from its own id, which stops ten thousand monitors on a 60 second interval from all firing in the same second. The channel feeding it is bounded at eight. That number is small on purpose. If a Postgres refresh gets slow, I would rather the refresh wait than let a backlog build up behind the dispatcher.
A failing check is not an incident. Detection applies a region quorum before anything opens, so you decide whether one region failing counts or whether a majority has to agree. Only the race winner opens the incident and pages, which is what stops two workers waking the same person twice. Who is on call is worked out at the moment of paging and never stored, so a rota change takes effect straight away instead of at the next incident.
Here is what that design gives up. Detection is a poller on a 30 second tick, not an event listener. It costs a little latency, and it buys a system that cannot quietly stop noticing.
One detail I am glad I got right early: when a notification fails, the error is stored with every URL cut back to scheme and host first. Slack puts its secret in the path and Telegram puts its bot token in /bot<token>/, so a naive error log writes working credentials into your own database.
Deleting an account has to hold in both stores. Removing the Postgres rows first and the ClickHouse rows second would orphan data if the worker died in between, invisible to every query and resident on disk forever. So erasure runs through a queue, and the queue row only clears once a count proves zero rows across the raw table and both rollups.
Five flows worth reading
The map animates every path. These five carry the ideas above. Open one to see what is passed at each hop.
Scheduled HTTP checkControl plane probes a monitor in its own region
- refresh() Full re-list of enabled targets for this region, diffed into added / updated (by updated_at) / removed.
- RegistryDiff Sent over a bounded mpsc(8) so a slow Postgres query can never stall dispatch.
- due target Min-heap pop at interval ± hash(uuid) jitter. Paused monitors stay in the heap and are tombstoned by seq.
- gates passed in-flight set → semaphore try_acquire (fail-fast) → circuit breaker → per-tenant (org, host, port) bulkhead.
- timed_connect No connection pool: connecting fresh is what makes DNS, TCP and TLS separately timeable.
- probe Resolve → SSRF filter → happy eyeballs (250 ms stagger) → TLS → request over h1/h2 by ALPN.
- CheckResult Status + duration + dns/connect/tls/ttfb ms. Unexpected 429/503 classify as Degraded, not Down.
- write_batch Flush on size or timeout; retries re-send the identical block so the CH dedup window collapses duplicates.
- materialized view Minute rollup, 30 day TTL. The source for sparklines and buckets under 30 days.
- materialized view Hour rollup, 13 month TTL. Day strips and anything older than 30 days.
Regional agent probeA stateless agent pulls config, probes, and posts results back
- GET /api/agent/targets Bearer sm_agent_ token plus If-None-Match. A 401/403 is terminal: the agent clears its cache and stops probing.
- targets for region Region and agent id come from the token, never from the payload.
- open credentials Sealed check credentials are decrypted only for the agent that owns the region.
- AgentTargetDto list 304 when unchanged. Variables are resolved here, and a target whose vars fail to resolve is dropped rather than probed with literal braces.
- probe in-region Same executors as the control plane; the agent runs its own scheduler, pool and batcher.
- POST /results Stable batch_id reused across internal retries so a lost ack is deduped rather than double-counted.
- ingest Batch cap 10k, 300 s future-skew guard; org_id is stamped from the region assignment, and rows for unassigned targets are dropped, not rejected.
Detection → incident → pageFailing checks become a confirmed incident and page on-call
- poll, 30 s tick Keyset-paginates enabled targets cross-tenant and reads per lookback tier. This is a poller, not an event listener.
- insert_open decide_multi applies the region quorum (Any / Majority / All / Count). insert_open returns Option<Uuid>, so only the race winner pages.
- IncidentSignal The single push edge in detection. try_send drops with a metric rather than stalling the writer.
- resolve ladder Policy steps → targets (user / schedule / channel). Who is on call is computed at page time and never stored.
- page rung Deduped per channel; a per-incident sharded mutex stops the sweep and an inbound signal double-paging.
- deliver Delivery errors have every URL reduced to scheme://host before persisting. Slack's secret is in the path, Telegram's token in /bot<token>/.
- notification row Retry base×2^n capped, vendor retry_after hints honoured, then dead-letter. Acknowledge silences further paging.
Public status page renderAnonymous visitor on a tenant subdomain, cache miss
- TLS + per-IP limit Wildcard cert for *.{domain}; the operator host keeps its own cert to cap blast radius.
- classify host {slug}.{domain} parses to Subdomain → TenantPublic.
- tenant fence Default-deny: only /, /status, /subscribe, /.well-known/security.txt and four prefixes survive. Login and operator API 404 here.
- public_status view Host resolves the page; a verified custom domain wins over the slug host for link building.
- try_get_with Single-flight per page. On compute failure a last-good layer serves stale rather than erroring.
- aggregator Six loads in one try_join! so the snapshot is atomic: maintenance, active + recent incidents, markers, paint windows, day presence.
- confirmed incidents Outage paint comes from confirmed incident windows only, so a blip that never confirmed leaves the day green.
- day presence The hour rollup answers "did we check at all", which is what separates NoData from Operational.
Account deletion → cross-store erasureThe 30-day claim has to hold in both databases
- DELETE /me Blocked with OWNS_SHARED_ORGS if the user solely owns an org with other members.
- soft delete Tombstones plus a hashed recovery row, all inside one locked transaction.
- confirmation mail Sent after commit, never inside the transaction.
- daily purge tick Runs under a job advisory lock so only one replica executes; owner rows are locked in id order to stay deadlock-free.
- enqueue Deleting in PG then CH would orphan rows if the worker died between them, invisible to queries and resident on disk forever.
- ALTER … DELETE Runs with mutations_sync=2 so it returns only once applied. Replay-safe by construction.
- settle The queue row clears only after count() proves zero rows across the raw table and both rollups.