Relay — a self-healing data pipeline
A fault-tolerant ingestion pipeline that cut data loss to near-zero and made backfills a one-command affair.
Relay replaced a fragile chain of cron jobs that quietly dropped events under load. The brief was simple to say and hard to do: never lose an event, and make it obvious when something is wrong.
What I built
- An idempotent ingestion layer with at-least-once delivery and dead-letter recovery.
- A backfill mode that replays any time window from cold storage with a single command.
- First-class observability: every stage emits structured metrics, so incidents are diagnosed in minutes, not hours.
What changed
Data loss dropped from ~0.4% to effectively zero. On-call pages for the pipeline fell by 80% in the first quarter, and a backfill that used to take a careful afternoon now takes one command and a coffee.