Case study · Data · Streaming

Real-time event pipeline

Reports that took a night to compute now update in under a minute. The pipeline absorbs traffic spikes without losing data or waking anyone up.

The short version

The problem

Reports were computed overnight and broke under traffic spikes.

Built a streaming pipeline that ingests, deduplicates and aggregates events continuously.

Reports update in under a minute and spikes no longer lose data.

14kevents/s

Sustained ingest rate

Production2025

Kafka consumer throughput, 30-day peak

38s

Event to report, 95% of events

Production2025

p95 end-to-end lag, 30 days

For engineers: constraints, architecture, trade-offs and what I'd do differently.

01 · Context and constraints

Placeholder. Replace with the real case study.