Data & ML · Intermediate
Streaming Event Analytics Pipeline
Ingest a firehose of clickstream events, aggregate them in real time, and serve sub-second metrics to a live dashboard.
You will build a real-time analytics system that ingests high-volume clickstream events through Kafka, aggregates them continuously in ClickHouse or DuckDB, and exposes the results through a FastAPI service and a Grafana dashboard. This project mirrors the production stack at companies like Cloudflare, Notion, and Shopify that need sub-second query latency over billions of rows. Building it teaches you the full path from raw event to actionable metric, including partitioning strategy, materialized views, and the operational trade-offs between batch and stream processing.
What you build
- Produce synthetic clickstream events (page views, clicks, purchases) to a Kafka or Redpanda topic using a Python producer
- Consume and transform events with a Python consumer that writes micro-batches to ClickHouse or DuckDB
- Define aggregation logic as dbt models (hourly active users, funnel conversion rates, top pages) backed by materialized views
- Expose pre-aggregated metrics through a FastAPI endpoint with query params for time range and event type
- Display live-updating charts in a Grafana dashboard polling the FastAPI layer every few seconds
- Handle late-arriving events and out-of-order timestamps with a configurable grace-period window
- Emit a dead-letter topic for malformed or schema-invalid events so no data is silently dropped
What it teaches
- Stream partitioning and consumer group semantics in Kafka
- ClickHouse MergeTree engine design including primary key selection and partition pruning
- Incremental dbt models and materialized views for continuous aggregation
- Back-pressure handling and micro-batch sizing trade-offs in real-time pipelines
- FastAPI dependency injection and async database clients for low-latency metric serving
- Observability of streaming pipelines through consumer lag monitoring and dead-letter queues
How it works
- 1
Producer
- Python script
- Synthetic clicks
- JSON events
↓ Configurable events/sec
- 2
Kafka Topic
- raw-events
- Partitioned by user_id
↓ Redpanda or Apache Kafka
- 3
Consumer
- Batch insert
- Schema validation
- Dead-letter queue
- 4
ClickHouse + dbt
- MergeTree table
- Materialized views
- Hourly aggregates
↓ Sub-second queries
- 5
FastAPI + Grafana
- /metrics endpoint
- Live dashboard
Sign in to open the build guide
Free account. Get the step-by-step build and every resource link.
Take it further
- Add a schema registry (Confluent Schema Registry or Redpanda's built-in) so producers and consumers share an Avro or Protobuf contract and incompatible changes fail fast.
- Replace the polling Grafana panel with a Server-Sent Events stream from FastAPI so the dashboard updates in true push mode without repeated polling.
- Deploy the full stack to a cloud VM using Docker Compose with Caddy as a TLS reverse proxy, and add a GitHub Actions workflow that runs dbt test on every PR.
