Data & ML · Advanced
Real-Time Fraud Detection Pipeline
Stream transactions through Kafka and Flink, compute online features in a feature store, score them with a model in under a second, and watch precision and recall drift on a live dashboard.
You build an end-to-end streaming fraud detection system that ingests raw payment transactions from Apache Kafka, computes real-time aggregation features with Apache Flink, stores and retrieves those features via Feast and Redis, and scores each transaction with an XGBoost model in under one second. A live monitoring dashboard tracks precision, recall, and feature drift as synthetic fraud patterns shift over time. This project is worth building because it covers the full MLOps loop from data ingestion to low-latency inference to production monitoring, skills that are in high demand but rarely taught together.
What you build
- Kafka producer that emits a realistic synthetic transaction stream with tunable fraud rate and seasonal patterns
- Flink streaming job that computes rolling-window aggregates per user (spend velocity, transaction count, merchant diversity) in real time
- Feast feature store backed by Redis for sub-millisecond online feature retrieval at scoring time
- XGBoost binary classifier trained offline on historical batches and served as a lightweight Python scoring service
- Decision engine that emits APPROVE, REVIEW, or DECLINE verdicts with confidence scores back to a Kafka output topic
- Grafana dashboard wired to a Prometheus metrics sink showing throughput, latency percentiles, precision, and recall in real time
- Automated model retraining trigger that fires when precision or recall drifts beyond a configurable threshold
What it teaches
- Stateful stream processing with Apache Flink including event-time windows, watermarks, and late-data handling
- Online feature store design with Feast and Redis for low-latency feature retrieval separate from batch training
- Gradient-boosted tree modeling with XGBoost including threshold calibration for imbalanced fraud datasets
- Production ML monitoring using Prometheus and Grafana to track precision, recall, and feature distribution drift
- End-to-end MLOps pipeline design connecting ingestion, feature engineering, inference, and retraining in one system
- Kafka topic design for streaming ML workloads including partitioning strategy and consumer group offsets
How it works
- 1
Ingest
- Kafka topic
- raw-transactions
↓ consume
- 2
Feature Eng
- Flink job
- rolling windows
- aggregates
↓ write
- 3
Feature Store
- Feast
- Redis
↓ retrieve
- 4
Score
- XGBoost
- scoring service
↓ verdict
- 5
Output
- Kafka topic
- verdicts
↓ observe
- 6
Monitor
- Prometheus
- Grafana
Sign in to open the build guide
Free account. Get the step-by-step build and every resource link.
Take it further
- Add a graph-based feature layer using a real-time GNN to detect fraud rings by modeling transaction networks across shared devices or merchant IDs.
- Implement a champion-challenger A/B routing layer that splits 10% of traffic to a retrained model and compares precision-recall curves before promoting it to production.
- Replace the static XGBoost scorer with an online learning loop using River so the model updates its weights incrementally with each labeled verdict.
