Design a Real-Time Order Pipeline for a Fintech Payments Platform
Overview
What this challenge is about.
Design a real-time Kafka streaming pipeline for fintech payments with under 5-minute latency and earn a verifiable certificate.
The scenario
The fintech (around 220 staff, around USD 4 billion in annual payment volume) currently runs everything on nightly Airflow batches and has lost two merchant accounts to a faster competitor in the last quarter.
The Brief
What you'll do, and what you'll demonstrate.
Replace the nightly batch reporting flow with a streaming pipeline that delivers enriched transactions to analytics in under five minutes at p95.
Earning criteria — what you'll demonstrate
- Design a streaming pipeline with explicit latency and delivery guarantees
- Trade off stream-processing engines on cost, complexity, and team familiarity
- Model an enrichment join between a hot stream and a static dimension
- Estimate cloud cost at multiple traffic scales
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Data Engineering and Big Data Systems
Master · Data Engineering
Strong alignment
This challenge maps to Data Engineering and Big Data Systems at the Master level. It sharpens the same practical skills your coursework expects — but in a real industry context with actual constraints and deliverables.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
- Streaming Data
Apply streaming data to solve real industry problems and demonstrate production-level capability.
- Kafka
Apply kafka to solve real industry problems and demonstrate production-level capability.
- Stream Processing
Apply stream processing to solve real industry problems and demonstrate production-level capability.
- Data Modeling
Apply data modeling to solve real industry problems and demonstrate production-level capability.
- Cost Modeling
Apply cost modeling to solve real industry problems and demonstrate production-level capability.
- System Design
Apply system design to solve real industry problems and demonstrate production-level capability.
Careers
Career paths this challenge builds toward
Completing this challenge demonstrates skills that transfer directly to these roles:
Data Engineer
Designing a streaming pipeline against real latency and cost budgets is exactly the first month of work for a junior data engineer at any payments or analytics company.
This challenge sharpens
- streaming-data
- kafka
- stream-processing
MLOps Engineer
The same streaming primitives feed online fraud and recommendation models; MLOps engineers own this layer end-to-end at fintechs.
This challenge sharpens
- streaming-data
- system-design
- stream-processing
AI Solutions Architect
Writing a costed, tradeoff-justified architecture memo for a head-of-data is the bread and butter of a solutions architect role.
This challenge sharpens
- system-design
- cost-modeling
- data-modeling