Design a Real-Time Order Pipeline for a Fintech Payments Platform
Overview
What this challenge is about.
You receive a synthetic Kafka stream of around 500 transactions per second, a static merchant dimension table (about 80,000 rows), and a daily FX rate snapshot. Design an end-to-end streaming pipeline using Kafka plus a stream-processing layer (Flink, Spark Structured Streaming, or Kafka Streams), with the enriched output landing in a columnar store (BigQuery, Snowflake, or ClickHouse). Success is end-to-end latency under five minutes at p95 (95th-percentile, meaning 95 of every 100 transactions arrive within that bound), an at-least-once delivery guarantee with documented deduplication, and a monthly cost projection for ten times today's volume.
The Brief
What you'll do, and what you'll demonstrate.
Replace the nightly batch reporting flow with a streaming pipeline that delivers enriched transactions to analytics in under five minutes at p95.
Earning criteria — what you'll demonstrate
- Design a streaming pipeline with explicit latency and delivery guarantees
- Trade off stream-processing engines on cost, complexity, and team familiarity
- Model an enrichment join between a hot stream and a static dimension
- Estimate cloud cost at multiple traffic scales
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
Careers
Roles this prepares you for.
Real titles. Real skill bridges. Pick the one closest to your trajectory.
Data Engineer
Designing a streaming pipeline against real latency and cost budgets is exactly the first month of work for a junior data engineer at any payments or analytics company.
This challenge sharpens
- streaming-data
- kafka
- stream-processing
MLOps Engineer
The same streaming primitives feed online fraud and recommendation models; MLOps engineers own this layer end-to-end at fintechs.
This challenge sharpens
- streaming-data
- system-design
- stream-processing
AI Solutions Architect
Writing a costed, tradeoff-justified architecture memo for a head-of-data is the bread and butter of a solutions architect role.
This challenge sharpens
- system-design
- cost-modeling
- data-modeling