Design a Change-Data-Capture Pipeline for an E-Commerce Reseller
Overview
What this challenge is about.
Design a CDC pipeline for an e-commerce reseller using Debezium and Kafka, then measure freshness and correctness. Earn a verifiable certificate.
The scenario
The reseller (around 18M annual orders, family-owned, recently hired their first head of data) currently exports nightly — analysts cannot answer same-day operational questions.
The Brief
What you'll do, and what you'll demonstrate.
Ship a Debezium + Kafka CDC pipeline that mirrors MySQL to a warehouse with under-1-minute freshness on 8 priority tables, and a rollout plan for the remaining 212.
Earning criteria — what you'll demonstrate
- Design and operate a Debezium-based CDC pipeline end-to-end
- Handle schema changes, large transactions, and initial snapshotting
- Validate freshness + correctness with realistic replays
- Plan a 220-table rollout that doesn't melt the source database
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Big Data and Data-Intensive Systems
Master · Databases
Strong alignment
This challenge maps to Big Data and Data-Intensive Systems at the Master level. It sharpens the same practical skills your coursework expects — but in a real industry context with actual constraints and deliverables.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
- Change Data Capture
Apply change data capture to solve real industry problems and demonstrate production-level capability.
- Debezium
Apply debezium to solve real industry problems and demonstrate production-level capability.
- Kafka
Apply kafka to solve real industry problems and demonstrate production-level capability.
- Etl Pipelines
Apply etl pipelines to solve real industry problems and demonstrate production-level capability.
- Mysql
Apply mysql to solve real industry problems and demonstrate production-level capability.
- Documentation
Apply documentation to solve real industry problems and demonstrate production-level capability.
Careers
Career paths this challenge builds toward
Completing this challenge demonstrates skills that transfer directly to these roles: