Replace Nightly Batch Loads with Near-Real-Time Reporting at a Pump Manufacturer
Overview
What this challenge is about.
Replace Nightly Batch Loads with Near-Real-Time Reporting at a Pump Manufacturer. Advanced challenge in design. Designing real products under real constraint...
The Brief
What you'll do, and what you'll demonstrate.
Daily decisions are made on data that is up to a full day stale because nine source systems are consolidated only by a slow overnight batch that finishes after the morning operations meeting starts.
This is not a design exercise. It is the work a product designer does between a brief and a shipped interface. That distinction matters to every hiring manager who has seen candidates redesign Spotify's homepage and none who have worked under real product constraints.
When you finish, you will have something most graduates do not: a real-world deliverable, verified by Ewance, that you can show to a hiring manager and say "I did this. Here is the proof."
Earning criteria — what you'll demonstrate
- Match each source system to the most suitable change-capture technique based on its technology and constraints
- Design a Kafka topic layout and schema-registry strategy that scales across sources and survives schema changes
- Build an end-to-end change-data-capture flow from a relational source into Snowflake and measure its latency
- Set freshness and completeness targets that are defensible against documented current-state performance
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Aligned coursework coming soon.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
- Change Data Capture
Apply change data capture to solve real industry problems and demonstrate production-level capability.
- Kafka
Apply kafka to solve real industry problems and demonstrate production-level capability.
- Debezium
Apply debezium to solve real industry problems and demonstrate production-level capability.
- Snowflake
Apply snowflake to solve real industry problems and demonstrate production-level capability.
- Data Pipelines
Apply data pipelines to solve real industry problems and demonstrate production-level capability.
- Enterprise Integration
Apply enterprise integration to solve real industry problems and demonstrate production-level capability.
Careers
Career paths this challenge builds toward
Completing this challenge demonstrates skills that transfer directly to these roles:
Data Platform Architect
This challenge mirrors the architect's core job: surveying a messy estate of source systems and designing a coherent, scalable streaming platform that delivers trustworthy data on a deadline, balancing capture techniques, schema governance, and warehouse loading.
This challenge sharpens
- enterprise-integration
- data-pipelines
- snowflake
Streaming Data Engineer
Building and measuring an end-to-end change-capture flow from a relational source into a warehouse is exactly the daily work of a streaming engineer, who must make events flow reliably and prove latency targets are met with real measurements.
This challenge sharpens
- change-data-capture
- kafka
- debezium
Analytics Engineering Lead
Translating raw streamed changes into dependable, near-real-time reporting tables and setting defensible freshness targets bridges directly to leading an analytics engineering function that owns the warehouse and its service levels.
This challenge sharpens
- snowflake
- data-pipelines
- enterprise-integration