Replace Nightly Batch Loads with Near-Real-Time Reporting at a Pump Manufacturer
Overview
What this challenge is about.
Classify nine source systems, build a Kafka-to-Snowflake streaming pipeline, and measure end-to-end delay. Get a verifiable certificate.
The scenario
The company builds business-to-business industrial pumps and runs a lean business intelligence team that today depends entirely on a fragile overnight batch consolidating nine disconnected systems across enterprise software, two factory-floor manufacturing execution systems, sales, and supplier portals.
The Brief
What you'll do, and what you'll demonstrate.
Daily decisions are made on data that is up to a full day stale because nine source systems are consolidated only by a slow overnight batch that finishes after the morning operations meeting starts.
Earning criteria — what you'll demonstrate
- Match each source system to the most suitable change-capture technique based on its technology and constraints
- Design a Kafka topic layout and schema-registry strategy that scales across sources and survives schema changes
- Build an end-to-end change-data-capture flow from a relational source into Snowflake and measure its latency
- Set freshness and completeness targets that are defensible against documented current-state performance
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Aligned coursework coming soon.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
- Change Data Capture
Apply change data capture to solve real industry problems and demonstrate production-level capability.
- Kafka
Apply kafka to solve real industry problems and demonstrate production-level capability.
- Debezium
Apply debezium to solve real industry problems and demonstrate production-level capability.
- Snowflake
Apply snowflake to solve real industry problems and demonstrate production-level capability.
- Data Pipelines
Apply data pipelines to solve real industry problems and demonstrate production-level capability.
- Enterprise Integration
Apply enterprise integration to solve real industry problems and demonstrate production-level capability.
Careers
Career paths this challenge builds toward
Completing this challenge demonstrates skills that transfer directly to these roles:
Data Platform Architect
This challenge mirrors the architect's core job: surveying a messy estate of source systems and designing a coherent, scalable streaming platform that delivers trustworthy data on a deadline, balancing capture techniques, schema governance, and warehouse loading.
This challenge sharpens
- enterprise-integration
- data-pipelines
- snowflake
Streaming Data Engineer
Building and measuring an end-to-end change-capture flow from a relational source into a warehouse is exactly the daily work of a streaming engineer, who must make events flow reliably and prove latency targets are met with real measurements.
This challenge sharpens
- change-data-capture
- kafka
- debezium
Analytics Engineering Lead
Translating raw streamed changes into dependable, near-real-time reporting tables and setting defensible freshness targets bridges directly to leading an analytics engineering function that owns the warehouse and its service levels.
This challenge sharpens
- snowflake
- data-pipelines
- enterprise-integration