Overview
What this challenge is about.
Halve a Daily Spark Bill Without Breaking the SLA. Advanced challenge in analysis. Analyzing real datasets and building models that drive decisions, earn a b...
The Brief
What you'll do, and what you'll demonstrate.
Cut a 6-terabyte-per-day Spark job's monthly bill by at least half without breaching its 6-hour completion deadline, and prove the savings well enough to fund the work.
This is not a data exercise. It is the work an analyst does when stakeholders need answers from messy data. That distinction matters to every hiring manager who has seen candidates describe statistical methods and none who have extracted insight from messy, real-world data.
When you finish, you will have something most graduates do not: a real-world deliverable, verified by Ewance, that you can show to a hiring manager and say "I did this. Here is the proof."
Earning criteria — what you'll demonstrate
- Diagnose Spark cost drivers from performance telemetry rather than intuition, distinguishing skew, shuffle, and misprovisioning
- Prototype and measure targeted Spark optimizations on a representative subset instead of guessing at full scale
- Build a transparent subset-to-full extrapolation model whose assumptions a finance reader can audit
- Write a recommendation that sequences changes by risk and payoff so a non-engineer can fund and approve it
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Aligned coursework coming soon.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
- Spark
Apply spark to solve real industry problems and demonstrate production-level capability.
- Cost Optimization
Apply cost optimization to solve real industry problems and demonstrate production-level capability.
- Etl Pipelines
Apply etl pipelines to solve real industry problems and demonstrate production-level capability.
- Benchmarking
Apply benchmarking to solve real industry problems and demonstrate production-level capability.
- Pyspark
Apply pyspark to solve real industry problems and demonstrate production-level capability.
- Documentation
Apply documentation to solve real industry problems and demonstrate production-level capability.
Careers
Career paths this challenge builds toward
Completing this challenge demonstrates skills that transfer directly to these roles:
Data Platform / Backend Engineer
Owning the reliability and cost of batch data pipelines is core platform work. This challenge rehearses exactly that: reading telemetry, tuning Spark jobs, and proving an SLA holds after changes ship.
This challenge sharpens
- spark
- etl-pipelines
- pyspark
Cloud Cost / FinOps Engineer
FinOps engineers turn cloud spend into engineering action. Here you quantify drivers, model savings against live pricing, and write the funding case, the exact loop a FinOps role runs across an organization's workloads.
This challenge sharpens
- cost-optimization
- benchmarking
- documentation
Senior Data Engineer
Senior data engineers are trusted to make defensible performance trade-offs at scale. This challenge builds that muscle: prototyping on a representative subset, extrapolating honestly, and sequencing rollout by risk.
This challenge sharpens
- benchmarking
- etl-pipelines
- cost-optimization