Overview
What this challenge is about.
Rank cost drivers in PySpark telemetry, prototype three optimizations, and model savings at scale for a verifiable certificate.
The scenario
Pulse runs a venture-backed observability platform for early-stage developer-tool companies, and its single largest cloud line item is the nightly batch job that turns raw customer events into the dashboards its own customers depend on. With fundraising tight, the finance team has made this one job a board-level cost target.
The Brief
What you'll do, and what you'll demonstrate.
Cut a 6-terabyte-per-day Spark job's monthly bill by at least half without breaching its 6-hour completion deadline, and prove the savings well enough to fund the work.
Earning criteria — what you'll demonstrate
- Diagnose Spark cost drivers from performance telemetry rather than intuition, distinguishing skew, shuffle, and misprovisioning
- Prototype and measure targeted Spark optimizations on a representative subset instead of guessing at full scale
- Build a transparent subset-to-full extrapolation model whose assumptions a finance reader can audit
- Write a recommendation that sequences changes by risk and payoff so a non-engineer can fund and approve it
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Aligned coursework coming soon.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
- Spark
Apply spark to solve real industry problems and demonstrate production-level capability.
- Cost Optimization
Apply cost optimization to solve real industry problems and demonstrate production-level capability.
- Etl Pipelines
Apply etl pipelines to solve real industry problems and demonstrate production-level capability.
- Benchmarking
Apply benchmarking to solve real industry problems and demonstrate production-level capability.
- Pyspark
Apply pyspark to solve real industry problems and demonstrate production-level capability.
- Documentation
Apply documentation to solve real industry problems and demonstrate production-level capability.
Careers
Career paths this challenge builds toward
Completing this challenge demonstrates skills that transfer directly to these roles:
Data Platform / Backend Engineer
Owning the reliability and cost of batch data pipelines is core platform work. This challenge rehearses exactly that: reading telemetry, tuning Spark jobs, and proving an SLA holds after changes ship.
This challenge sharpens
- spark
- etl-pipelines
- pyspark
Cloud Cost / FinOps Engineer
FinOps engineers turn cloud spend into engineering action. Here you quantify drivers, model savings against live pricing, and write the funding case, the exact loop a FinOps role runs across an organization's workloads.
This challenge sharpens
- cost-optimization
- benchmarking
- documentation
Senior Data Engineer
Senior data engineers are trusted to make defensible performance trade-offs at scale. This challenge builds that muscle: prototyping on a representative subset, extrapolating honestly, and sequencing rollout by risk.
This challenge sharpens
- benchmarking
- etl-pipelines
- cost-optimization