Scale Feature Pipelines for a Hyperscaler Search-Ranking Team
Overview
What this challenge is about.
Fix three bottlenecks in a PySpark ranking pipeline using an 80 GB sample, prototype fixes, and estimate production impact. Earn a verifiable certificate.
The scenario
The platform's ranking team (around 60 engineers, infra spend in the low eight figures per year) cannot ship daily ranker experiments because each one waits on the 14-hour pipeline; they want to get to 6-7 hours without sinking the cluster bill.
The Brief
What you'll do, and what you'll demonstrate.
Halve a 14-hour Spark feature pipeline's wall-clock time at unchanged compute spend.
Earning criteria — what you'll demonstrate
- Read and interpret a Spark UI to find real bottlenecks
- Apply skew-handling, partition-tuning, and UDF-elimination patterns
- Extrapolate sample-scale measurements to production reliably
- Write a pre-RFC that engineering peers can review
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Aligned coursework coming soon.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
- Spark
Apply spark to solve real industry problems and demonstrate production-level capability.
- Distributed Systems
Apply distributed systems to solve real industry problems and demonstrate production-level capability.
- Performance Profiling
Apply performance profiling to solve real industry problems and demonstrate production-level capability.
- Data Pipelines
Apply data pipelines to solve real industry problems and demonstrate production-level capability.
- Python
Write clean, efficient Python for data processing, automation, and backend services.
- Cost Aware Engineering
Apply cost aware engineering to solve real industry problems and demonstrate production-level capability.
Careers
Career paths this challenge builds toward
Completing this challenge demonstrates skills that transfer directly to these roles:
Data Engineer
Profiling and tuning a production-grade Spark pipeline with a written pre-RFC is the textbook senior-data-engineer task at any hyperscaler.
This challenge sharpens
- spark
- data-pipelines
- performance-profiling
Machine Learning Engineer
Owning feature-pipeline performance is increasingly part of the MLE remit because slow features starve model iteration.
This challenge sharpens
- data-pipelines
- performance-profiling
- python
MLOps Engineer
Cost-aware infrastructure work on shared compute platforms is core MLOps territory; this challenge practices the discipline.
This challenge sharpens
- spark
- cost-aware-engineering
- distributed-systems