Overview
What this challenge is about.
Migrate a 200TB data lake from Parquet to Iceberg. Prototype on 80 tables, run validation queries, measure performance. You get a verifiable certificate.
The scenario
The media platform (around USD 240M ARR) has lost 11 engineer-weeks in 6 months to schema-evolution problems on raw Parquet — Iceberg's headlining feature is the business case.
The Brief
What you'll do, and what you'll demonstrate.
Migrate a 200TB Parquet data lake to Iceberg with no read regression and a runbook the data-platform team can execute table-by-table.
Earning criteria — what you'll demonstrate
- Choose the right Iceberg catalog for a multi-engine read pattern
- Design per-table migration strategies for varying scale
- Validate that schema evolution + time travel work end-to-end
- Produce a runbook the data platform team can execute table-by-table
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Big Data and Data-Intensive Systems
Master · Databases
Strong alignment
This challenge maps to Big Data and Data-Intensive Systems at the Master level. It sharpens the same practical skills your coursework expects — but in a real industry context with actual constraints and deliverables.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
- Iceberg
Apply iceberg to solve real industry problems and demonstrate production-level capability.
- Parquet
Apply parquet to solve real industry problems and demonstrate production-level capability.
- Data Lake
Apply data lake to solve real industry problems and demonstrate production-level capability.
- Spark
Apply spark to solve real industry problems and demonstrate production-level capability.
- Etl Pipelines
Apply etl pipelines to solve real industry problems and demonstrate production-level capability.
- Documentation
Apply documentation to solve real industry problems and demonstrate production-level capability.
Careers
Career paths this challenge builds toward
Completing this challenge demonstrates skills that transfer directly to these roles: