Migrate a Legacy Warehouse to a Lakehouse for an Edtech AI Platform
Overview
What this challenge is about.
Migrate a 50 GB Postgres dump to a lakehouse on S3, rebuild transformations in Spark, and benchmark cohort retention. Earn your verifiable certificate.
The scenario
The edtech (around 60 staff, around 400,000 monthly active students across the UK and Ireland) is preparing a Series B and needs to demonstrate that data infrastructure can scale 10x without proportional cost growth.
The Brief
What you'll do, and what you'll demonstrate.
Prove that migrating one critical mart from Postgres to a lakehouse cuts runtime to under 30 minutes and lowers cost per query at the next growth tier.
Earning criteria — what you'll demonstrate
- Model an open-table-format dataset (Delta or Iceberg) with appropriate partitioning
- Translate dbt-style transformations to a distributed engine
- Benchmark and explain analytical query performance fairly
- Plan a low-risk warehouse-to-lakehouse migration
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Data Engineering and Big Data Systems
Master · Data Engineering
Strong alignment
This challenge maps to Data Engineering and Big Data Systems at the Master level. It sharpens the same practical skills your coursework expects — but in a real industry context with actual constraints and deliverables.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
- Lakehouse Architecture
Apply lakehouse architecture to solve real industry problems and demonstrate production-level capability.
- Delta Lake
Apply delta lake to solve real industry problems and demonstrate production-level capability.
- Spark
Apply spark to solve real industry problems and demonstrate production-level capability.
- Dbt
Apply dbt to solve real industry problems and demonstrate production-level capability.
- Data Modeling
Apply data modeling to solve real industry problems and demonstrate production-level capability.
- Performance Benchmarking
Apply performance benchmarking to solve real industry problems and demonstrate production-level capability.
Careers
Career paths this challenge builds toward
Completing this challenge demonstrates skills that transfer directly to these roles:
Data Engineer
Lakehouse migrations are the dominant data-engineering project of the moment; shipping one PoC end-to-end is a strong portfolio piece.
This challenge sharpens
- lakehouse-architecture
- delta-lake
- spark
AI Solutions Architect
Writing a cost-and-risk migration plan a CTO can sign off on is core solutions-architect output.
This challenge sharpens
- data-modeling
- performance-benchmarking
- lakehouse-architecture
MLOps Engineer
Unblocking the morning retraining job is a quintessentially MLOps win; the lakehouse skills carry directly into MLOps platform work.
This challenge sharpens
- spark
- dbt
- performance-benchmarking