Design a Distributed Training Job for a 13B-Parameter Model
Overview
What this challenge is about.
Select parallelism for a 13B model, calculate throughput, and write a runbook. Earn a verifiable certificate.
The scenario
The Munich consultancy (around 60 engineers, paid AI training engagements for German enterprise) needs to deliver a reproducible distributed-training story or risks losing the engagement.
The Brief
What you'll do, and what you'll demonstrate.
Pick and justify a distributed-training strategy for a 13B-param model on 32 H100s, validated on a smaller proxy, and write the runbook.
Earning criteria — what you'll demonstrate
- Choose between FSDP, TP, PP, and hybrid parallelism for a real workload
- Calculate per-GPU memory and throughput budgets from first principles
- Design fault-tolerant checkpoint cadence and storage
- Communicate distributed-training design to a non-systems audience
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Aligned coursework coming soon.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
- Distributed Training
Apply distributed training to solve real industry problems and demonstrate production-level capability.
- Fsdp
Apply fsdp to solve real industry problems and demonstrate production-level capability.
- Pytorch
Apply pytorch to solve real industry problems and demonstrate production-level capability.
- Gpu Systems
Apply gpu systems to solve real industry problems and demonstrate production-level capability.
- Checkpointing
Apply checkpointing to solve real industry problems and demonstrate production-level capability.
- Throughput Modeling
Apply throughput modeling to solve real industry problems and demonstrate production-level capability.
Careers
Career paths this challenge builds toward
Completing this challenge demonstrates skills that transfer directly to these roles:
Machine Learning Engineer
Designing a real distributed-training plan with throughput modeling and a runbook is the work that staff-track MLEs lead at any team training 10B+ models.
This challenge sharpens
- distributed-training
- fsdp
- throughput-modeling
AI Solutions Architect
Choosing parallelism strategies and writing the runbook that a client research team executes against is core AI solutions architecture work at cloud and consulting orgs.
This challenge sharpens
- distributed-training
- checkpointing
- gpu-systems
MLOps Engineer
Checkpoint cadence, failure recovery, and infrastructure-shaped training plans are the daily concerns of MLOps engineers on training-platform teams.
This challenge sharpens
- checkpointing
- gpu-systems
- pytorch