Design a Distributed-Training Strategy for a Mid-Sized LLM
Overview
What this challenge is about.
Write a design memo for fine-tuning a 13B LLM on 32 H100 GPUs and validate with a small-scale proxy run. Earn a verifiable certificate.
The scenario
The lab (around 25 researchers, Series B AI startup) burns through compute fast and has only one shot per quarter at a 32-GPU job; the infra lead refuses to approve the run without a defensible design.
The Brief
What you'll do, and what you'll demonstrate.
Pick and defend a parallelism strategy that hits a throughput and cost target for a 13B-parameter fine-tune on 32 GPUs.
Earning criteria — what you'll demonstrate
- Pick a parallelism strategy (data/tensor/pipeline/hybrid) with quantitative justification
- Compute memory-per-GPU and tokens-per-second analytically
- Validate small-scale and extrapolate to production-scale
- Communicate distributed-training design choices to infrastructure leadership
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Aligned coursework coming soon.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
- Distributed Training
Apply distributed training to solve real industry problems and demonstrate production-level capability.
- Parallelism Strategies
Apply parallelism strategies to solve real industry problems and demonstrate production-level capability.
- Llm Training
Apply llm training to solve real industry problems and demonstrate production-level capability.
- Throughput Modeling
Apply throughput modeling to solve real industry problems and demonstrate production-level capability.
- Pytorch
Apply pytorch to solve real industry problems and demonstrate production-level capability.
- Cost Estimation
Apply cost estimation to solve real industry problems and demonstrate production-level capability.
Careers
Career paths this challenge builds toward
Completing this challenge demonstrates skills that transfer directly to these roles:
ML Researcher
Choosing and defending a distributed-training strategy for an actual planned run is the daily reality of ML researchers at any LLM-training shop.
This challenge sharpens
- distributed-training
- llm-training
- throughput-modeling
Machine Learning Engineer
Memory and throughput modeling are exactly the skills MLEs use to keep large training runs from blowing up.
This challenge sharpens
- distributed-training
- throughput-modeling
- pytorch
MLOps Engineer
Cost-aware design at the 32-GPU scale is core MLOps work at any AI-research shop with a finite compute budget.
This challenge sharpens
- distributed-training
- cost-estimation
- parallelism-strategies