Design a Distributed-Training Strategy for a Mid-Sized LLM
Overview
What this challenge is about.
You will write a 5-page design memo that picks a parallelism strategy for fine-tuning a 13B model on 32 H100 GPUs, with a tokens-per-second estimate, a memory-per-GPU calculation, and a cost-per-experiment in dollars. Validate the throughput estimate by running a 1B-parameter proxy at small scale (4 GPUs or even Colab Pro), then extrapolate. Include a risk register for the top three things that could derail the production run. Deliver the memo, the validation notebook, and a 1-page exec summary.
The Brief
What you'll do, and what you'll demonstrate.
Pick and defend a parallelism strategy that hits a throughput and cost target for a 13B-parameter fine-tune on 32 GPUs.
Earning criteria — what you'll demonstrate
- Pick a parallelism strategy (data/tensor/pipeline/hybrid) with quantitative justification
- Compute memory-per-GPU and tokens-per-second analytically
- Validate small-scale and extrapolate to production-scale
- Communicate distributed-training design choices to infrastructure leadership
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
Careers
Roles this prepares you for.
Real titles. Real skill bridges. Pick the one closest to your trajectory.
ML Researcher
Choosing and defending a distributed-training strategy for an actual planned run is the daily reality of ML researchers at any LLM-training shop.
This challenge sharpens
- distributed-training
- llm-training
- throughput-modeling
Machine Learning Engineer
Memory and throughput modeling are exactly the skills MLEs use to keep large training runs from blowing up.
This challenge sharpens
- distributed-training
- throughput-modeling
- pytorch
MLOps Engineer
Cost-aware design at the 32-GPU scale is core MLOps work at any AI-research shop with a finite compute budget.
This challenge sharpens
- distributed-training
- cost-estimation
- parallelism-strategies