Overview
What this challenge is about.
Audit a fine-tuned chemistry model for catastrophic forgetting, run a mitigation experiment, and report your findings. Earn a verifiable certificate.
The scenario
The pharma AI startup (around 50 people, working with mid-cap pharma clients out of Basel and Boston) needs the fine-tuned model to be both a strong medicinal-chemistry assistant AND a competent general assistant for scientists; loss of general capability is a top user-reported complaint.
The Brief
What you'll do, and what you'll demonstrate.
Audit a domain fine-tuned LLM for catastrophic forgetting, propose mitigations, and write the safety memo that informs the next fine-tune cycle.
Earning criteria — what you'll demonstrate
- Design a catastrophic-forgetting audit for a domain fine-tune
- Run multi-benchmark LLM evaluation with statistical rigor
- Reason about mitigations (replay, merging, LoRA isolation)
- Communicate safety findings to platform leadership
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Fine-Tuning Large Language Models
Master · Generative Ai
Strong alignment
This challenge maps to Fine-Tuning Large Language Models at the Master level. It sharpens the same practical skills your coursework expects — but in a real industry context with actual constraints and deliverables.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
- Catastrophic Forgetting
Apply catastrophic forgetting to solve real industry problems and demonstrate production-level capability.
- Llm Evaluation
Apply llm evaluation to solve real industry problems and demonstrate production-level capability.
- Fine Tuning
Apply fine tuning to solve real industry problems and demonstrate production-level capability.
- Model Merging
Apply model merging to solve real industry problems and demonstrate production-level capability.
- Huggingface
Apply huggingface to solve real industry problems and demonstrate production-level capability.
- Benchmarking
Apply benchmarking to solve real industry problems and demonstrate production-level capability.
Careers
Career paths this challenge builds toward
Completing this challenge demonstrates skills that transfer directly to these roles:
AI Safety Researcher
Designing and running a catastrophic-forgetting audit on a production fine-tune is exactly the day-one work of an AI safety researcher in any LLM-shipping organization.
This challenge sharpens
- catastrophic-forgetting
- llm-evaluation
- benchmarking
ML Researcher
Running mitigations like replay or model-merging and honestly reporting whether they close the gap is core ML-research work in industry labs.
This challenge sharpens
- model-merging
- fine-tuning
- llm-evaluation
Machine Learning Engineer
Building a reproducible LLM evaluation harness that another engineer can rerun is the MLE craft of shipping evaluation as code.
This challenge sharpens
- llm-evaluation
- huggingface
- benchmarking