The situation
The telco (around 38M subscribers, 24 contact centres globally) is one of the consultancy's top-3 accounts and the relationship is at risk if the post-mortem doesn't land.
Your task
Run a blameless post-mortem on a failed ML deployment and recommend three corrective actions that will land with both engineering and the business sponsor.
The brief
You start with
You receive a packet: original training data sample, post-launch production logs, three Slack-style threads from the on-call rotation, and a summary of the telco's complaints.
What you do
- Reconstruct the deployment timeline.
- Run a blameless root-cause analysis across data (label shift? leakage?), model (calibration drift? distribution drift?), and process (testing gap? hand-off gap?).
- Produce a 5-page post-mortem with three concrete corrective actions, each with an owner, due date proposal, and expected impact.
- Present in a 25-minute readout to engineering + business stakeholders.
What you will practice
- Run a blameless post-mortem across data, model, and process axes
- Reconstruct an ML deployment timeline from heterogeneous artifacts
- Recommend corrective actions that survive both engineering and business scrutiny
- Present ML failures to mixed audiences without losing either side
A suggested way in
- Use the '5 Whys' on each axis (data, model, process) — stop only when the root is a system, not a person
- Always pair each corrective action with an owner and a measurable expected impact
- Calibrate language: engineering wants logs, business wants dollars per minute of handle time
Skills you will practice
Root cause analysis, Stakeholder framing, Model monitoring, ML pipelines, Deployment, Case study analysis
Where it leads
AI Solutions Architect, MLOps Engineer, AI Product Manager
How it is assessed
- Root-cause depth35%
Causes are traced to systemic gaps (not single mistakes) across data, model, and process.
- Corrective actions25%
Three actions are concrete, owned, and have credible expected impact.
- Timeline reconstruction20%
Replay convincingly shows when and how the failure pattern emerged.
- Readout delivery20%
Deck and live readout land with engineering and business in the same room.
Resources
- PythonTool
Replay production logs and quantify drift after the fact.
- pandasTool
Slice and pivot the production logs around the failure window.
- MatplotlibTool
Plot the failure-pattern emergence over the six weeks.
- NotionTool
Author the 5-page post-mortem.
- Google SlidesTool
Build the mixed-audience readout deck.
- FigmaTool
Polish the timeline diagram for the readout.
Finish it. Earn this.
A verified certificate with your name. Anyone can check it.
Time to practice.
Solve this challengeMore like this.
- Applied AI ScientistFine-Tune a Vision-Language Model for Image Captioning
- Software EngineerBuild a Multi-Level Feedback Queue Scheduler in xv6
- Software EngineerReverse-Engineer a Stripped Binary to Restore Linking Symbols
- Software EngineerGreedy Delivery-Slot Assignment for a Munich Grocery Startup
-
Fine-Tune a Vision-Language Model for Image Captioning Applied AI Scientist Take BLIP-2 or LLaVA-1.6 as the base. Fine-tune (LoRA is fine) on a 4,000-image accessibility-curated dataset where each image has a useful caption wr vision language modelslora fine tuningpytorch Open challenge → -
Build a Multi-Level Feedback Queue Scheduler in xv6 Software Engineer Fork the xv6-riscv repository and implement MLFQ in proc.c with 3 priority queues, per-queue time slices (10ms / 20ms / 40ms), priority demotion on sl operating systemsschedulingc programming Open challenge → -
Reverse-Engineer a Stripped Binary to Restore Linking Symbols Software Engineer Receive the stripped shared object (Linux ELF, x86-64, ~280KB), a crash dump with 12 frames of indecipherable addresses, and 4 years of partial git hi reverse engineeringlinking and loadingmachine code Open challenge → -
Greedy Delivery-Slot Assignment for a Munich Grocery Startup Software Engineer Using the provided order history sample, rider roster, and manual dispatch baseline, design and implement a greedy slot-assignment algorithm in Python greedy algorithmsalgorithm analysispython Open challenge →