Ewance
← All challenges AI Solutions Architect

Run a Post-Mortem on a Failed ML Deployment

Duration
2 weeks
Level
Expert
Type
Presentation
Team
1 to 5 people

The situation

The telco (around 38M subscribers, 24 contact centres globally) is one of the consultancy's top-3 accounts and the relationship is at risk if the post-mortem doesn't land.

Your task

Run a blameless post-mortem on a failed ML deployment and recommend three corrective actions that will land with both engineering and the business sponsor.

The brief

You start with

You receive a packet: original training data sample, post-launch production logs, three Slack-style threads from the on-call rotation, and a summary of the telco's complaints.

What you do

  1. Reconstruct the deployment timeline.
  2. Run a blameless root-cause analysis across data (label shift? leakage?), model (calibration drift? distribution drift?), and process (testing gap? hand-off gap?).
  3. Produce a 5-page post-mortem with three concrete corrective actions, each with an owner, due date proposal, and expected impact.
  4. Present in a 25-minute readout to engineering + business stakeholders.

What you will practice

  • Run a blameless post-mortem across data, model, and process axes
  • Reconstruct an ML deployment timeline from heterogeneous artifacts
  • Recommend corrective actions that survive both engineering and business scrutiny
  • Present ML failures to mixed audiences without losing either side

A suggested way in

  1. Use the '5 Whys' on each axis (data, model, process) — stop only when the root is a system, not a person
  2. Always pair each corrective action with an owner and a measurable expected impact
  3. Calibrate language: engineering wants logs, business wants dollars per minute of handle time

Skills you will practice

Root cause analysis, Stakeholder framing, Model monitoring, ML pipelines, Deployment, Case study analysis

Where it leads

AI Solutions Architect, MLOps Engineer, AI Product Manager

How it is assessed

  • Root-cause depth35%

    Causes are traced to systemic gaps (not single mistakes) across data, model, and process.

  • Corrective actions25%

    Three actions are concrete, owned, and have credible expected impact.

  • Timeline reconstruction20%

    Replay convincingly shows when and how the failure pattern emerged.

  • Readout delivery20%

    Deck and live readout land with engineering and business in the same room.

Resources

  • PythonTool

    Replay production logs and quantify drift after the fact.

  • pandasTool

    Slice and pivot the production logs around the failure window.

  • MatplotlibTool

    Plot the failure-pattern emergence over the six weeks.

  • NotionTool

    Author the 5-page post-mortem.

  • Google SlidesTool

    Build the mixed-audience readout deck.

  • FigmaTool

    Polish the timeline diagram for the readout.

Sample Ewance certificate of completion, verified by LearnCoin
Sample certificate
Your proof

Finish it. Earn this.

A verified certificate with your name. Anyone can check it.

Time to practice.

Solve this challenge
Related challenges

More like this.