Run a Pre-Deployment Fairness + Drift Audit on a Hiring Model
Overview
What this challenge is about.
Audit a hiring model for fairness and drift using PSI and KL divergence. Propose mitigations and a monitoring plan. Get a verifiable certificate.
The scenario
The staffing agency (around 4,500 employees, around 1.1M placements per year) is preparing for a regulator-mandated AI Bill of Lading-style disclosure and needs sign-off from general counsel before rollout.
The Brief
What you'll do, and what you'll demonstrate.
Run a defensible pre-deployment fairness and drift audit with a six-month monitoring plan a non-technical counsel can sign off on.
Earning criteria — what you'll demonstrate
- Compute and interpret group-fairness metrics on a real classifier
- Apply drift-detection methods (PSI, KL) on tabular features
- Design a model-monitoring plan that maps thresholds to escalations
- Communicate audit findings to a legal/executive audience
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Machine Learning in Practice
Master · Machine Learning
Strong alignment
This challenge maps to Machine Learning in Practice at the Master level. It sharpens the same practical skills your coursework expects — but in a real industry context with actual constraints and deliverables.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
- Fairness Metrics
Apply fairness metrics to solve real industry problems and demonstrate production-level capability.
- Drift Detection
Apply drift detection to solve real industry problems and demonstrate production-level capability.
- Bias Mitigation
Apply bias mitigation to solve real industry problems and demonstrate production-level capability.
- Model Monitoring
Apply model monitoring to solve real industry problems and demonstrate production-level capability.
- Model Evaluation
Apply model evaluation to solve real industry problems and demonstrate production-level capability.
- Python
Write clean, efficient Python for data processing, automation, and backend services.
Careers
Career paths this challenge builds toward
Completing this challenge demonstrates skills that transfer directly to these roles:
AI Safety Researcher
Pre-deployment audits paired with monitoring plans are the entry-level deliverable for AI safety researchers at consultancies and in-house responsible-AI teams.
This challenge sharpens
- fairness-metrics
- drift-detection
- bias-mitigation
MLOps Engineer
Designing the drift-monitoring plan that ops teams will run for the next 6 months is core MLOps work.
This challenge sharpens
- drift-detection
- model-monitoring
- python
Applied AI Scientist
Communicating model risk to executives in their language is exactly what applied AI scientists are evaluated on in interviews.
This challenge sharpens
- model-evaluation
- fairness-metrics
- bias-mitigation