Audit a Hiring-Screen Classifier for Fairness Across Cohorts
Overview
What this challenge is about.
You receive the classifier as a black-box API and a synthetic-but-realistic dataset of 8,000 CVs with imputed demographic proxies (gender, age band, regional cluster) and labeled outcomes. Compute demographic parity, equal opportunity, and calibration-within-group across cohorts. Identify the largest disparity, run a counterfactual test (perturb only the proxy field), and produce a buyer-facing 6-page audit summary plus a 2-page internal remediation plan.
The Brief
What you'll do, and what you'll demonstrate.
Audit a hiring-screen classifier for cohort fairness with three standard metrics and produce both buyer-facing and internal remediation outputs.
Earning criteria — what you'll demonstrate
- Compute standard fairness metrics across cohorts
- Run counterfactual perturbation tests on a black-box classifier
- Translate audit findings into buyer-facing and internal documents
- Document audit methodology defensibly under regulatory scrutiny
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
Careers
Roles this prepares you for.
Real titles. Real skill bridges. Pick the one closest to your trajectory.
AI Safety Researcher
Running a fairness audit with standard metrics and counterfactual tests is the AI safety researcher's textbook job at any regulated AI vendor.
This challenge sharpens
- fairness-evaluation
- audit-methodology
- counterfactual-testing
Data Scientist
Per-cohort metric work with honest CIs and a buyer-facing summary is senior data-science craft at HR-tech and finance-AI vendors.
This challenge sharpens
- fairness-evaluation
- disparate-impact
- audit-methodology
AI Product Manager
Owning the buyer-facing fairness story plus the internal remediation roadmap is increasingly part of the AI PM's job in regulated AI products.
This challenge sharpens
- audit-methodology
- risk-assessment
- fairness-evaluation