Benchmark Conformal Prediction for a Healthcare Risk-Score
Overview
What this challenge is about.
You receive a labeled dataset of about 25,000 patient encounters with the current risk-score's predictions and ground-truth 1-year outcomes. Implement and compare split conformal prediction, jackknife+, and Mondrian conformal (conditional on age band). Evaluate coverage and interval width on a held-out fold. Document where each method preserves clinically meaningful subgroup coverage. Deliver the benchmark notebook, results, and a 4-page methodology memo for the clinical lead.
The Brief
What you'll do, and what you'll demonstrate.
Pick the conformal-prediction variant that gives honest coverage with the tightest intervals across clinically meaningful subgroups.
Earning criteria — what you'll demonstrate
- Apply conformal prediction to a real classifier output
- Measure marginal and conditional coverage honestly
- Compare conformal variants on interval tightness + subgroup fairness
- Document methodology for regulatory-grade submissions
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
Careers
Roles this prepares you for.
Real titles. Real skill bridges. Pick the one closest to your trajectory.
Research Scientist
Applying conformal prediction with rigorous coverage analysis to a regulatory-bound product is the kind of work junior research scientists at clinical-ML startups own.
This challenge sharpens
- conformal-prediction
- uncertainty-quantification
- subgroup-analysis
ML Researcher
Benchmarking modern uncertainty methods against subgroup fairness is core ML-researcher craft at any safety-critical AI shop.
This challenge sharpens
- conformal-prediction
- calibration
- evaluation
AI Safety Researcher
Honest coverage analysis and dossier-grade documentation is exactly the AI safety researcher's contribution to medical-AI launches.
This challenge sharpens
- uncertainty-quantification
- subgroup-analysis
- calibration