The situation
The startup (around 50 people, working with mid-cap US insurers) competes on regulator-friendly modeling; an interpretable-by-design model can shave months from the carrier's internal review cycle even at a small accuracy cost.
Your task
Quantify the accuracy/interpretability trade-off between an Explainable Boosting Machine and a black-box LightGBM for claims-reserve estimation.
The brief
You start with
You receive an anonymized claims dataset (around 60,000 claims, target: log reserve), a feature schema (22 features), and an existing LightGBM baseline (held-out R^2 of 0.78).
What you do
- Train an Explainable Boosting Machine (EBM, a modern interpretable GAM) on the same features, tune via cross-validation, and produce per-feature shape plots.
- Compare EBM vs.
- LightGBM on held-out R^2, calibration by reserve decile, and actuarial sniff-test (you produce a per-feature shape-plot deck).
- Write a 2-page memo on the accuracy/interpretability trade-off and recommend a path forward.
What you will practice
- Train and tune Explainable Boosting Machines as interpretable-by-design models
- Compare interpretable vs. black-box models fairly on regression
- Evaluate calibration alongside accuracy
- Communicate model trade-offs to an actuarial audience
A suggested way in
- interpretml's EBM is the modern, scikit-learn-compatible GAM implementation
- Cross-validate both models to avoid bogus single-split conclusions
- Use decile calibration plots; insurers care more about tail calibration than overall R^2
Skills you will practice
Generalized additive models, Ebm, Interpretability, Model comparison, Calibration, Python
Where it leads
Data Scientist, AI Safety Researcher, Machine Learning Engineer
How it is assessed
- EBM modeling quality30%
EBM hits a defensible R^2 close to the LightGBM baseline.
- Calibration analysis25%
Per-decile calibration is reported with confidence intervals for both models.
- Shape-plot interpretability25%
Per-feature shape plots tell a coherent actuarial story.
- Memo + recommendation20%
Memo gives leadership a clear answer on which model to ship, with risks.
Resources
- PythonTool
Standard language for interpretml and modeling glue.
- interpretTool
Microsoft's interpretml library with EBM implementation and shape plots.
- LightGBMTool
Black-box baseline for the comparison.
- PandasTool
Process the claims data and per-decile calibration tables.
- MatplotlibTool
Render calibration curves and per-feature shape plots.
- NotionTool
Compose the 2-page accuracy-vs-interpretability memo.
Finish it. Earn this.
A verified certificate with your name. Anyone can check it.
Time to practice.
Solve this challengeMore like this.
- Data ScientistExplain a Credit-Risk Model with SHAP for a Fintech
- Senior AccountantIFRS 16 Lease Accounting for a Renewable Energy Firm
- Software EngineerODE Integrator Benchmark for a Pharma PK/PD Pipeline
- Data Protection OfficerData Protection Impact Assessment for a Fintech App
-
Explain a Credit-Risk Model with SHAP for a Fintech Data Scientist You receive a trained XGBoost credit-risk model (binary default prediction), the training feature schema (38 features), and a held-out 10,000-sample t shapinterpretabilityfairness analysis Open challenge → -
IFRS 16 Lease Accounting for a Renewable Energy Firm Senior Accountant You are a senior accountant at SunPower Renewables. The company is adopting IFRS 16 for the first time. You are given a sample lease for a wind turbin ifrs 16lease accountingright of use asset Open challenge → -
ODE Integrator Benchmark for a Pharma PK/PD Pipeline Software Engineer Use SciPy's solve_ivp + SUNDIALS (via scikit-sundae or diffeqpy) to benchmark: RK45 (baseline), LSODA, BDF (CVODE), Radau, and Rosenbrock. Run on thre numerical odesstiff integratorsscientific python Open challenge → -
Data Protection Impact Assessment for a Fintech App Data Protection Officer You are the data protection officer (DPO) for the fintech. Perform a DPIA for the new AI-driven loan feature, following ICO guidelines. Identify at le gdpr compliancedata protection impact assessmentrisk management Open challenge →