Spec Trust-and-Safety Eval Harness for an LLM-Powered Customer-Support Bot
Overview
What this challenge is about.
Spec a trust-and-safety eval harness for an LLM support bot and build a reference Python implementation to earn your verifiable certificate.
The Brief
What you'll do, and what you'll demonstrate.
Spec and reference-implement a nightly trust-and-safety harness for an LLM customer-support bot covering jailbreaks, PII, and toxicity.
Earning criteria — what you'll demonstrate
- Design a multi-axis safety evaluation harness for LLM products
- Curate jailbreak and PII test sets at useful scale
- Integrate a toxicity classifier into automated gating
- Document a harness so engineering can pick it up next sprint
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
Careers
Roles this prepares you for.
Real titles. Real skill bridges. Pick the one closest to your trajectory.
Career paths this builds toward
Canonical rolesAI Safety Researcher
Designing a nightly safety eval harness for an LLM product is the AI safety researcher's textbook job at any enterprise-AI vendor.
This challenge sharpens
- llm-evaluation
- red-teaming
- pii-detection
MLOps Engineer
Gating deploys on regression-detection thresholds is the MLOps engineer's craft applied to safety axes.
This challenge sharpens
- regression-detection
- harness-design
- python
Prompt Engineer
Curating jailbreak test sets and reasoning about LLM failure axes is core prompt-engineer territory in safety-conscious product teams.
This challenge sharpens
- llm-evaluation
- red-teaming
- harness-design