Overview
What this challenge is about.
Build an LLM evaluation harness in Python with four test suites, then compare two models. Earn a verifiable certificate.
The scenario
The consultancy (~600 staff) reuses evaluation harnesses across engagements; a strong, defensible harness saves about 3 engineer-weeks per future client.
The Brief
What you'll do, and what you'll demonstrate.
Build a reusable LLM evaluation harness that covers helpfulness, grounding, refusal, and prompt-injection resistance, and use it to pick a base model.
Earning criteria — what you'll demonstrate
- Design an evaluation harness that covers safety and quality dimensions
- Apply LLM-as-judge with rubrics and inter-rater calibration
- Test for prompt injection with a meaningful threat model
- Communicate evaluation results as a model-selection decision
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Large Language Models
Master · Generative Ai
Strong alignment
This challenge maps to Large Language Models at the Master level. It sharpens the same practical skills your coursework expects — but in a real industry context with actual constraints and deliverables.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
- Llm Evaluation
Apply llm evaluation to solve real industry problems and demonstrate production-level capability.
- Llm As Judge
Apply llm as judge to solve real industry problems and demonstrate production-level capability.
- Prompt Injection Testing
Apply prompt injection testing to solve real industry problems and demonstrate production-level capability.
- Grounding Evaluation
Apply grounding evaluation to solve real industry problems and demonstrate production-level capability.
- Benchmark Design
Apply benchmark design to solve real industry problems and demonstrate production-level capability.
- Python
Write clean, efficient Python for data processing, automation, and backend services.
Careers
Career paths this challenge builds toward
Completing this challenge demonstrates skills that transfer directly to these roles:
AI Safety Researcher
Building a multi-dimensional LLM evaluation harness is core safety-research work at any enterprise-AI vendor.
This challenge sharpens
- llm-evaluation
- prompt-injection-testing
- grounding-evaluation
ML Researcher
Designing test cases and judge calibration is the methodological core of LLM-as-judge research.
This challenge sharpens
- llm-as-judge
- benchmark-design
- llm-evaluation
AI Engineer
Wiring a reusable evaluation harness into the engagement workflow is the AI-engineer skillset that consultancies hire for.
This challenge sharpens
- python
- llm-evaluation
- benchmark-design