Design Prompt Versioning and Observability for a Coding Assistant
Overview
What this challenge is about.
You will (1) design a prompt-registry data model (versions, owners, environments, change log) and implement it in Postgres + a small Python SDK, (2) instrument the assistant to tag every LLM request with prompt_version_id, (3) log requests and responses to an analytics table with PII scrubbing (regex + a small detector for code containing API keys), (4) build a Streamlit dashboard showing per-version request count, p50/p95 latency, cost, and a basic thumbs-up/down rate. Deliver: schema, SDK, instrumentation patch, dashboard, and a 4-page rollout plan including a backfill strategy.
The Brief
What you'll do, and what you'll demonstrate.
Make prompt changes observable and versioned, with PII-safe response logging and a per-version metrics dashboard.
Earning criteria — what you'll demonstrate
- Design a prompt-registry schema with versions, owners, and environments
- Instrument an LLM-app for per-request observability
- Apply PII scrubbing to LLM request/response logs
- Build a metrics dashboard that catches silent regressions
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
Careers
Roles this prepares you for.
Real titles. Real skill bridges. Pick the one closest to your trajectory.
MLOps Engineer
Prompt versioning and observability is core MLOps work for any LLM-powered product team.
This challenge sharpens
- prompt-versioning
- observability
- pii-scrubbing
AI Engineer
Instrumenting an LLM app for observability is what AI engineers ship next at any scaling AI startup.
This challenge sharpens
- fastapi
- observability
- prompt-versioning
Prompt Engineer
A prompt registry and a metrics dashboard are exactly the tooling prompt engineers need to do their job at production scale.
This challenge sharpens
- prompt-versioning
- observability
- pii-scrubbing