Build Hybrid Keyword-Plus-Embeddings Search for a Legal Document Portal
Overview
What this challenge is about.
Build a hybrid BM25 and embeddings search for a legal portal, compare three configurations, and earn a verifiable certificate.
The scenario
Legal-technology vendors serving Southeast Asian law firms compete on how quickly a lawyer can find the one controlling clause inside millions of contracts, judgments, and filings — where a missed passage can mean a missed argument. This company's portal indexes about 2.4 million documents but ranks them with keyword matching alone, so semantically equivalent wording is invisible to search.
The Brief
What you'll do, and what you'll demonstrate.
Design and prove a hybrid keyword-plus-embeddings retrieval system that measurably out-ranks the keyword-only baseline on a curated relevance set while remaining safe to ship to a live legal portal.
Earning criteria — what you'll demonstrate
- Combine lexical (BM25) and dense-embedding retrieval into a single ranked result using reciprocal rank fusion
- Select an embedding model by reasoning explicitly about indexing cost, latency, and retrieval quality trade-offs
- Evaluate retrieval quality with mean reciprocal rank and Recall@10, including confidence intervals and a labelled relevance set
- Assess relevance-judgment quality, including coverage of real user intents and agreement between annotators
- Design a low-risk production rollout using feature flags, telemetry, and an explicit rollback path
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Aligned coursework coming soon.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
- Information Retrieval
Apply information retrieval to solve real industry problems and demonstrate production-level capability.
- Bm25
Apply bm25 to solve real industry problems and demonstrate production-level capability.
- Vector Search
Apply vector search to solve real industry problems and demonstrate production-level capability.
- Hybrid Retrieval
Apply hybrid retrieval to solve real industry problems and demonstrate production-level capability.
- Relevance Evaluation
Apply relevance evaluation to solve real industry problems and demonstrate production-level capability.
- Feature Flags
Apply feature flags to solve real industry problems and demonstrate production-level capability.
Careers
Career paths this challenge builds toward
Completing this challenge demonstrates skills that transfer directly to these roles:
Search / Retrieval Engineer
This challenge mirrors the core of a search-engineering role: standing up keyword and vector retrieval, fusing them, and proving quality gains on a labelled set. You leave able to defend an embedding-model choice and a relevance methodology to product and engineering stakeholders.
This challenge sharpens
- bm25
- vector-search
- hybrid-retrieval
Machine Learning Engineer (Search & Ranking)
Ranking-focused ML roles live or die by evaluation discipline. By building a relevance set, reporting mean reciprocal rank and Recall@10 with confidence intervals, and reasoning about embeddings, you practice the exact loop these teams run before shipping a model change.
This challenge sharpens
- relevance-evaluation
- vector-search
- information-retrieval
Backend Platform Engineer
Shipping a retrieval change to a 2.4-million-document production portal demands rollout discipline, not just model accuracy. The feature-flag staging, telemetry, and rollback work here maps directly to how platform engineers release risky changes safely.
This challenge sharpens
- feature-flags
- information-retrieval
- hybrid-retrieval