The situation
The startup (Series A, around 35 staff) is rolling out to 5 enterprise clients next quarter; the logistics client is the lighthouse account whose case study unlocks the rest.
Your task
Build a hybrid (BM25 + dense + reranker) RAG that materially beats vector-only retrieval on an HR-policy assistant.
The brief
You start with
You receive 1,800 pages of policy documents (Markdown) and 150 labeled question-answer pairs with the gold source policy IDs.
What you do
- Build a hybrid retrieval pipeline: BM25 + dense embeddings, fused via Reciprocal Rank Fusion (RRF), then a cross-encoder reranker on the top-20.
- Generate a final answer with explicit policy-ID citations.
- Evaluate on (a) Hits@5 vs. vector-only baseline, (b) RAGAS faithfulness score on a 50-pair sample, and (c) citation-ID precision (the cited policy ID matches the gold).
Success looks like
Hits@5 improvement of at least 12 points, faithfulness above 0.90, citation precision above 95 percent.
What you will practice
- Combine sparse (BM25) and dense retrieval with rank fusion
- Apply a cross-encoder reranker to lift top-k quality
- Evaluate RAG with multiple metrics (retrieval, faithfulness, citation)
- Communicate retrieval improvements to a non-ML client
A suggested way in
- Use BM25 via Elasticsearch or the rank_bm25 Python library for simplicity
- Reciprocal Rank Fusion with k=60 is a solid default; do not invent novel fusion
- BAAI/bge-reranker-large is a strong open-source reranker baseline
Skills you will practice
Hybrid search, BM25, Dense retrieval, Reranking, RAG evaluation, Python
Where it leads
AI Engineer, NLP Engineer, Machine Learning Engineer
How it is assessed
- Retrieval lift35%
Hits@5 improves by at least 12 points over the vector-only baseline.
- Faithfulness25%
RAGAS faithfulness above 0.90 on the 50-pair sample.
- Citation precision20%
Cited policy IDs match the gold on at least 95 percent of pairs.
- Client-facing note20%
Note shows real before/after examples and explains the lift in plain language.
Resources
- QdrantTool
Open-source vector store with built-in hybrid support and good docs.
- rank_bm25Tool
Lightweight Python BM25 implementation for the sparse leg.
- RAGASTool
Standard library for RAG evaluation (faithfulness, answer relevance).
- OpenAI APITool
Embeddings and answer generation.
- LangChainTool
Glue for retrieval, fusion, and generation chains.
- NotionTool
Write the client-facing comparison note.
Finish it. Earn this.
A verified certificate with your name. Anyone can check it.
Time to practice.
Solve this challengeMore like this.
- Software EngineerBuild a Multi-Level Feedback Queue Scheduler in xv6
- Software EngineerReverse-Engineer a Stripped Binary to Restore Linking Symbols
- Software EngineerImplement a Distance-Vector Routing Algorithm
- AI EngineerEnd-to-End Lane Following on a Donkeycar Platform
-
Build a Multi-Level Feedback Queue Scheduler in xv6 Software Engineer Fork the xv6-riscv repository and implement MLFQ in proc.c with 3 priority queues, per-queue time slices (10ms / 20ms / 40ms), priority demotion on sl operating systemsschedulingc programming Open challenge → -
Reverse-Engineer a Stripped Binary to Restore Linking Symbols Software Engineer Receive the stripped shared object (Linux ELF, x86-64, ~280KB), a crash dump with 12 frames of indecipherable addresses, and 4 years of partial git hi reverse engineeringlinking and loadingmachine code Open challenge → -
Implement a Distance-Vector Routing Algorithm Software Engineer Implement distance-vector routing in Python: each simulated router maintains a distance vector, exchanges vectors with neighbors every 1 second of sim routingdistance vectornetwork simulation Open challenge → -
End-to-End Lane Following on a Donkeycar Platform AI Engineer Use the public Donkeycar Tub dataset (or collect about 30 minutes of driving on the simulator). Train a CNN-policy baseline (the Donkeycar default arc end to end learningimitation learningpytorch Open challenge →