Ewance
← All challenges AI Engineer

Hybrid Search RAG for a HR-Policy Assistant

Duration
2 weeks
Level
Intermediate
Type
Code
Team
1 to 5 people

The situation

The startup (Series A, around 35 staff) is rolling out to 5 enterprise clients next quarter; the logistics client is the lighthouse account whose case study unlocks the rest.

Your task

Build a hybrid (BM25 + dense + reranker) RAG that materially beats vector-only retrieval on an HR-policy assistant.

The brief

You start with

You receive 1,800 pages of policy documents (Markdown) and 150 labeled question-answer pairs with the gold source policy IDs.

What you do

  1. Build a hybrid retrieval pipeline: BM25 + dense embeddings, fused via Reciprocal Rank Fusion (RRF), then a cross-encoder reranker on the top-20.
  2. Generate a final answer with explicit policy-ID citations.
  3. Evaluate on (a) Hits@5 vs. vector-only baseline, (b) RAGAS faithfulness score on a 50-pair sample, and (c) citation-ID precision (the cited policy ID matches the gold).

Success looks like

Hits@5 improvement of at least 12 points, faithfulness above 0.90, citation precision above 95 percent.

What you will practice

  • Combine sparse (BM25) and dense retrieval with rank fusion
  • Apply a cross-encoder reranker to lift top-k quality
  • Evaluate RAG with multiple metrics (retrieval, faithfulness, citation)
  • Communicate retrieval improvements to a non-ML client

A suggested way in

  1. Use BM25 via Elasticsearch or the rank_bm25 Python library for simplicity
  2. Reciprocal Rank Fusion with k=60 is a solid default; do not invent novel fusion
  3. BAAI/bge-reranker-large is a strong open-source reranker baseline

Skills you will practice

Hybrid search, BM25, Dense retrieval, Reranking, RAG evaluation, Python

Where it leads

AI Engineer, NLP Engineer, Machine Learning Engineer

How it is assessed

  • Retrieval lift35%

    Hits@5 improves by at least 12 points over the vector-only baseline.

  • Faithfulness25%

    RAGAS faithfulness above 0.90 on the 50-pair sample.

  • Citation precision20%

    Cited policy IDs match the gold on at least 95 percent of pairs.

  • Client-facing note20%

    Note shows real before/after examples and explains the lift in plain language.

Resources

  • QdrantTool

    Open-source vector store with built-in hybrid support and good docs.

  • rank_bm25Tool

    Lightweight Python BM25 implementation for the sparse leg.

  • RAGASTool

    Standard library for RAG evaluation (faithfulness, answer relevance).

  • OpenAI APITool

    Embeddings and answer generation.

  • LangChainTool

    Glue for retrieval, fusion, and generation chains.

  • NotionTool

    Write the client-facing comparison note.

Sample Ewance certificate of completion, verified by LearnCoin
Sample certificate
Your proof

Finish it. Earn this.

A verified certificate with your name. Anyone can check it.

Time to practice.

Solve this challenge
Related challenges

More like this.