Ewance
← All challenges NLP Engineer

LLM-Powered FAQ Chatbot for 40-Person SaaS Scale-up

Duration
2 weeks
Level
Advanced
Type
Code
Team
1 to 5 people

The situation

TaskFlow is a project management SaaS company based in Lisbon, with 40 employees. They have a comprehensive knowledge base but lack an automated way to handle common support queries, especially during the shift to enterprise clients.

Your task

How can TaskFlow leverage LLMs to automatically answer customer FAQs and reduce support workload?

The brief

What you do

  1. You have access to TaskFlow's internal documentation, help articles, and a sample of 500 support tickets.
  2. Your task is to build a retrieval-augmented generation (RAG) pipeline: index the documents, retrieve relevant chunks given a user query, and generate a concise answer using an LLM (e.g., GPT-3.5 via API).
  3. You must also evaluate the chatbot's accuracy on a set of 50 test questions.

Success looks like

Success means the chatbot correctly answers at least 70% of test questions and provides a demo.

What you will practice

  • Understand and implement retrieval-augmented generation (RAG) architecture
  • Use embeddings and vector databases for document retrieval
  • Integrate with an LLM API (e.g., OpenAI) to generate answers
  • Evaluate chatbot performance using metrics like accuracy and relevance

A suggested way in

  1. Use LangChain or LlamaIndex to orchestrate the pipeline
  2. Consider using sentence-transformers for embedding and FAISS for vector storage
  3. Preprocess documents by splitting into chunks of appropriate size
  4. Test different chunk sizes and retrieval strategies (e.g., top-k)

Skills you will practice

LLM, RAG, Information retrieval, Python, Langchain, API integration

How it is assessed

  • Pipeline Implementation40%

    Correctness and completeness of the RAG pipeline, including indexing and retrieval.

  • Performance & Evaluation30%

    Accuracy on test set and thoroughness of evaluation.

  • Creativity & Design20%

    Innovative solutions to handle edge cases or improve user experience.

  • Team Collaboration10%

    Effective teamwork and clear division of roles.

Resources

  • LangChainTool

    Simplifies building RAG pipelines with modular components for indexing and retrieval.

  • ChromaDBTool

    Open-source vector database for efficient similarity search on document embeddings.

  • OpenAI APITool

    Provides access to GPT-3.5/4 for generating answers from retrieved context.

  • StreamlitTool

    Quickly build an interactive web UI for the chatbot prototype.

  • RAGASTool

    Framework for evaluating RAG pipelines with metrics like faithfulness and answer relevancy.

  • GitHubTool

    Version control and collaboration platform for code and documentation.

  • NotionTool

    All-in-one workspace for project planning, documentation, and team coordination.

Sample Ewance certificate of completion, verified by LearnCoin
Sample certificate
Your proof

Finish it. Earn this.

A verified certificate with your name. Anyone can check it.

Time to practice.

Solve this challenge
Related challenges

More like this.