The situation
TaskFlow is a project management SaaS company based in Lisbon, with 40 employees. They have a comprehensive knowledge base but lack an automated way to handle common support queries, especially during the shift to enterprise clients.
Your task
How can TaskFlow leverage LLMs to automatically answer customer FAQs and reduce support workload?
The brief
What you do
- You have access to TaskFlow's internal documentation, help articles, and a sample of 500 support tickets.
- Your task is to build a retrieval-augmented generation (RAG) pipeline: index the documents, retrieve relevant chunks given a user query, and generate a concise answer using an LLM (e.g., GPT-3.5 via API).
- You must also evaluate the chatbot's accuracy on a set of 50 test questions.
Success looks like
Success means the chatbot correctly answers at least 70% of test questions and provides a demo.
What you will practice
- Understand and implement retrieval-augmented generation (RAG) architecture
- Use embeddings and vector databases for document retrieval
- Integrate with an LLM API (e.g., OpenAI) to generate answers
- Evaluate chatbot performance using metrics like accuracy and relevance
A suggested way in
- Use LangChain or LlamaIndex to orchestrate the pipeline
- Consider using sentence-transformers for embedding and FAISS for vector storage
- Preprocess documents by splitting into chunks of appropriate size
- Test different chunk sizes and retrieval strategies (e.g., top-k)
Skills you will practice
LLM, RAG, Information retrieval, Python, Langchain, API integration
How it is assessed
- Pipeline Implementation40%
Correctness and completeness of the RAG pipeline, including indexing and retrieval.
- Performance & Evaluation30%
Accuracy on test set and thoroughness of evaluation.
- Creativity & Design20%
Innovative solutions to handle edge cases or improve user experience.
- Team Collaboration10%
Effective teamwork and clear division of roles.
Resources
- LangChainTool
Simplifies building RAG pipelines with modular components for indexing and retrieval.
- ChromaDBTool
Open-source vector database for efficient similarity search on document embeddings.
- OpenAI APITool
Provides access to GPT-3.5/4 for generating answers from retrieved context.
- StreamlitTool
Quickly build an interactive web UI for the chatbot prototype.
- RAGASTool
Framework for evaluating RAG pipelines with metrics like faithfulness and answer relevancy.
- GitHubTool
Version control and collaboration platform for code and documentation.
- NotionTool
All-in-one workspace for project planning, documentation, and team coordination.
Finish it. Earn this.
A verified certificate with your name. Anyone can check it.
Time to practice.
Solve this challengeMore like this.
- Software EngineerBuild a Multi-Level Feedback Queue Scheduler in xv6
- Software EngineerReverse-Engineer a Stripped Binary to Restore Linking Symbols
- Software EngineerImplement a Distance-Vector Routing Algorithm
- AI EngineerEnd-to-End Lane Following on a Donkeycar Platform
-
Build a Multi-Level Feedback Queue Scheduler in xv6 Software Engineer Fork the xv6-riscv repository and implement MLFQ in proc.c with 3 priority queues, per-queue time slices (10ms / 20ms / 40ms), priority demotion on sl operating systemsschedulingc programming Open challenge → -
Reverse-Engineer a Stripped Binary to Restore Linking Symbols Software Engineer Receive the stripped shared object (Linux ELF, x86-64, ~280KB), a crash dump with 12 frames of indecipherable addresses, and 4 years of partial git hi reverse engineeringlinking and loadingmachine code Open challenge → -
Implement a Distance-Vector Routing Algorithm Software Engineer Implement distance-vector routing in Python: each simulated router maintains a distance vector, exchanges vectors with neighbors every 1 second of sim routingdistance vectornetwork simulation Open challenge → -
End-to-End Lane Following on a Donkeycar Platform AI Engineer Use the public Donkeycar Tub dataset (or collect about 30 minutes of driving on the simulator). Train a CNN-policy baseline (the Donkeycar default arc end to end learningimitation learningpytorch Open challenge →