Human-in-the-Loop Knowledge Refinement for RAG-Based Healthcare Chatbots
Large language models (LLMs) can be used to answer user questions in healthcare contexts, but producing accurate responses that rely on specialized knowledge remains challenging. Retrieval-Augmented Generation (RAG) addresses this issue by retrieving relevant information from a knowledge base during response generation. However, the effectiveness of RAG strongly depends on the quality of the knowledge base, which must be kept accurate and up to date through continuous maintenance.
This thesis presents the design and implementation of a German-language RAG-based healthcare chatbot that incorporates a human-in-the-loop knowledge refinement process. Domain data from Breathment is used to construct the initial knowledge base. A review dashboard allows experts to inspect chatbot responses, identify missing or incorrect information, and refine the knowledge base accordingly.
The thesis investigates how different knowledge base representations, such as unstructured documents and structured Q&A pairs, as well as different update strategies, influence answer quality. The system is evaluated using a domain-specific question set reviewed by physiotherapists. Responses are assessed for correctness, completeness, and potential harmfulness. The results analyze how answer quality changes across human-in-the-loop refinement cycles and how knowledge base representations and update strategies influence these improvements.
Research Questions:
RQ1:What is the current state of the art in human-in-the-loop and LLM-based refinement approaches for Retrieval-Augmented Generation (RAG) systems?
RQ2: How do different knowledge base representations (eg. unstructured text, Q&A pairs) and update strategies (eg. modifying existing entries, appending new entries) affect answer quality in a German-language, domain-specific RAG-based healthcare chatbot?
RQ3: To what extent does the implemented continuous knowledge feedback loop improve the overall answer quality of the RAG-based chatbot over iteration cycles?
| Attribute | Value |
|---|---|
| Title (de) | Human-in-the-Loop Knowledge Refinement for RAG-Based Healthcare Chatbots |
| Title (en) | Human-in-the-Loop Knowledge Refinement for RAG-Based Healthcare Chatbots |
| Project | |
| Type | Master's Thesis |
| Status | started |
| Student | Berkay Atatop |
| Advisor | Katharina Sommer |
| Supervisor | Prof. Dr. Florian Matthes |
| Start Date | 11.03.2026 |
| Sebis Contributor Agreement signed on | |
| Checklist filled | |
| Submission date | 11.09.2026 |