Comparing Cloud and On-Premise RAG Solutions for Large-Scale Medical Question Answering
Retrieval-Augmented Generation (RAG) has become the standard approach for grounding large language models in domain-specific knowledge, and medical question answering is one of the areas where it matters most, since answers need to be supported by correct citations. Teams building these systems today face a choice between adopting a managed cloud service such as Google's Agent Platform, which is fast to set up and easy to operate, or building an on-premise pipeline, which takes more effort but allows control over every component. The trade-offs between the two approaches are discussed often but rarely measured, and no published evaluation compares them on large-scale medical corpora.
Two pipelines are built and compared using an established medical question answering benchmark over a PubMed corpus. The first pipeline is built end-to-end on Google's Agent Platform (formerly Vertex AI), the managed cloud option. The second is a configurable on-premise stack that allows individual components to be swapped and tuned. Both pipelines are evaluated at varying corpus sizes to make scaling behavior visible, and along three dimensions: architectural flexibility, performance at scale, and the contribution of individual components to overall quality and scalability.
The methodology relies on a two-part evaluation. Retrieval quality is measured against gold snippets using standard information retrieval metrics such as recall, precision, and MRR, while end-to-end answer quality is measured separately using LLM-as-a-judge against gold answers. This separation enables a systematic ablation in which pipeline hyperparameters are varied and each configuration is evaluated at the closest metric. The same procedure is repeated across various corpus sizes to identify which factors dominate scaling behavior.
The thesis delivers three contributions: a reproducible benchmark for comparing managed and self-hosted medical RAG at scale, an empirical attribution of quality and scalability differences to individual pipeline components, and practical guidance for medical informatics teams choosing between vendor-managed and in-house RAG infrastructure.
| Attribute | Value |
|---|---|
| Title (de) | Comparing Cloud and On-Premise RAG Solutions for Large-Scale Medical Question Answering |
| Title (en) | Comparing Cloud and On-Premise RAG Solutions for Large-Scale Medical Question Answering |
| Project | AIdvice |
| Type | Master's Thesis |
| Status | started |
| Student | Erhan Varlik |
| Advisor | Jonas Gottal, Fabian Karl |
| Supervisor | Prof. Dr. Florian Matthes |
| Start Date | 01.05.2026 |
| Sebis Contributor Agreement signed | yes |
| Checklist filled | yes |
| Submission date | 02.11.2026 |