Diversifying Knowledge Enhancement of Biomedical Language Models using Adapter Modules and Knowledge Graphs
Recent advances in natural language processing (NLP) owe their success to pre-training language models on large amounts of unstructured data. Still, there is an increasing effort to combine the unstructured nature of LMs with structured knowledge and reasoning. Particularly in the rapidly evolving field of biomedical NLP, knowledge-enhanced language models (KELMs) have emerged as promising tools to bridge the gap between large language models and domain-specific knowledge, considering the available biomedical knowledge graphs (KGs) curated by experts over the decades. In this paper, we develop an approach that uses lightweight adapter modules to inject structured biomedical knowledge into pre-trained language models (PLMs). We use two large KGs, the biomedical knowledge system UMLS and the novel biochemical ontology OntoChem, with two prominent biomedical PLMs, PubMedBERT and BioLinkBERT. The approach includes partitioning knowledge graphs into smaller subgraphs, fine-tuning adapter modules for each subgraph, and combining the knowledge in a fusion layer. We test the performance on three downstream tasks: document classification,question answering, and natural language inference. We show that our methodology leads to performance improvements in several instances while keeping requirements in computing power low. Finally, we provide a detailed interpretation of the results and report valuable insights for future work.
| Attribute | Value |
|---|---|
| Address | Rome, Italy |
| Authors | Juraj Vladika , Alexander Fichtl , Prof. Dr. Florian Matthes |
| Citation | Vladika, J.; Fichtl, A. and Matthes, F. (2024). Diversifying Knowledge Enhancement of Biomedical Language Models Using Adapter Modules and Knowledge Graphs. In Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART; ISBN 978-989-758-680-4; ISSN 2184-433X, SciTePress, pages 376-387. |
| Key | |
| Research project | Scientific Claim Verification with Evidence from Text and Structured Knowledge (VeriSci) |
| Title | Diversifying Knowledge Enhancement of Biomedical Language Models using Adapter Modules and Knowledge Graphs |
| Type of publication | Conference |
| Year | 2024 |
| Publication URL | https://www.scitepress.org/PublicationsDetail.aspx?ID=DV0AJkeWcVo=&t=1 |
| Acronym | ICAART 2024 |
| Project | |
| Team members |