Enhancing Graphical User Interface Understanding for Applications

Bachelor & Master Thesis

Graphical User Interfaces (GUIs) are the primary medium through which users interact with mobile and web applications. Despite their importance, understanding of GUIs remains a complex and underexplored challenge. A key issue lies in the disconnect between the GUI’s rendered appearance, underlying structural representation, and human cognitive understanding. In many cases, the visual layout does not match the hierarchical structure, such as elements may appear misaligned, invisible ("ghost") elements might occupy space without being perceptible, and the actual interactive areas might differ significantly from what the hierarchy suggests. Furthermore, GUIs often contain visually ambiguous layouts, unclear boundaries between functional elements, and cognitive overload caused by dense or inconsistent designs. These limitations pose significant barriers for tasks such as automated GUI testing, interface design, and usability analysis.

This project aims to investigate how program analysis, machine learning, computer vision, and multimodal LLMs can be combined to enhance GUI understanding. The student will begin by (collecting and) understanding a dataset of real-world mobile or web application interfaces. Using this data, the student will explore patterns of structural misalignment, perceptual ambiguity, and interaction logic. This project will also involve designing and evaluating models that interpret GUI. 

This research has the potential to advance both software engineering (SE) and human-computer interaction (HCI) by making GUIs more interpretable for both humans and machines. By providing deeper insights into layout quality, visual saliency, and user-perceived interaction regions, it can support designers, developers, and testers in creating more accessible, maintainable, and user-friendly applications. 

Required knowledge:

Strong skills in Python programming and data processing. Familiar with (multimodal) large language models. Prior knowledge of GUI design, development or testing is a plus. The student should be self-motivated and demonstrate a strong passion for the thesis topic.