Skip to content
Technical University of Munich

Mechanistic Interpretability for Informed Computer Vision

Type: Master

Date: Immediately

Supervisor: Vasileios Xanthakis

Background

Informed Machine Learning integrates prior knowledge (such as physical laws, logical rules, domain constraints, knowledge graphs, or simulation results) explicitly into the ML pipeline, rather than relying on data alone. This can happen at various stages, including the training data, model architecture, learning algorithm, or learning objective. The goal is to improve generalization, data efficiency, robustness, or consistency with established domain knowledge.

Mechanistic Interpretability takes a complementary perspective: rather than specifying how a model should use knowledge, it attempts to reverse-engineer an already-trained model's internal computations into human-understandable features, representations, and algorithms. This often involves identifying the features and "circuits" responsible for particular model behaviors.

These two fields raise a natural question: when prior knowledge is explicitly incorporated into a model, is that knowledge reflected in the model's internal representations and computations? A model may benefit from injected knowledge without necessarily learning the representations or computational mechanisms that were intended by the model designer.

Computer vision, and particularly convolutional neural networks (CNNs), provides a suitable setting for investigating this question. CNNs learn hierarchical spatial representations that can be examined at the level of individual features, channels, layers, and circuits. This makes it possible to compare the internal representations of otherwise similar models trained with and without domain knowledge.

The central aim is therefore to investigate whether informed and non-informed computer vision models learn systematically different representations, and whether mechanistic interpretability can reveal how and where the injected knowledge influences the model's internal computations. This creates a direct connection between the two perspectives: informed ML puts knowledge into a model, while mechanistic interpretability can be used to investigate what the model actually learned from that knowledge.

Objectives

Possible directions include:

  • Comparing the internal representations of informed and non-informed CNNs across different layers
  • Investigating whether injected domain knowledge leads to different or more domain-aligned learned features
  • Identifying and comparing features and computational circuits responsible for specific visual behaviors in informed and non-informed models
  • Using mechanistic interpretability methods to determine whether the knowledge provided during training is reflected in the model's internal computations
  • Investigating whether differences in representations can be linked to differences in model behavior, generalization, or robustness
  • Detecting mismatches between the intended prior knowledge and the representations or mechanisms actually learned by the model
  • Investigating whether informed models develop more structured, compositional, or interpretable representations than their non-informed counterparts

The exact form of the prior knowledge, computer vision task, CNN architecture, and mechanistic interpretability methods will be scoped together based on feasibility and research interests.

Requirements

  • Solid ML/deep learning fundamentals
  • Interest in mechanistic interpretability and/or knowledge-guided modeling
  • Python programming (PyTorch or similar)
  • Understanding of convolutional neural networks and representation learning
  • Comfort reading recent, fast-moving literature (often preprints)

Introductory Literature

Von Rueden et al. (2021). Informed Machine Learning—A Taxonomy and Survey of Integrating Prior Knowledge into Learning Systems. IEEE Transactions on Knowledge and Data Engineering, 1–1. https://doi.org/10.1109/TKDE.2021.3079836

Bereska, L., & Gavves, E. (2024). Mechanistic Interpretability for AI Safety—A Review. arXiv:2404.14082. https://doi.org/10.48550/arXiv.2404.14082

Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., & Carter, S. (2020). Zoom In: An Introduction to Circuits. Distill, 5(3), e00024.001. https://doi.org/10.23915/distill.00024.001

A Mathematical Framework for Transformer Circuits. (2021). Transformer Circuits. https://transformer-circuits.pub/2021/framework/index.html

To top