New Paper Published in Behaviour & Information Technology: “Quest for Quality: A Review of Design Knowledge on Gamified AI Training Data Annotation Systems”

Authored by Simon Krohmann, Sascha Rank, Manuel Schmidt-Kraepelin, Scott Thiebes and Ali Sunyaev, the study investigates the current state of design knowledge on gamified AI training data annotation systems based on a review of 56 scientific publications. The main results include

• A broad overview of design artifacts and research activities
• Synthesis and maturity assessment of 13 solution streams describing relationships between goals and gamification-based solutions
• Recommendations for more mature design knowledge on gamified annotation systems

Read the full paper here (open access): https://doi.org/10.1080/0144929X.2025.2568932.

Abstract. Data annotation is a tedious, yet vital task to create AI training data. Gamification can support annotation quality by motivating and engaging annotators. Existing gamified annotation system artifacts are largely instantiations; it remains unclear how the ingrained design knowledge transcends individual and situational factors and can inform the design of gamified annotation systems as a system class. This study synthesises extant gamified annotation system artifacts and gauges the maturity of ingrained design knowledge. We conduct a semi-systematic review of 56 articles which present gamified annotation system artifacts. Beyond a broad overview of design artifacts and research activities, we derive 13 solution streams that describe design knowledge as means-end relationships between goals and gamification-based solutions. While some solution streams exhibit high maturity, the results largely confirm our initial assumption that the design knowledge base on gamified annotation systems is immature. To advance maturity, we recommend to move beyond creating new instantiations as the primary research activity and to focus on developing actionable design prescriptions. Our study contributes to creating more mature design knowledge on gamified annotation systems and offers valuable perspectives on the maturity and scholarly progression of gamification research.