Accurate is not the same as reliable: AI in clinical settings

New publication

A model that scores well on accuracy tests will not necessarily work reliably in a real hospital. In a paper for the 34th International Conference on Information Systems Development (ISD 2026), Sano’s Paulina Tworek, Yousef Khan, Miłosz Bargieł and Jose Sousa propose a new way to judge this. Drawing on Actor-Network Theory – an approach that studies how people, tools and rules come to work together as one system – they introduce the “enrollability” framework: four criteria that describe when a language-processing method can become a dependable part of clinical infrastructure. The team then compared a rule-based system with two large language models (LLMs), Llama-3-8B-it and Gemma-7b-it, on 1,679 Polish clinical summaries (epicrises) of young patients. The two yardsticks – accuracy and enrollability – gave different verdicts. The authors suggest that common LLM failures, such as hallucinations (confidently made-up content), variability between physicians and mismatched notation, may be linked to how these models generate text – by predicting the next word – and not only to the limits of what they can do.

“Actor-Network Theory Analysis of When LLMs Can and Cannot Be Enrolled”
Autors: Paulina Tworek, Yousef Khan, Miłosz Bargieł, Tomasz Pełech-Pilichowski, Marek Mikołajczyk, Roman Lewandowski, Jose Sousa