Expert annotations in brain tumor MRI segmentation are inherently variable, capturing multiple valid interpretations rather than a single ground truth. Treating this variability as label noise can push models to over-optimize similarity to a single reference, without improving real-world utility. Instead, multiple expert segmentations can be used to estimate a consensus reference, and expert-to-expert differences can be used to define a practical AI performance range\u2014so a model is judged against what\u2019s clinically accurate and achievable, not a single annotator\u2019s mask.<\/p>\n\n\n\n
As part of the BraTS-METS team, data analyst Fabian Umeh and Sano researcher Monika Pytlarz<\/a>, PhD Student in Computational Neuroscience, have been investigating this challenge using inter-\/intra-rater reliability analysis and probabilistic consensus modeling.<\/p>\n\n\n\n Monika had the opportunity to present the Brats-METS team\u2019s latest research findings at the RSNA 2025 Annual Meeting \u2014 one of the world\u2019s leading conferences dedicated to innovation in medical imaging and radiology.<\/p>\n\n\n\n \u201cAt RSNA 2025, I had the opportunity to present our BraTS-METS team\u2019s research on inter- and intra-rater variability in brain tumor segmentation on MRI \u2014 an important factor to consider when training and validating AI algorithms to measure and track brain lesions. The quality of annotations impacts how well AI tools learn from training data, and it can also affect surgical planning, radiation treatment targeting, and follow-up comparisons. When radiologists segment tumors on MRI, even experienced specialists can draw slightly different borders. Our work focused on brain metastases using cases from the BraTS-METS Lighthouse 2025 Challenge, where each scan was outlined multiple times by expert neuroradiologists (including both fully manual work and AI-assisted refinement). To account for variability, we used a method called STAPLE, which combines multiple expert annotations into a probabilistic consensus map. Our results showed that manual expert segmentation shows only fair inter- and intra-rater reliability (mean Dice score \u2248 0.81; \u03ba \u2248 0.20). Importantly, AI-assisted segmentations were more consistent than manual-only outlines. Why this matters: better estimation of segmentation ground truth improves reproducibility, strengthens quality assurance, and helps build robust AI tools that account for real-world variability, ultimately supporting more reliable clinical decision-making.\u201d<\/p>\n\n\n\n A recording of Monika\u2019s full presentation, where she discusses the results in more detail and reflects on their implications for AI in neuroimaging, is available to watch via the Sano Seminars YouTube channel link<\/a>. <\/p>\n\n\n\nInvestigating Reliability through Multi-Annotator Data<\/h2>\n\n\n\n