137. Advancing Noninvasive Diagnostics Through the Fusion of Imaging and Clinical Data

Abstract

Many clinical procedures involve collecting data samples in the form of imaging and tabular data. To enhance predictive capabilities, novel deep learning architectures are emerging, aiming to fuse information from both sources of information. Still, most solutions are based on simple concatenation or affine transformation mechanisms which limits the interaction between two modalities. In this seminar, we will focus on the integration of imaging and tabular data in Vision Models. As examples of possible applications, we will use a fetal birth weight prediction task and the estimation of mean Pulmonary Artery Pressure (mPAP). FBW prediction is a challenging task requiring clinicians to collect ultrasound videos of fetal body parts and fetal biometry measurements, while mPAP is currently measured invasively. Firstly, we will explore the feasibility of fetal birth weight prediction solely from imaging data, employing hybrid architectures that combine CNNs with Transformer-based models. Subsequently, we will focus on the refinement of predictions by incorporating the attention mechanism, computed with the assistance of tabular data. Then, we will explore the mPAP estimation with a modality mixing method based on multi-layer perceptrons. The seminar will end with an example of a different approach for tabular data utilization via the construction of interpretable scoring systems.

About the author

Michał Grzeszczyk is a graduate of Computer Science studies at the Warsaw University of Technology and Technical University of Berlin (dual Master’s degree). Earlier, he was a visiting researcher at The University of The West of Scotland and the Institute of Biocybernetics and Biomedical Engineering (Polish Academy of Sciences). Currently, he is a PhD student at Sano Centre for Computational Medicine in the Health Informatics Group. His research is based on multimodal data fusion with the biggest focus on combining imaging and tabular data in medicine. His work led to publications at the top-tier medical image processing conference – MICCAI. After hours, Michał develops mobile applications, especially Chefs’ appwhich is devoted to storing and sharing cooking recipes coming from multiple sources like images or websites.