141. Advancements in Synthetic Data Generation and Novel View Synthesis for Enhanced Medical Image Analysis and Computer-Assisted Surgical Systems

Abstract:

The medical community faces significant challenges due to limited data availability, data scarcity, and the burdensome task of data labeling, which are crucial for training deep learning models and autonomous surgical robots. Recently, synthetic data generation has emerged, allowing for the creation of highly realistic data indistinguishable from real data. This advancement benefits systems reliant on deep learning models trained on complex and difficult-to-annotate data. Additionally, novel view synthesis has seen remarkable growth, thanks to neural rendering and Gaussian splatting, enabling new reconstruction capabilities and accurate novel view synthesis. 

In this seminar, we will explore innovative approaches to bridge the gap between synthetic and real data, enhancing the effectiveness of both medical image analysis and computer-assisted surgical (CAS) systems.We will begin with a method that combines Stable Diffusion and ControlNet to generate realistic endoscopic images from synthetic data. This approach significantly reduces the domain gap and requires less input data compared to traditional methods. The method is effective in simplified scenarios and extends to a fully-fledged surgical simulator, generating comprehensive annotations and simulating complex surgical interactions. By integrating an image-to-image translation model, it maintains the integrity of the generated annotations and enhances the realism of synthetic datasets.

Next, we will discuss how generative models provide exceptional representations useful for other tasks. We utilize a diffusion model to create a shared representation with a classification model, demonstrating its applicability to semi-supervised classification with very limited labeled data and its capability for self-explainability. This approach combines generative tasks with standard classification, learning from both labeled and unlabeled data while offering explainable results through counterfactual examples. It addresses the critical need for high performance and confidence in medical applications, achieving results comparable to recent semi-supervised methods.

Finally, we will briefly discuss novel view synthesis approaches related to computer graphics, aiming to increase the quality and utility of medical data.

About the author:

Joanna Kaleta is a PhD student at Sano and has also joined the doctoral school at Warsaw University of Technology, Faculty of Computer Science. She holds a Master’s in Computer Science from the same institution. Her research has primarily focused on generative modeling and bridging the gap between simulation and real medical data. Recently, she has expanded her interests to include computer graphics and novel view synthesis.