206. Optical Flow-Guided Diffusion: Bridging the Sim-to-Real Gap in Medical Image Generation

Abstract

The domain gap between synthetic surgical simulators and real-world operating room footage remains a critical bottleneck in developing medical surgical training. While surgical simulators provide abundant data with perfect anatomical control, their visual representation lacks the complex textures, lighting, and tissue dynamics of actual procedures.

This presentation will explore methods for Sim-to-Real translation in cataract surgery leveraging advanced diffusion models. Starting from the dataset analysis and transformation pipeline required to harmonize a complex, multi-modal dataset. Focusing on methodologies used to ensure cross-domain image pairs – simulated and realistic are spatially and temporally synchronized, resulting in a high-fidelity, geometrically consistent dataset optimized for generative modeling. To ensure the generated images maintain strict anatomical integrity and temporal consistency across video frames, the translation process is explicitly conditioned on optical flow representations, depth maps, and ground truth semantic masks. By mapping these multi-modal simulation inputs to realistic motion dynamics, the approach preserves the geometric constraints of both the surgical instruments and the anatomy. 

This flow-guided methodology translates synthetic inputs into high-fidelity, photorealistic surgical sequences. The framework effectively bridges the simulation-to-reality gap, providing a method for medical image generation that can significantly enhance surgical training simulators.

About the author:

Michał holds a Master’s degree in biomedical engineering. At Sano he is part of the Medical Image analysis team. His PhD research initially focused on autonomous ultrasound robotic systems and ultrasound data analysis. He has since transitioned to utilizing diffusion models to bridge the gap between surgical simulations and real-world clinical data