168. Distributed Memory Access Mechanisms on HPC Clusters for Memory-intensive Medical Applications

Abstract

While nodes on the HPC and Cloud instances offer huge computational resources they are not always sufficient for each medical use-case. An important examples of applications that exceeds the capacity of single node in the cluster due to the need of huge Random Access Memory (RAM) ammount are biomedical applications esp. from genomics domain. On the other hand, there are applications that must be run on multiple nodes to get results in acceptable time yet each instance of may share most of the data. For example, Spliced Transcripts Alignment to a Reference (STAR) tool needs to load the same index on multiple nodes, which wastes valuable resources if not handled properly.

During the seminar we will present the work focusing on various mechanisms for remote access to memory in distributed systems based on two distinct HPC clusters. We are comparing solutions based on the shared storage and MPI (over Infiniband and Slingshot) to the local memory access. We have found out that results for remote access esp. backed by MPI are similar to local memory access.

About the author:

Jan Meizner

Extreme-scale Data and Computing Team, Sano Centre for Computational Medicine