Non-Invasive Dynamic FaaS Infrastructure with HPC-Whisk – Sano in Dallas, USA

13-18th November 2022 in Dallas, USA had place the International Conference for High-Performance Computing, Networking, Storage, and Analysis. This is one of the most important events in supercomputing field and Sano Science was there! 

Maciej Malawski, the Director of Sano Science and team leader of the Extreme-scale Data and Computing research group in Sano is one of the authors of the paper presented at SC22 conference. 

The presentation was on Wednesday, November 16th, and was titled “Non-Invasive Dynamic FaaS Infrastructure with HPC-Whisk”. 

As we could read in abstract: 

Modern HPC workload managers and their careful tuning contribute to the high utilization of HPC clusters. However, due to inevitable uncertainty it is impossible to completely avoid node idleness. Although such idle slots are usually too short for any HPC job, they are too long to ignore them. Function-as-a-Service (FaaS) paradigm promisingly fills this gap, and can be a good match, as typical FaaS functions last seconds, not hours. Here we show how to build a FaaS infrastructure on idle nodes in an HPC cluster in such a way that it does not affect the performance of the HPC jobs significantly. We dynamically adapt to a changing set of idle physical machines, by integrating open-source software Slurm and OpenWhisk. 

We designed and implemented a prototype solution that allowed us to cover up to 90% of the idle time slots on a 50k-core cluster that runs production workloads. 

Congratulations!

Also Marek Michalewicz, PhD, Sano’s Visiting Professor was in Dallas at SC Conference Series and he was one of the Wednesday session “Bridging the HPC/Data Divide” leaders. 

More details about this session: https://lnkd.in/dV7BVjS7