{"id":14927,"date":"2024-01-16T13:02:09","date_gmt":"2024-01-16T12:02:09","guid":{"rendered":"https:\/\/sano.science\/?post_type=research&#038;p=14927"},"modified":"2024-01-16T13:02:09","modified_gmt":"2024-01-16T12:02:09","slug":"using-unused-non-invasive-dynamic-faas-infrastructure-with-hpc-whisk","status":"publish","type":"research","link":"https:\/\/sano.science\/research\/using-unused-non-invasive-dynamic-faas-infrastructure-with-hpc-whisk\/","title":{"rendered":"Using Unused: Non-Invasive Dynamic FaaS Infrastructure with HPC-Whisk"},"content":{"rendered":"\n<h2 class=\"wp-block-heading eplus-wrapper\">Bart\u0142omiej Przybylski, Maciej Pawlik, Pawe\u0142 \u017buk, Bart\u0142omiej \u0141agosz, Maciej Malawski, Krzysztof Rzadca<\/h2>\n\n\n\n<div style=\"height:50px\" aria-hidden=\"true\" class=\"wp-block-spacer eplus-wrapper\"><\/div>\n\n\n\n<p class=\" eplus-wrapper\">Modern HPC workload managers and their careful tuning contribute to the high utilization of HPC clusters. However, due to inevitable uncertainty it is impossible to completely avoid node idleness. Although such idle slots are usually too short for any HPC job, they are too long to ignore them. Function-as-a-Service (FaaS) paradigm promisingly fills this gap, and can be a good match, as typical FaaS functions last seconds, not hours. Here we show how to build a FaaS infrastructure on idle nodes in an HPC cluster in such a way that it does not affect the performance of the HPC jobs significantly. We dynamically adapt to a changing set of idle physical machines, by integrating open-source software Slurm and OpenWhisk.<br>We designed and implemented a prototype solution that allowed us to cover up to 90\\% of the idle time slots on a 50k-core cluster that runs production workloads.<\/p>\n\n\n\n<div style=\"height:50px\" aria-hidden=\"true\" class=\"wp-block-spacer eplus-wrapper\"><\/div>\n\n\n\n\t\n    \n        \n\t\t\t<a href=\"https:\/\/arxiv.org\/abs\/2211.00717\" target=\"_blank\" rel= \"noopener noreferrer nofollow\" class=\"button primary \">\n\n\t\t\t\t<span>\n\t\t\t\t\tREAD HERE\n\t\t\t\t<\/span>\n\n\t\t\t<\/a>\n\n        \n    \n","protected":false},"excerpt":{"rendered":"<p>In: International Conference for High Performance Computing, Networking, Storage, and Analysis (SC22), 2022.<\/p>\n","protected":false},"featured_media":0,"template":"","research_type":[8],"research_team":[16],"class_list":["post-14927","research","type-research","status-publish","hentry","research_type-publications","research_team-extreme-scale-data-and-computing"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.4 (Yoast SEO v27.4) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Using Unused: Non-Invasive Dynamic FaaS Infrastructure with HPC-Whisk - Centre for Computational Personalized Medicine<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/sano.science\/research\/using-unused-non-invasive-dynamic-faas-infrastructure-with-hpc-whisk\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Using Unused: Non-Invasive Dynamic FaaS Infrastructure with HPC-Whisk\" \/>\n<meta property=\"og:description\" content=\"In: International Conference for High Performance Computing, Networking, Storage, and Analysis (SC22), 2022.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/sano.science\/research\/using-unused-non-invasive-dynamic-faas-infrastructure-with-hpc-whisk\/\" \/>\n<meta property=\"og:site_name\" content=\"Centre for Computational Personalized Medicine\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/sano.science\/\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@sanoscience\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/sano.science\\\/research\\\/using-unused-non-invasive-dynamic-faas-infrastructure-with-hpc-whisk\\\/\",\"url\":\"https:\\\/\\\/sano.science\\\/research\\\/using-unused-non-invasive-dynamic-faas-infrastructure-with-hpc-whisk\\\/\",\"name\":\"Using Unused: Non-Invasive Dynamic FaaS Infrastructure with HPC-Whisk - Centre for Computational Personalized Medicine\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/sano.science\\\/#website\"},\"datePublished\":\"2024-01-16T12:02:09+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/sano.science\\\/research\\\/using-unused-non-invasive-dynamic-faas-infrastructure-with-hpc-whisk\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/sano.science\\\/research\\\/using-unused-non-invasive-dynamic-faas-infrastructure-with-hpc-whisk\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/sano.science\\\/research\\\/using-unused-non-invasive-dynamic-faas-infrastructure-with-hpc-whisk\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/sano.science\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Research\",\"item\":\"https:\\\/\\\/sano.science\\\/research\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Publications\",\"item\":\"https:\\\/\\\/sano.science\\\/research-type\\\/publications\\\/\"},{\"@type\":\"ListItem\",\"position\":4,\"name\":\"Using Unused: Non-Invasive Dynamic FaaS Infrastructure with HPC-Whisk\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/sano.science\\\/#website\",\"url\":\"https:\\\/\\\/sano.science\\\/\",\"name\":\"Centre for Computational Personalized Medicine\",\"description\":\"Sano \u2013 Centre for Computational Medicine\",\"publisher\":{\"@id\":\"https:\\\/\\\/sano.science\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/sano.science\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/sano.science\\\/#organization\",\"name\":\"Sano \u2013 Centre for Computational Medicine\",\"alternateName\":\"Sano\",\"url\":\"https:\\\/\\\/sano.science\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/sano.science\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/sano.science\\\/wp-content\\\/uploads\\\/2024\\\/05\\\/logo_sano_podstawowe.png\",\"contentUrl\":\"https:\\\/\\\/sano.science\\\/wp-content\\\/uploads\\\/2024\\\/05\\\/logo_sano_podstawowe.png\",\"width\":700,\"height\":265,\"caption\":\"Sano \u2013 Centre for Computational Medicine\"},\"image\":{\"@id\":\"https:\\\/\\\/sano.science\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/sano.science\\\/\",\"https:\\\/\\\/x.com\\\/sanoscience\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/sanoscience\\\/\",\"https:\\\/\\\/www.youtube.com\\\/channel\\\/UCDZ_8TcjMWUG2ZcgKKgfpwQ\",\"https:\\\/\\\/bsky.app\\\/profile\\\/sanoscience.bsky.social\"]}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Using Unused: Non-Invasive Dynamic FaaS Infrastructure with HPC-Whisk - Centre for Computational Personalized Medicine","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/sano.science\/research\/using-unused-non-invasive-dynamic-faas-infrastructure-with-hpc-whisk\/","og_locale":"en_US","og_type":"article","og_title":"Using Unused: Non-Invasive Dynamic FaaS Infrastructure with HPC-Whisk","og_description":"In: International Conference for High Performance Computing, Networking, Storage, and Analysis (SC22), 2022.","og_url":"https:\/\/sano.science\/research\/using-unused-non-invasive-dynamic-faas-infrastructure-with-hpc-whisk\/","og_site_name":"Centre for Computational Personalized Medicine","article_publisher":"https:\/\/www.facebook.com\/sano.science\/","twitter_card":"summary_large_image","twitter_site":"@sanoscience","twitter_misc":{"Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/sano.science\/research\/using-unused-non-invasive-dynamic-faas-infrastructure-with-hpc-whisk\/","url":"https:\/\/sano.science\/research\/using-unused-non-invasive-dynamic-faas-infrastructure-with-hpc-whisk\/","name":"Using Unused: Non-Invasive Dynamic FaaS Infrastructure with HPC-Whisk - Centre for Computational Personalized Medicine","isPartOf":{"@id":"https:\/\/sano.science\/#website"},"datePublished":"2024-01-16T12:02:09+00:00","breadcrumb":{"@id":"https:\/\/sano.science\/research\/using-unused-non-invasive-dynamic-faas-infrastructure-with-hpc-whisk\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/sano.science\/research\/using-unused-non-invasive-dynamic-faas-infrastructure-with-hpc-whisk\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/sano.science\/research\/using-unused-non-invasive-dynamic-faas-infrastructure-with-hpc-whisk\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/sano.science\/"},{"@type":"ListItem","position":2,"name":"Research","item":"https:\/\/sano.science\/research\/"},{"@type":"ListItem","position":3,"name":"Publications","item":"https:\/\/sano.science\/research-type\/publications\/"},{"@type":"ListItem","position":4,"name":"Using Unused: Non-Invasive Dynamic FaaS Infrastructure with HPC-Whisk"}]},{"@type":"WebSite","@id":"https:\/\/sano.science\/#website","url":"https:\/\/sano.science\/","name":"Centre for Computational Personalized Medicine","description":"Sano \u2013 Centre for Computational Medicine","publisher":{"@id":"https:\/\/sano.science\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/sano.science\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/sano.science\/#organization","name":"Sano \u2013 Centre for Computational Medicine","alternateName":"Sano","url":"https:\/\/sano.science\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/sano.science\/#\/schema\/logo\/image\/","url":"https:\/\/sano.science\/wp-content\/uploads\/2024\/05\/logo_sano_podstawowe.png","contentUrl":"https:\/\/sano.science\/wp-content\/uploads\/2024\/05\/logo_sano_podstawowe.png","width":700,"height":265,"caption":"Sano \u2013 Centre for Computational Medicine"},"image":{"@id":"https:\/\/sano.science\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/sano.science\/","https:\/\/x.com\/sanoscience","https:\/\/www.linkedin.com\/company\/sanoscience\/","https:\/\/www.youtube.com\/channel\/UCDZ_8TcjMWUG2ZcgKKgfpwQ","https:\/\/bsky.app\/profile\/sanoscience.bsky.social"]}]}},"acf":[],"gutenberg_blocks":[{"blockName":"custom-styles","attrs":{"styles":""}},{"blockName":"core\/heading","attrs":{"epAnimationGeneratedClass":"edplus_anim-Ds2atf","epGeneratedClass":"eplus-wrapper"},"innerBlocks":[],"innerHTML":"\n<h2 class=\"wp-block-heading eplus-wrapper\">Bart\u0142omiej Przybylski, Maciej Pawlik, Pawe\u0142 \u017buk, Bart\u0142omiej \u0141agosz, Maciej Malawski, Krzysztof Rzadca<\/h2>\n","innerContent":["\n<h2 class=\"wp-block-heading eplus-wrapper\">Bart\u0142omiej Przybylski, Maciej Pawlik, Pawe\u0142 \u017buk, Bart\u0142omiej \u0141agosz, Maciej Malawski, Krzysztof Rzadca<\/h2>\n"]},{"blockName":"core\/spacer","attrs":{"height":"50px","epAnimationGeneratedClass":"edplus_anim-czzMWU","epGeneratedClass":"eplus-wrapper"},"innerBlocks":[],"innerHTML":"\n<div style=\"height:50px\" aria-hidden=\"true\" class=\"wp-block-spacer eplus-wrapper\"><\/div>\n","innerContent":["\n<div style=\"height:50px\" aria-hidden=\"true\" class=\"wp-block-spacer eplus-wrapper\"><\/div>\n"]},{"blockName":"core\/paragraph","attrs":{"epAnimationGeneratedClass":"edplus_anim-wM6Nk9","epGeneratedClass":"eplus-wrapper"},"innerBlocks":[],"innerHTML":"\n<p class=\" eplus-wrapper\">Modern HPC workload managers and their careful tuning contribute to the high utilization of HPC clusters. However, due to inevitable uncertainty it is impossible to completely avoid node idleness. Although such idle slots are usually too short for any HPC job, they are too long to ignore them. Function-as-a-Service (FaaS) paradigm promisingly fills this gap, and can be a good match, as typical FaaS functions last seconds, not hours. Here we show how to build a FaaS infrastructure on idle nodes in an HPC cluster in such a way that it does not affect the performance of the HPC jobs significantly. We dynamically adapt to a changing set of idle physical machines, by integrating open-source software Slurm and OpenWhisk.<br>We designed and implemented a prototype solution that allowed us to cover up to 90\\% of the idle time slots on a 50k-core cluster that runs production workloads.<\/p>\n","innerContent":["\n<p class=\" eplus-wrapper\">Modern HPC workload managers and their careful tuning contribute to the high utilization of HPC clusters. However, due to inevitable uncertainty it is impossible to completely avoid node idleness. Although such idle slots are usually too short for any HPC job, they are too long to ignore them. Function-as-a-Service (FaaS) paradigm promisingly fills this gap, and can be a good match, as typical FaaS functions last seconds, not hours. Here we show how to build a FaaS infrastructure on idle nodes in an HPC cluster in such a way that it does not affect the performance of the HPC jobs significantly. We dynamically adapt to a changing set of idle physical machines, by integrating open-source software Slurm and OpenWhisk.<br>We designed and implemented a prototype solution that allowed us to cover up to 90\\% of the idle time slots on a 50k-core cluster that runs production workloads.<\/p>\n"]},{"blockName":"core\/spacer","attrs":{"height":"50px","epAnimationGeneratedClass":"edplus_anim-czzMWU","epGeneratedClass":"eplus-wrapper"},"innerBlocks":[],"innerHTML":"\n<div style=\"height:50px\" aria-hidden=\"true\" class=\"wp-block-spacer eplus-wrapper\"><\/div>\n","innerContent":["\n<div style=\"height:50px\" aria-hidden=\"true\" class=\"wp-block-spacer eplus-wrapper\"><\/div>\n"]},{"blockName":"acf\/button","attrs":{"title":"READ HERE","button_type":"link","url":"https:\/\/arxiv.org\/abs\/2211.00717","button_style":"primary","target":"_blank","button_extra_classes":""},"innerBlocks":[],"innerHTML":"","innerContent":[]}],"meta_data":{"is_automatically_other_posts":true,"number_of_posts":"3","is_automatically_check_also_posts":true},"_links":{"self":[{"href":"https:\/\/sano.science\/index.php\/wp-json\/wp\/v2\/research\/14927","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sano.science\/index.php\/wp-json\/wp\/v2\/research"}],"about":[{"href":"https:\/\/sano.science\/index.php\/wp-json\/wp\/v2\/types\/research"}],"version-history":[{"count":2,"href":"https:\/\/sano.science\/index.php\/wp-json\/wp\/v2\/research\/14927\/revisions"}],"predecessor-version":[{"id":14929,"href":"https:\/\/sano.science\/index.php\/wp-json\/wp\/v2\/research\/14927\/revisions\/14929"}],"wp:attachment":[{"href":"https:\/\/sano.science\/index.php\/wp-json\/wp\/v2\/media?parent=14927"}],"wp:term":[{"taxonomy":"research_type","embeddable":true,"href":"https:\/\/sano.science\/index.php\/wp-json\/wp\/v2\/research_type?post=14927"},{"taxonomy":"research_team","embeddable":true,"href":"https:\/\/sano.science\/index.php\/wp-json\/wp\/v2\/research_team?post=14927"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}