A HPC Co-scheduler with Reinforcement Learning

High Performance Computing (HPC) datacenters process thousands of diverse applications, supporting many scientific and business endeavours. Although users understand minimum coarse resource job req ...

Computer Science Engineering

View graph Source

Paper

References (45)

01Waiting Game: Optimally Provisioning Fixed Resources for Cloud-Enabled SchedulersPradeep Ambati, Noman Bashir, D. Irwin et al.2020 · International Conference for High Performance Computing, Networking, Storage and Analysis · 21 citations

02ASA - The Adaptive Scheduling ArchitectureAbel Souza, K. Pelckmans, D. Ghoshal et al.2020 · IEEE International Symposium on High-Performance Parallel Distributed Computing · 3 citations

03Borg: the next generationMuhammad Tirmazi, A. Barker, Nan Deng et al.2020 · European Conference on Computer Systems · 411 citations

04uPredict: A User-Level Profiler-Based Predictive Framework in Multi-Tenant CloudsHamidreza Moradi, Wei Wang, Amanda Fernandez et al.2020 · 2020 IEEE International Conference on Cloud Engineering (IC2E) · 6 citations

05Dynamic Colocation Policies with Reinforcement LearningYuhao Li, Dan Sun, Benjamin C. Lee2020 · ACM Transactions on Architecture and Code Optimization (TACO) · 17 citations

06AI for ScienceR. Stevens, V. Taylor, Jeffrey A. Nichols et al.2020 · 41 citations

07CLITE: Efficient and QoS-Aware Co-Location of Multiple Latency-Critical Jobs for Warehouse Scale ComputersTirthak Patel, Devesh Tiwari2020 · International Symposium on High-Performance Computer Architecture · 175 citations

08RLScheduler: Learn to Schedule HPC Batch Jobs Using Deep Reinforcement LearningDi Zhang, Dong Dai, Youbiao He et al.2019 · arXiv.org · 9 citations In Library

09Hugo: A Cluster Scheduler that Efficiently Learns to Select Complementary Data-Parallel JobsL. Thamsen, I. Verbitskiy, S. Nedelkoski et al.2019 · Euro-Par Workshops · 6 citations

10Speculative Scheduling for Stochastic HPC ApplicationsAna Gainaru, Guillaume Pallez, Hongyang Sun et al.2019 · International Conference on Parallel Processing · 14 citations

11Learning scheduling algorithms for data processing clustersHongzi Mao, Malte Schwarzkopf, Shaileshh Bojja Venkatakrishnan et al.2018 · Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication · 816 citations In Library

12User Estimates Inaccuracy Study in HPC SchedulerMariusz Uchroński, W. Bożejko, Z. Krajewski et al.2018 · International Conference on Dependability of Computer Systems · 4 citations

Scroll for more · 33 remaining

Similar papers

RLScheduler: An Automated HPC Batch Job Scheduler Using Reinforcement LearningDi Zhang, Dong Dai, Youbiao He et al.2020 · International Conference for High Performance Computing, Networking, Storage and Analysis · 90 citations In Library

RLScheduler: An Automated HPC Batch Job Scheduler Using Reinforcement LearningDi Zhang, Dong Dai, Youbiao He et al.2019 · arXiv.org · 9 citations In Library

A Deep Reinforcement Learning Scheduler with Back-filling for High Performance ComputingLingfei Wang, Aaron Harwood, Maria A. Rodriguez2021 · 2021 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) In Library

Deep Reinforcement Agent for Scheduling in HPCYuping Fan, Z. Lan, T. Childers et al.2021 · IEEE International Parallel and Distributed Processing Symposium · 61 citations In Library

DRAS: Deep Reinforcement Learning for Cluster Scheduling in High Performance ComputingYuping Fan, Boyang Li, Dustin Favorite et al.2022 · IEEE Transactions on Parallel and Distributed Systems · 34 citations In Library

IRLS: An Improved Reinforcement Learning Scheduler for High Performance Computing SystemsThanh Hoang Le Hai, L. Dinh, Dat Ngo Tien et al.2023 · International Conference on System Science and Engineering · 2 citations In Library

A Reinforcement Learning Based Backfilling Strategy for HPC Batch JobsElliot Kolker-Hicks, Di Zhang, Dong Dai2023 · SC Workshops · 17 citations In Library

DRAS-CQSim: A Reinforcement Learning based Framework for HPC Cluster SchedulingYuping Fan, Z. Lan2021 · Softw. Impacts · 22 citations In Library

Hyper-Heuristic Task Scheduling Algorithm Based on Reinforcement Learning in Cloud ComputingLei Yin, Chang Sun, Ming Gao et al.2023 · Intelligent Automation and Soft Computing · 8 citations In Library

Data Centers Job Scheduling with Deep Reinforcement LearningSisheng Liang, Zhou Yang, Fang Jin et al.2019 · Pacific-Asia Conference on Knowledge Discovery and Data Mining · 35 citations In Library

Patents near this paper

HIGH PERFORMANCE COMPUTING AS A SERVICEUS 10,089,147 · INTERNATIONAL BUSINESS MACHINES CORPORATION · 2018 AUTOMATIC RECONFIGURATION OF HIGH PERFORMANCE COMPUTING JOB SCHEDULERS BASED ON USER BEHAVIOR, USER FEEDBACK, AND JOB PERFORMANCE MONITORINGUS 10,331,482 · INTERNATIONAL BUSINESS MACHINES CORPORATION · 2019 AUTOMATIC RECONFIGURATION OF HIGH PERFORMANCE COMPUTING JOB SCHEDULERS BASED ON USER BEHAVIOR, USER FEEDBACK, AND JOB PERFORMANCE MONITORINGUS 10,331,481 · INTERNATIONAL BUSINESS MACHINES CORPORATION · 2019 PERFORMANCE-AWARE JOB SCHEDULING UNDER POWER CONSTRAINTSUS 9,189,273 · INTERNATIONAL BUSINESS MACHINES CORPORATION · 2015 JOB SCHEDULING USING REINFORCEMENT LEARNINGUS 11,803,999 · ADVANCED MICRO DEVICES, INC. · 2023

Source of record

arXiv