SELF-ADAPTIVE BATCH DATASET PARTITIONING FOR DISTRIBUTED DEEP LEARNING USING HYBRID SET OF ACCELERATORS

Systems and methods are provided for implementing a self-adaptive batch dataset partitioning control process which is utilized in conjunction with a distributed deep learning model training process to optimize load balancing among a set of accelerator resources. An iterative batch size tuning process is configured to determine an optimal job partition ratio for partitioning mini-batch datasets into sub-batch datasets for processing by a set of hybrid accelerator resources, wherein the sub-batch datasets are partitioned into optimal batch sizes for processing by respective accelerator resources to minimize a time for completing the deep learning model training process.