DISTRIBUTED PARALLEL COMPUTATION WITH ACCELERATION DEVICES

A method for distributed computing between a host computer and at least one accelerator device interconnected through a network includes profiling a data transfer rate and a computation rate for a range of data sizes to find an optimal chunk size for the data transfer through the network; splitting or aggregating a size of the data stored in a memory in the host computer for encapsulating the data into a chunk with the optimal chunk size; dispatching the encapsulated data to the accelerator device; and instructing pipeline computation to the accelerator device with respect to the encapsulated data received.