Systems and methods to achieve load balancing among a plurality of compute elements accessing a shared memory pool

Various systems and methods to achieve load balancing among a plurality of compute elements accessing a shared memory pool. The shared memory pool is configured to store and serve a plurality of data sets associated with a task, a first data interface's internal registry is configured to keep track of which data sets have been extracted from the shared memory pool and served to the compute elements, the first data interface is configured to extract from the shared memory pool and serve to the compute elements data sets which have not yet been extracted and served, the rate at which data sets are extracted and served to each particular compute element is proportional to the rate at which that compute element requests data sets, and the system may continues to extract, serve, and process data sets until all of the data sets associated with the task have been processed once.