SHUFFLE OPTIMIZATION IN MAP-REDUCE PROCESSING

A computer implemented method provides shuffle optimization in map-reduce processing. The computer implemented method obtains intermediate results from a plurality of mappers for an application on a computing device. The intermediate results are combined from the plurality of mappers and are then partitioned into intermediate results for respective reducers for the application. The intermediate results for the respective reducers are stored into a buffer storage, which is partitioned into buffer sections for the respective reducers. Based on a rate of increase in the intermediate results, the method predicts a time at which a data capture notification shall be sent. The method sends the data capture notification based on a predicted time being reached, to indicate that the intermediate results for respective reducers stored in the buffer storage can be transmitted to the respective reducers, and transmits the intermediate results for the respective reducers to the respective reducers.