Intelligent Partitioning Engine for Cluster Computing

An intelligent data partitioning engine processes instructions to monitor an input queue of a cluster computing framework processing on a distributed computing system. The intelligent data partitioning engine calculates data requirements for processing of one or more program files in the input queue and determines, based on a block size and available processing resources of a plurality of nodes of the distributed computing system, a number of data partitions. Based on the data partitions, the intelligent data partitioning engine triggers execution of the one or more program files by the cluster computing framework, where the cluster computing framework is configured based on the block size and the number of data partitions and updates the data requirements for processing of the one or more program files based on feedback from the cluster computing framework corresponding to one or more previous processing runs of the one or more program files.