Modelling and Prediction of Resource Utilization of Hadoop Clusters: A Machine Learning Approach

Abstract

Hadoop is a distributed computing framework that has a large number of configurable parameters. These parameters have impact on system resources and execution time. Optimizing the performance of a Hadoop cluster by tuning such a large number of parameters is a tedious task. Most current big data modeling approaches does not include complex interaction between configuration parameters and the cluster environment changes such as different datasets or query. This makes it difficult to predict the performance or resource utilization of a cluster when we use real-world datasets because of their size and content. This paper presents the modeling of resource utilization of Hadoop cluster on the basis of Hadoop configuration parameters and dataset structure. Our approach builds a machine learning based-model using Hive-based Hadoop query and then predict the outcome for a particular parameter setting and query type. We used decision trees to build models for each of our performance metric measures. Decision rules were extracted from these tree-based models and evaluated for their ability to generalize to unseen data. Our experiments predicted that the percentage of columns selected, mappers and replica has a statistically significant impact over the utilization of different resources in Hadoop cluster.