OPTIMIZING GRADIENT BOOSTING FEATURE SELECTION

Gradient Boosting Decision Tree (GBDT) successively stacks many decision trees which at each step try to fix the residual errors from the previous steps. The final score produced by the GBDT is the sum of the individual scores obtained by the decision trees for an input vector. Overfitting in GBDT can be reduced by removing the input values that have the least impact on the output from the training data. One way to determine which input variable has the lowest predictive value is to determine the input variable that is used for the first time in the latest decision tree in the GBDT. This method of identifying the low-predictive features to be removed does not require that earlier trees be regenerated to generate the new GBDT. Since the removed feature was already not used in the earlier trees, those trees already ignore the removed feature.