Distributed training in deep learning (DL) is common practice as data and\nmodels grow. The current practice for distributed training of deep neural\nnetworks faces the challenges of communication bottlenecks when operating at\nscale, and model accuracy deterioration with an increase in global batch size.\nPresent solutions focus on improving message exchange efficiency as well as\nimplementing techniques to tweak batch sizes and models in the training\nprocess. The loss of training accuracy typically happens because the loss\nfunction gets trapped in a local minima. We observe that the loss landscape\nminimization is shaped by both the model and training data and propose a data\noptimization approach that utilizes machine learning to implicitly smooth out\nthe loss landscape resulting in fewer local minima. Our approach filters out\ndata points which are less important to feature learning, enabling us to speed\nup the training of models on larger batch sizes to improved accuracy.\n