Modern deep learning (DL) applications are built using DL libraries and\nframeworks such as TensorFlow and PyTorch. These frameworks have complex\nparameters and tuning them to obtain good training and inference performance is\nchallenging for typical users, such as DL developers and data scientists.\nManual tuning requires deep knowledge of the user-controllable parameters of DL\nframeworks as well as the underlying hardware. It is a slow and tedious\nprocess, and it typically delivers sub-optimal solutions.\n In this paper, we treat the problem of tuning parameters of DL frameworks to\nimprove training and inference performance as a black-box optimization problem.\nWe then investigate applicability and effectiveness of Bayesian optimization\n(BO), genetic algorithm (GA), and Nelder-Mead simplex (NMS) to tune the\nparameters of TensorFlow's CPU backend. While prior work has already\ninvestigated the use of Nelder-Mead simplex for a similar problem, it does not\nprovide insights into the applicability of other more popular algorithms.\nTowards that end, we provide a systematic comparative analysis of all three\nalgorithms in tuning TensorFlow's CPU backend on a variety of DL models. Our\nfindings reveal that Bayesian optimization performs the best on the majority of\nmodels. There are, however, cases where it does not deliver the best results.\n