Tuning the hyperparameters in the differentially private stochastic gradient\ndescent (DPSGD) is a fundamental challenge. Unlike the typical SGD, private\ndatasets cannot be used many times for hyperparameter search in DPSGD; e.g.,\nvia a grid search. Therefore, there is an essential need for algorithms that,\nwithin a given search space, can find near-optimal hyperparameters for the best\nachievable privacy-utility tradeoffs efficiently. We formulate this problem\ninto a general optimization framework for establishing a desirable\nprivacy-utility tradeoff, and systematically study three cost-effective\nalgorithms for being used in the proposed framework: evolutionary, Bayesian,\nand reinforcement learning. Our experiments, for hyperparameter tuning in DPSGD\nconducted on MNIST and CIFAR-10 datasets, show that these three algorithms\nsignificantly outperform the widely used grid search baseline. As this paper\noffers a first-of-a-kind framework for hyperparameter tuning in DPSGD, we\ndiscuss existing challenges and open directions for future studies. As we\nbelieve our work has implications to be utilized in the pipeline of private\ndeep learning, we open-source our code at\nhttps://github.com/AmanPriyanshu/DP-HyperparamTuning.\n