We consider three different variants of differential privacy (DP), namely\napproximate DP, R\\'enyi DP (RDP), and hypothesis test DP. In the first part, we\ndevelop a machinery for optimally relating approximate DP to RDP based on the\njoint range of two $f$-divergences that underlie the approximate DP and RDP. In\nparticular, this enables us to derive the optimal approximate DP parameters of\na mechanism that satisfies a given level of RDP. As an application, we apply\nour result to the moments accountant framework for characterizing privacy\nguarantees of noisy stochastic gradient descent (SGD). When compared to the\nstate-of-the-art, our bounds may lead to about 100 more stochastic gradient\ndescent iterations for training deep learning models for the same privacy\nbudget. In the second part, we establish a relationship between RDP and\nhypothesis test DP which allows us to translate the RDP constraint into a\ntradeoff between type I and type II error probabilities of a certain binary\nhypothesis test. We then demonstrate that for noisy SGD our result leads to\ntighter privacy guarantees compared to the recently proposed $f$-DP framework\nfor some range of parameters.\n