The Private Aggregation of Teacher Ensembles (PATE) framework is one of the\nmost promising recent approaches in differentially private learning. Existing\ntheoretical analysis shows that PATE consistently learns any VC-classes in the\nrealizable setting, but falls short in explaining its success in more general\ncases where the error rate of the optimal classifier is bounded away from zero.\nWe fill in this gap by introducing the Tsybakov Noise Condition (TNC) and\nestablish stronger and more interpretable learning bounds. These bounds provide\nnew insights into when PATE works and improve over existing results even in the\nnarrower realizable setting. We also investigate the compelling idea of using\nactive learning for saving privacy budget, and empirical studies show the\neffectiveness of this new idea. The novel components in the proofs include a\nmore refined analysis of the majority voting classifier - which could be of\nindependent interest - and an observation that the synthetic "student" learning\nproblem is nearly realizable by construction under the Tsybakov noise\ncondition.\n