We propose a simple model selection approach for algorithms in stochastic\nbandit and reinforcement learning problems. As opposed to prior work that\n(implicitly) assumes knowledge of the optimal regret, we only require that each\nbase algorithm comes with a candidate regret bound that may or may not hold\nduring all rounds. In each round, our approach plays a base algorithm to keep\nthe candidate regret bounds of all remaining base algorithms balanced, and\neliminates algorithms that violate their candidate bound. We prove that the\ntotal regret of this approach is bounded by the best valid candidate regret\nbound times a multiplicative factor. This factor is reasonably small in several\napplications, including linear bandits and MDPs with nested function classes,\nlinear bandits with unknown misspecification, and LinUCB applied to linear\nbandits with different confidence parameters. We further show that, under a\nsuitable gap-assumption, this factor only scales with the number of base\nalgorithms and not their complexity when the number of rounds is large enough.\nFinally, unlike recent efforts in model selection for linear stochastic\nbandits, our approach is versatile enough to also cover cases where the context\ninformation is generated by an adversarial environment, rather than a\nstochastic one.\n