We consider the best-of-both-worlds problem for learning an episodic Markov\nDecision Process through $T$ episodes, with the goal of achieving\n$\\widetilde{\\mathcal{O}}(\\sqrt{T})$ regret when the losses are adversarial and\nsimultaneously $\\mathcal{O}(\\text{polylog}(T))$ regret when the losses are\n(almost) stochastic. Recent work by [Jin and Luo, 2020] achieves this goal when\nthe fixed transition is known, and leaves the case of unknown transition as a\nmajor open question. In this work, we resolve this open problem by using the\nsame Follow-the-Regularized-Leader ($\\text{FTRL}$) framework together with a\nset of new techniques. Specifically, we first propose a loss-shifting trick in\nthe $\\text{FTRL}$ analysis, which greatly simplifies the approach of [Jin and\nLuo, 2020] and already improves their results for the known transition case.\nThen, we extend this idea to the unknown transition case and develop a novel\nanalysis which upper bounds the transition estimation error by (a fraction of)\nthe regret itself in the stochastic setting, a key property to ensure\n$\\mathcal{O}(\\text{polylog}(T))$ regret.\n