End-to-end models that condition the output label sequence on all previously\npredicted labels have emerged as popular alternatives to conventional systems\nfor automatic speech recognition (ASR). Since unique label histories correspond\nto distinct models states, such models are decoded using an approximate\nbeam-search process which produces a tree of hypotheses.\n In this work, we study the influence of the amount of label context on the\nmodel's accuracy, and its impact on the efficiency of the decoding process. We\nfind that we can limit the context of the recurrent neural network transducer\n(RNN-T) during training to just four previous word-piece labels, without\ndegrading word error rate (WER) relative to the full-context baseline. Limiting\ncontext also provides opportunities to improve the efficiency of the\nbeam-search process during decoding by removing redundant paths from the active\nbeam, and instead retaining them in the final lattice. This path-merging scheme\ncan also be applied when decoding the baseline full-context model through an\napproximation. Overall, we find that the proposed path-merging scheme is\nextremely effective allowing us to improve oracle WERs by up to 36% over the\nbaseline, while simultaneously reducing the number of model evaluations by up\nto 5.3% without any degradation in WER.\n