Current State-of-the-Art models in Named Entity Recognition (NER) are neural\nmodels with a Conditional Random Field (CRF) as the final network layer, and\npre-trained "contextual embeddings". The CRF layer is used to facilitate global\ncoherence between labels, and the contextual embeddings provide a better\nrepresentation of words in context. However, both of these improvements come at\na high computational cost. In this work, we explore two simple techniques that\nsubstantially improve NER performance over a strong baseline with negligible\ncost. First, we use multiple pre-trained embeddings as word representations via\nconcatenation. Second, we constrain the tagger, trained using a cross-entropy\nloss, during decoding to eliminate illegal transitions. While training a tagger\non CoNLL 2003 we find a $786$\\% speed-up over a contextual embeddings-based\ntagger without sacrificing strong performance. We also show that the\nconcatenation technique works across multiple tasks and datasets. We analyze\naspects of similarity and coverage between pre-trained embeddings and the\ndynamics of tag co-occurrence to explain why these techniques work. We provide\nan open source implementation of our tagger using these techniques in three\npopular deep learning frameworks --- TensorFlow, Pytorch, and DyNet.\n