We evaluate named entity representations of BERT-based NLP models by\ninvestigating their robustness to replacements from the same typed class in the\ninput. We highlight that on several tasks while such perturbations are natural,\nstate of the art trained models are surprisingly brittle. The brittleness\ncontinues even with the recent entity-aware BERT models. We also try to discern\nthe cause of this non-robustness, considering factors such as tokenization and\nfrequency of occurrence. Then we provide a simple method that ensembles\npredictions from multiple replacements while jointly modeling the uncertainty\nof type annotations and label predictions. Experiments on three NLP tasks show\nthat our method enhances robustness and increases accuracy on both natural and\nadversarial datasets.\n