Language models based on the Transformer architecture have achieved\nstate-of-the-art performance on a wide range of NLP tasks such as text\nclassification, question-answering, and token classification. However, this\nperformance is usually tested and reported on high-resource languages, like\nEnglish, French, Spanish, and German. Indian languages, on the other hand, are\nunderrepresented in such benchmarks. Despite some Indian languages being\nincluded in training multilingual Transformer models, they have not been the\nprimary focus of such work. In order to evaluate the performance on Indian\nlanguages specifically, we analyze these language models through extensive\nexperiments on multiple downstream tasks in Hindi, Bengali, and Telugu\nlanguage. Here, we compare the efficacy of fine-tuning model parameters of\npre-trained models against that of training a language model from scratch.\nMoreover, we empirically argue against the strict dependency between the\ndataset size and model performance, but rather encourage task-specific model\nand method selection. We achieve state-of-the-art performance on Hindi and\nBengali languages for text classification task. Finally, we present effective\nstrategies for handling the modeling of Indian languages and we release our\nmodel checkpoints for the community :\nhttps://huggingface.co/neuralspace-reverie.\n