Imagine you are in a supermarket. You have two bananas in your basket and\nwant to buy four apples. How many fruits do you have in total? This seemingly\nstraightforward question can be challenging for data-driven language models,\neven if trained at scale. However, we would expect such generic language models\nto possess some mathematical abilities in addition to typical linguistic\ncompetence. Towards this goal, we investigate if a commonly used language\nmodel, BERT, possesses such mathematical abilities and, if so, to what degree.\nFor that, we fine-tune BERT on a popular dataset for word math problems,\nAQuA-RAT, and conduct several tests to understand learned representations\nbetter. Since we teach models trained on natural language to do formal\nmathematics, we hypothesize that such models would benefit from training on\nsemi-formal steps that explain how math results are derived. To better\naccommodate such training, we also propose new pretext tasks for learning\nmathematical rules. We call them (Neighbor) Reasoning Order Prediction (ROP or\nNROP). With this new model, we achieve significantly better outcomes than\ndata-driven baselines and even on-par with more tailored models. We also show\nhow to reduce positional bias in such models.\n