Tracking entities throughout a procedure described in a text is challenging\ndue to the dynamic nature of the world described in the process. Firstly, we\npropose to formulate this task as a question answering problem. This enables us\nto use pre-trained transformer-based language models on other QA benchmarks by\nadapting those to the procedural text understanding. Secondly, since the\ntransformer-based language models cannot encode the flow of events by\nthemselves, we propose a Time-Stamped Language Model~(TSLM model) to encode\nevent information in LMs architecture by introducing the timestamp encoding.\nOur model evaluated on the Propara dataset shows improvements on the published\nstate-of-the-art results with a $3.1\\%$ increase in F1 score. Moreover, our\nmodel yields better results on the location prediction task on the NPN-Cooking\ndataset. This result indicates that our approach is effective for procedural\ntext understanding in general.\n