Despite recent successes of large pre-trained language models in solving\nreasoning tasks, their inference capabilities remain opaque. We posit that such\nmodels can be made more interpretable by explicitly generating interim\ninference rules, and using them to guide the generation of task-specific\ntextual outputs. In this paper we present COINS, a recursive inference\nframework that i) iteratively reads context sentences, ii) dynamically\ngenerates contextualized inference rules, encodes them, and iii) uses them to\nguide task-specific output generation. We apply COINS to a Narrative Story\nCompletion task that asks a model to complete a story with missing sentences,\nto produce a coherent story with plausible logical connections, causal\nrelationships, and temporal dependencies. By modularizing inference and\nsentence generation steps in a recurrent model, we aim to make reasoning steps\nand their effects on next sentence generation transparent. Our automatic and\nmanual evaluations show that the model generates better story sentences than\nSOTA baselines, especially in terms of coherence. We further demonstrate\nimproved performance over strong pre-trained LMs in generating commonsense\ninference rules. The recursive nature of COINS holds the potential for\ncontrolled generation of longer sequences.\n