Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QA

Previous work finds that recent long-context language models fail to make\nequal use of information in the middle of their inputs, preferring pieces of\ninformation located at the tail ends which creates an undue bias in situations\nwhere we would like models to be equally capable of using different parts of\nthe input. Thus far, the problem has mainly only been considered in settings\nwith single pieces of critical information, leading us to question what happens\nwhen multiple necessary pieces of information are spread out over the inputs.\nHere, we demonstrate the effects of the "lost in the middle" problem in the\nmulti-hop question answering setting -- in which multiple reasoning "hops" over\ndisconnected documents are required -- and show that performance degrades not\nonly with respect to the distance of information from the edges of the context,\nbut also between pieces of information. Additionally, we experiment with means\nof alleviating the problem by reducing superfluous document contents through\nknowledge graph triple extraction and summarization, and prompting models to\nreason more thoroughly using chain-of-thought prompting.\n