There are many existing retrieval and question answering datasets. However,\nmost of them either focus on ranked list evaluation or single-candidate\nquestion answering. This divide makes it challenging to properly evaluate\napproaches concerned with ranking documents and providing snippets or answers\nfor a given query. In this work, we present FiRA: a novel dataset of\nFine-Grained Relevance Annotations. We extend the ranked retrieval annotations\nof the Deep Learning track of TREC 2019 with passage and word level graded\nrelevance annotations for all relevant documents. We use our newly created data\nto study the distribution of relevance in long documents, as well as the\nattention of annotators to specific positions of the text. As an example, we\nevaluate the recently introduced TKL document ranking model. We find that\nalthough TKL exhibits state-of-the-art retrieval results for long documents, it\nmisses many relevant passages.\n