With the surge in user-generated textual information, there has been a recent\nincrease in the use of summarization algorithms for providing an overview of\nthe extensive content. Traditional metrics for evaluation of these algorithms\n(e.g. ROUGE scores) rely on matching algorithmic summaries to human-generated\nones. However, it has been shown that when the textual contents are\nheterogeneous, e.g., when they come from different socially salient groups,\nmost existing summarization algorithms represent the social groups very\ndifferently compared to their distribution in the original data. To mitigate\nsuch adverse impacts, some fairness-preserving summarization algorithms have\nalso been proposed. All of these studies have considered normative notions of\nfairness from the perspective of writers of the contents, neglecting the\nreaders' perceptions of the underlying fairness notions. To bridge this gap, in\nthis work, we study the interplay between the fairness notions and how readers\nperceive them in textual summaries. Through our experiments, we show that\nreader's perception of fairness is often context-sensitive. Moreover, standard\nROUGE evaluation metrics are unable to quantify the perceived (un)fairness of\nthe summaries. To this end, we propose a human-in-the-loop metric and an\nautomated graph-based methodology to quantify the perceived bias in textual\nsummaries. We demonstrate their utility by quantifying the (un)fairness of\nseveral summaries of heterogeneous socio-political microblog datasets.\n