Multiple different responses are often plausible for a given open domain\ndialog context. Prior work has shown the importance of having multiple valid\nreference responses for meaningful and robust automated evaluations. In such\ncases, common practice has been to collect more human written references.\nHowever, such collection can be expensive, time consuming, and not easily\nscalable. Instead, we propose a novel technique for automatically expanding a\nhuman generated reference to a set of candidate references. We fetch plausible\nreferences from knowledge sources, and adapt them so that they are more fluent\nin context of the dialog instance in question. More specifically, we use (1) a\ncommonsense knowledge base to elicit a large number of plausible reactions\ngiven the dialog history (2) relevant instances retrieved from dialog corpus,\nusing similar past as well as future contexts. We demonstrate that our\nautomatically expanded reference sets lead to large improvements in\ncorrelations of automated metrics with human ratings of system outputs for\nDailyDialog dataset.\n