PROBABILISTIC MULTIGRAPH MODELING FOR IMPROVING THE QUALITY OF CROWDSOURCED AFFECTIVE DATA

A method of improving quality of crowdsourced affective data based on agreement relationship between a plurality of annotators include receiving, by a processor, a collection of stimuli previously given affective labels by the plurality of annotators, executing, by a processor, an algorithm operative to perform the steps including constructing an agreement multigraph as a probabilistic model including a pair-wise status of agreement between the affective labels given by different ones of the plurality of annotators, learning the probabilistic model computationally using the crowdsourced affective data, identifying a reliability of each of the plurality of annotators based on the learned model, and adjusting the crowdsourced affective data by calculating the affective labels of each stimuli based on the identified reliability of each of the plurality of annotators, thereby improving the quality of the crowdsourced affective data.