OBTAINING RATINGS FOR CONTENT ITEMS FROM A GROUP OF USERS WHILE LIMITING A NUMBER OF TIMES THE CONTENT ITEMS ARE PRESENTED TO THE GROUP OF USERS WITHIN A TIME INTERVAL

An online system presents content items to a group of users who provide ratings for the content items. Based on ratings received from various users of the group, the online system generates data describing presentation of the content items to users of the group. Because of a limited number of users in the group, the online system enforces rules that limit the ability to show content items to users of the group within a time interval. Accordingly, when a set of content items are selected for presentation to a user of the group, the online system replaces content items of the set that were previously shown to the user within the time interval with alternative content items. The online system also retrieves a previously received rating for a content item replaced by an alternative content item to use along with ratings received for content items of the set.