Patent US 6,749,435

A method and system for managing evaluators of data items to maintain consistent and efficient scoring of data items. Collaborative scoring allows two evaluators to score the same data item and resolves potential discrepancies in the scores provided by the evaluators. Quality items monitor a evaluator's scoring performance against a known standard. A quality-control data item, having a known score, is transmitted to a evaluator. After receiving a score from the evaluator, the known score is compared with the evaluator's score and a quality profile is generated for the evaluator.