Object-Based Color Detection and Correction

A system may utilize a projector, a camera, and a depth sensor to produce images within the environment of a user and to detect and respond to user actions. Depth data from the depth sensor may be analyzed to detect and identify items within the environment. A coordinate transformation may then be used to identify corresponding color values from camera data, which can in turn be analyzed to determine the colors of detected items. A similar coordinate transformation may be used to identify color values of a projected image that correspond to the detected items. In some cases, camera color values corresponding to an item may be corrected based on the corresponding color values of a projected image. In other cases, projected color values corresponding to an item may be corrected based on the corresponding camera color values.