High-Speed Depth Sensing With A Hybrid Camera Setup

In a method, a first video stream is received from a video camera, and a second video stream is received from a depth camera. A pixel mapping between the video camera and the depth camera is known. The video camera has an update rate greater than that of the depth camera. Optical flow in successive frames of the first video stream is measured, and a portion of the optical flow attributable to depth change is extracted. A scaling factor is calculated for each pixel in successive frames of the first video stream to determine whether a depth change has occurred. A perspective depth correction is applied to each pixel having a depth change. The perspective depth correction is based upon the depth of the corresponding pixel in the most recent frame from the second video stream. A combined video stream having an update rate of the video camera and depth information from the depth camera is obtained.