RATE DISTORTION OPTIMIZATION ENCODING SYSTEM AND METHOD OF OPERATING THE SAME

An encoding system includes a prediction mode decision unit and an encoding unit. The prediction mode decision unit is configured to receive a target image and decide an optimized mode by applying rate-distortion optimization (RDO) to n (n is a natural number) modes of the target image. The encoding unit is configured to encode the target image according to the decided mode. The prediction mode decision unit includes a distortion calculation unit configured to calculate distortion by mean squared error (MSE) calculation between the target image and the n modes, a Lagrangian multiplier calculation unit configured to calculate a Lagrangian multiplier based on multi-scale structural similarity (MS-SSIM) calculation, and a bitrate calculation unit configured to calculate a bit rate of the n modes.