This work tackles the fidelity objective in the perceptual super-resolution (SR) task. Specifically, we address the shortcomings of pixel-level ℒp loss (ℒpix) in the GAN-based SR framework. Since ℒpix is known to have a trade-off relationship against perceptual quality, prior methods often multiply a small scale factor or utilize low-pass filters. However, this work shows that these circumventions fail to address the fundamental factor that induces blurring. Accordingly, we focus on two points: 1) precisely discriminating the subcomponent of ℒpix that contributes to blurring, and 2) only guiding based on the factor that is free from this trade-off relationship. We show that they can be achieved in a surprisingly simple manner, with an Auto-Encoder (AE) pretrained with ℒpix. Accordingly, we propose the Auto-Encoded Supervision for Optimal Penalization loss (ℒAESOP), a novel loss function that measures distance in the AE space 1, instead of the raw pixel space. By simply substituting ℒpix with ℒAESOP, we can provide effective reconstruction guidance without compromising perceptual quality. Designed for simplicity, our method enables easy integration into existing SR frameworks. Extensive experiments demonstrate the effectiveness of AESOP.