A Structured Analysis of the Video Degradation Effects on the Performance of a Machine Learning-enabled Pedestrian Detector

ML-enabled software systems have been incorporated in many public\ndemonstrations for automated driving (AD) systems. Such solutions have also\nbeen considered as a crucial approach to aim at SAE Level 5 systems, where the\npassengers in such vehicles do not have to interact with the system at all\nanymore. Already in 2016, Nvidia demonstrated a complete end-to-end approach\nfor training the complete software stack covering perception, planning and\ndecision making, and the actual vehicle control. While such approaches show the\ngreat potential of such ML-enabled systems, there have also been demonstrations\nwhere already changes to single pixels in a video frame can potentially lead to\ncompletely different decisions with dangerous consequences. In this paper, a\nstructured analysis has been conducted to explore video degradation effects on\nthe performance of an ML-enabled pedestrian detector. Firstly, a baseline of\napplying YOLO to 1,026 frames with pedestrian annotations in the KITTI Vision\nBenchmark Suite has been established. Next, video degradation candidates for\neach of these frames were generated using the leading video codecs libx264,\nlibx265, Nvidia HEVC, and AV1: 52 frames for the various compression presets\nfor color and gray-scale frames resulting in 104 degradation candidates per\noriginal KITTI frame and 426,816 images in total. YOLO was applied to each\nimage to compute the intersection-over-union (IoU) metric to compare the\nperformance with the original baseline. While aggressively lossy compression\nsettings result in significant performance drops as expected, it was also\nobserved that some configurations actually result in slightly better IoU\nresults compared to the baseline. The findings show that carefully chosen lossy\nvideo configurations preserve a decent performance of particular ML-enabled\nsystems while allowing for substantial savings when storing or transmitting\ndata.\n