AdaptiveISP: Learning an Adaptive Image Signal Processor for Object Detection

Summary

Image Signal Processors (ISP) are software pipelines that aim to improve images for their visual quality or application-dependent downstream tasks. This work presents AdaptiveISP, a method to simultaneously optimize an ISP pipeline, consisting of individual functions such as image sharpening or color correction and the functions' parameters. Learning policies that map an image into the optimal ISP structure and parameters while considering computation cost and the specific downstream task improves performance compared to the prior art and allows for real-time application, adapting to newly shot images in time.

Strengths

- When creating novel ML methods, it is important to consider not only accuracy on pre-recorded datasets but also how these methods can be applied in the field, including adaptations of parameters and cost of computation. This work tackles these questions concerning image signal processing tasks, a significant endeavor for any robotic system equipped with a camera. - The presented method incorporates the specific downstream task, e.g., object detection, together with an adaptive trade-off for computation time, providing a comprehensive framework that has been presented with high clarity and incorporates a good level of originality, e.g., not aiming for visual clarity but seeking to obtain images best suited for the employed network.

Weaknesses

- Since real-time applicability is presented as a key motivation for the paper, this work could benefit from an expanded evaluation of the method's computational flexibility. For example, Table 3 shows the average running time for two settings of the $\lambda_c$ parameter. It would be more informative to see the results for a range of values for $\lambda_c$ instead, not just showing that the expected impact is achieved but what the limitations and behavior of the system are when trying to tune for accuracy or speed. Similarly, one could experimentally consider how the method's time and memory demands change with varying pools of ISP modules to choose from.

Questions

- Figure 1 shows how different ISP modules are placed in the pipeline to modify the image's color values. Have you considered further calibration steps to be part of the adaptive pipeline, e.g., corrections for fish eye effects and other optical deficiencies?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have adequately addressed the limitations.

Summary

This paper proposes AdaptiveISP, a task-driven and scene-adaptive ISP, which uses deep reinforcement learning to automatically generate an optimal ISP pipeline and associated ISP parameters, aiming to maximize the detection performance. Experimental results show that AdaptiveISP outperforms prior state-of-the-art methods for object detection, and it effectively manages the trade-off between detection performance and computational cost, demonstrating its effectiveness in dynamic scenes.

Strengths

1. The AdaptiveISP pipeline combines ISP and detection together and turns the fixed process into a task-oriented tuning problem, which demonstrates greater potential for specific tasks. 2. The results of the algorithm on some datasets look good.

Weaknesses

1. The experiments are performed only on YOLO detectors. The conclusion and findings could be more solid and convincing if experiments are available on other architectures. 2. It would be better to analyze and validate the generalization ability with some 3rd-party datasets.

Questions

1. The tuning process could be costly when applied to real scenes since there are more complex factors in real scenes than in the experimental datasets. For example, the light condition can change rapidly in urban scenes during the nighttime. It is doubtful whether adaptiveISP can react to the change. 2. Is it stable to tune adaptiveISP using RL? Will the tuning lead to even worse results than traditional ISPs? How can you evaluate the risk of your work in applications.?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

See weakness

Summary

This paper proposes a novel approach to image signal processing (ISP) specifically tailored for object detection tasks, leveraging deep reinforcement learning to optimize both ISP structures and parameters. This method dynamically adjusts the ISP pipeline in response to different scene requirements, which enhances detection performance.

Strengths

1. The figures in this paper are of good quality and easy to understand. 2. AdaptiveISP can dynamically adjust the ISP pipeline according to different input images to adapt to different scene changes.

Weaknesses

1. The system's performance heavily relies on the quality of the pre-trained object detection models. There is a potential limitation in cases where these models do not generalize well or when transitioning to different object detection tasks that were not part of the initial training set. 2. The challenges and contributions are too general and not prepared objectively. It should briefly highlight the paper's novelty as what is the main problem, how has it been resolved and where the novelty lies? 3. Although the paper conducted experiments on multiple datasets, the limited diversity and coverage of these datasets may not be sufficient to fully validate the performance of AdaptiveISP in various real-world application scenarios. For example, there is a lack of testing on the DAWN dataset, a dataset that covers multiple weather scenarios and is well-suited for dynamic testing.

Questions

The author mentions "Our method only takes 1.2 ms per stage during inference" , what does "per stage" mean, is it the end-to-end inference time for each image?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

YOLOv3 is an older model, how does the method in this paper perform under newer YOLO models or other models? It is suggested that the authors add other models to the experiment.

Summary

A new perspective of designing ISP pipeline. Good results with some problems should be addressed.

Strengths

1/ One method for raw detection which is still a new subarea waiting more discovery. 2/ Good performance compared with some methods. 3/ Discuss some orders in ISP pipeline.

Weaknesses

1/ Discussion on related works especially for ISP pipeline is not sufficient. Not only the ISP for task performance [1], but also for image quality [2,3]. 2/ The datasets used for comparison are not actually real raw detection data. It is doubtful whether the real scene performance is good. 3/ Also, the compared methods are for designing special ISP params or orders to get better performance on raw downstream tasks. However, what about using an existing ISP and detect on RGB images? For datasets such as LOD and COCO can do it. The results must be better than existing RGB detection SOTAs, so that this work can be meaningful. 4/ The order of ISP modules is still not fixed. There have been many works finding that it can be various and change its order according to tasks or image quality. Also, the pipelines can be different according to manufacturers. 5/ Some new benchmarks such as [1] should be used for evaluation. [1] Ruikang Xu, Chang Chen, Jingyang Peng, Cheng Li, Yibin Huang, Fenglong Song, Youliang Yan, Zhiwei Xiong: Toward RAW Object Detection: A New Benchmark and A New Model. CVPR 2023: 13384-13393 [2] Woohyeok Kim, Geonu Kim, Junyong Lee, Seungyong Lee, Seung-Hwan Baek, Sunghyun Cho: ParamISP: Learned Forward and Inverse ISPs using Camera Parameters. CoRR abs/2312.13313 (2023) [3] Syed Waqas Zamir, Aditya Arora, Salman H. Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, Ling Shao: CycleISP: Real Image Restoration via Improved Data Synthesis. CVPR 2020: 2693-2702

Questions

Please refer to the weaknesses section.

Rating

3

Confidence

5

Soundness

2

Presentation

2

Contribution

2

Limitations

1/ Lack of new benchmark, new methods. 2/ Must compare with RGB SOTAs.

After Rebuttal

The rebuttal has addressed most of my concerns. I will keep my initial rating. I strongly recommend the authors to add more discussions on image quality tasks, as pointed out by Reviewer Ln32, which is particularly important for an ISP.