Conditional Controllable Image Fusion

Summary

The paper proposes a controllable conditional image fusion method. This method enables dynamically controllable fusion for each image pairs. The core idea is to empirically construct a conditional bank and dynamically select different control conditions during the diffusion fusion process. The method is suitable for image fusion tasks of different modalities and can also perform controlled fusion given various downstream tasks, such as object detection.

Strengths

- The paper introduces the controllable conditional image fusion algorithm for the first time. This method makes the image fusion process based on the diffusion model controllable through manual feature constraints (empirical). - This method is suitable for different fusion tasks.

Weaknesses

- The section Sampling-adaptive Condition Selection (SCS) is mainly derived from "GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks", which is cited in the paper, but some of the formulas are not clearly expressed. For example, in Eq 11, Gate(C) has C on the left, but the condition C is not contained on the right. Another example is Line 170, Li(t) = E[wi(t)Li(t)]? Additionally, there are some typos, like Line 170, theta. Therefore, understanding the routing process without reading the GradNorm paper might be difficult. - There are some issues with the writing and formatting, such as the order of appearance of Tab1 and Tab2, and the captions for the tables do not specify the task for each tab (e.g., Multi-Focus, Multi-Modal, etc.) in the quantitative comparison results. The captions for the images also have this issue and need improvement. - There are too many empirical conditions, which may require different settings under various circumstances.

Questions

- In Fig1, the terms “Random”, “Content”, and “Detail” in the center of the diagram need to be clearly explained. - In Fig2, Top-K Routing is not contained in the main text. - In Fig3, the enlarged section does not seem to be as good as methods like Swinfusion and Text-IF. - Line 193, “the feature extracted by a detection network F = D(x)”—what scale/level of feature serves as a good guide? Are there any selection criteria? - For basic conditions, are the high-frequency information of both modality images (for example, infrared and visible images) extracted? - With so many conditions, low-frequency doesn’t seem as important as shown in Fig.1. Additionally, are there any other empirical conditions with potential? - In Fig8, the authors mention the network structure, but in Line 493, they state that they used “a pre-trained diffusion model.” So, this network structure should be derived from the paper on the pre-trained model? If so, a citation needs to be provided in the diagram.

Rating

5

Confidence

5

Soundness

3

Presentation

2

Contribution

4

Limitations

- The conditional bank is manually designed, and different experiments likely require different fine-tuning. This limits the potential of the proposed method. - It is necessary to specify the runtime, as time is usually a key metric for diffusion model-based methods.

Summary

This paper proposes a novel Controllable Condition Fusion (CCF) framework that utilizes a pre-trained diffusion model to achieve dynamic and adaptive condition selection without requiring specific training for general image fusion tasks. The authors presented a conditional bank conducted by various conditions to fit diverse scenarios. The conditions can be dynamically and adaptively selected according to the diffusion step, allowing CCF to conditionally calibrate the fused images step by step for each individual sample. Experimental results demonstrate the effectiveness of the proposed method.

Strengths

(1)The paper is well-structured and the proposed idea is novel. It introduces a conditional controllable fusion (CCF) framework for general image fusion tasks without specific training. The generation and denoising capabilities of DDPM are effectively leveraged to produce high-quality fused images, making its integration with image fusion both ingenious and well-suited. (2)A new dynamic conditional paradigm was introduced with a conditional bank that regulates image fusion. This allows for the adaptive selection of multiple conditions, enabling control over various image fusion processes. (3)The paper validates the superior performance of the CCF framework over SOTA methods through extensive experiments on various fusion tasks. This enhances the credibility and applicability of the proposed approach.

Weaknesses

(1)This paper introduces a controllable condition image fusion model. However, this ‘condition’ is not the common guidance condition, such as sketch, location map, pose image, etc, but evaluation metrics, such as SSIM, Edge Intensity et al. Therefore, suggesting that authors give a clear explanation or definition of the condition in Section Introduction. (2)An important comparison is missing. From the first impression, the proposed method is to leverage the reconstruction capability of DDPM. However, the article did not demonstrate that the fusion process utilized the reconstruction capability of DDPM, it is important to prove it systematically. (3)One of the core innovations in this paper is the Conditional Bank, which contains three types of conditions: basic fusion conditions, enhanced fusion conditions, and task-specific fusion conditions. However, there lacks a clear definition of three conditions to identify the difference between these conditions. (4)The paper contains some small mistakes in symbol, such as Line 170 theta, the lack of a subscript for eq 6 \epsilon and missing the parentheses. Additionally, there is indistinct use of "x0|t" and "x0" in Line 134. Overall, the work is very readable, but the problems with the writing should be corrected.

Questions

Please refer to the weaknesses above.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

The authors have discussed the limitations of this work. And I don’t think there are any direct negative societal implications. Other limitations and opportunities for improvement are addressed in my responses to previous questions.

Summary

This paper proposes a diffusion-based image fusion method with adaptive fusion conditions. It aims to solve the drawback of existing method, i.e., the application of distinct constraint designs tailored to specific scenes. This method builds a condition bank with basic, enhanced, and task-specific conditions. Then, it employs specific fusion constraints based on these conditions for each individual in practice. The proposed method is tested on various image fusion tasks, including multi-modal, multi-exposure, and multi-focus image fusion.

Strengths

1. The proposed method can combine the advantages of multiple types of loss functions and take consideration of downstream tasks, thus dynamically adjust the optimization direction during the sampling process. 2. This method is applicable to multiple image fusion tasks, including visible-infrared, medical, multi-exposure, and multi-focus image fusion. 3. The experiments are rich and the results are competitive.

Weaknesses

1. The basic and enhanced conditions are some widely used image fusion constraints, and the task-specific condition is based on the feature extracted by a downstream network. Thus, the main contribution lies in the design of the gate of conditions. However, the reason why is the gate defined in this way and the principle behind this definition not fully explained in detail. 2. It focuses on the problem that such data-driven fusion methods are hardly applicable to all scenarios, especially in rapidly changing environments and source images with dynamic differences. However, the experiment fails to show the advantages in these scenarios. 3. This method states that multiple conditions can be considered, but the actual considerations are limited, such as the SSIM-based enhanced condition and detection-based task-specific condition. 4. The article basically describes the process of the method, but the motivation, principle, and details of the method are not clear and explicit enough.

Questions

1. What does $s_\theta(x_t,t)$ in Eq. (10) represent? And what is the form of $C$ in Eq. (10)? As mentioned in the multiple types of conditions, some conditions are in pixel level, such as MSE, SSIM, feature similarity, while some is denoted with the characteristic, such as SD. The form is not unified, and the characteristic-based condition may have some effect on the content. 2. In the specific conditions, how to build the constraint between F(x_0) and F(V,I), as the first term takes a single image as input while the last term take the two source images as the input. 3. For the task-specific condition, the extracted features with the task network is used for constraint. Why not use the final performance of the task for constraint?

Rating

5

Confidence

5

Soundness

3

Presentation

2

Contribution

2

Limitations

1. The first and second contributions are essentially the same. 2. The conditions discussed in the Appendix are almost different basic conditions. And the results of MSE are a bit strange. 3. It lacks the experiment on challenging and dynamic scenarios.

Summary

This paper proposes a Conditional Controllable Fusion framework called CCF, effectively addressing the issue that existing data-driven fusion methods struggle to adapt to all scenarios. The authors conducted extensive experiments to demonstrate the effectiveness of the CCF. This manuscript is standardized and the writing is fluent.

Strengths

1. The idea of conditional controllable image fusion (CCF) is novel. 2. Extensive experiments are conducted to demonstrate the effectiveness of the CCF. 3. This manuscript adheres to a high writing standard, ensuring fluent and easily understandable content.

Weaknesses

1. How was the Selection frequency map in Figure 1 obtained? Is it based on real data or simulated? 2. It is not clear exactly what iterative refinement of sampling refers to. Why does sampling need to be iteratively refined? The sampling iteration is derived from the diffusion model DDPM and how is it related to the proposed CCF? 3. What are the advantages of using the DDPM for controlled image fusion? In other words, how do diffusion models contribute to the adaptability and controllability of image fusion? 4. Multiple sign ambiguities. Including but not limited to: 1) $c$ denotes both the channels and the given condition in Eq. (5). 2) What does the $f_{\theta}$ refer to? 3) What is the $s_{\theta}$? 5. How are task-specific conditions incorporated into DDPM? Lack of specific implementation details. 6. The theoretical discussion of adaptive customization of conditions for each sample is limited to sections 4.1 and 4.2, lacking specific customization procedures or visual case studies. Consequently, the credibility of this contribution in terms of condition customization is compromised. 7. The authors chose 8 enhanced conditions, i.e., SSIM, Content, Edge, Low-frequency, High-frequency, Spatial Frequency, Edge Intensity, and Standard Deviation enhancements. However, their selection lacks a strong foundation or rationale.

Questions

1. Could you kindly elucidate the methodology employed to generate the Selection frequency map depicted in Figure 1? Real or simulated? 2. Why Use DDPM to achieve conditional image fusion? 3. Can you give an example to explain the specific process of condition selection in the process of sampling iteration?

Rating

6

Confidence

5

Soundness

3

Presentation

2

Contribution

2

Limitations

Please refer to "Weakness".

Thank you for the detailed response.

Thank you for the detailed response. My concerns have been well addressed. Accordingly, I have raised my rating and highly recommend the authors include those discussions on adaptive conditions in the revised version.