Text-DiFuse: An Interactive Multi-Modal Image Fusion Framework based on Text-modulated Diffusion Model

Summary

A new paradigm of multi-modal image fusion named Text-DiFuse is introduced, based on the diffusion model. The paradigm embeds a mechanism for aggregating feature-level multi-modal image information into the diffusion process of degrading multi-modal images, addressing the optimization gap between "degradation removal" and "multi-modal information fusion". Additionally, a zero-shot model is introduced to modulate the fusion strategy based on user-input target text, enhancing the saliency of the target of interest. The conducted experiments suggest significant improvements in both human visual perception and advanced computer vision tasks.

Strengths

1) Embedding the mechanism of aggregating feature-level information into multiple diffusion processes to fuse multi-modal information is interesting. It is foreseeable that this diffusion paradigm produces fused images with better fidelity compared to methods based on likelihood-constrained diffusion models. 2) The coupled approach effectively resolves the issue of compound degradation in the process of multi-modal fusion, as evidenced by experimental results demonstrating significant advantages over the sequential approach. 3) The authors emphasize the importance of foreground targets in advanced visual tasks and propose enhancing target saliency through zero-shot assisted re-modulation. This approach diverges from traditional uniform fusion rules, demonstrating effectiveness. 4) This approach shows strong applicability. It demonstrates superior performance in multiple tasks including infrared and visible image fusion, medical image fusion, and polarization image fusion.

Weaknesses

1) After the diffusion model is effectively trained, the sampling process can follow different step intervals. The information fusion in this method is integrated into the diffusion process, but the article does not seem to specify the sampling interval at which the results are obtained. Also, this article does not discuss the impact of the sampling interval on the fusion performance. 2) The presentation is slightly unclear. For example, from Equation 2 to Equation 6, both the features and the images carry the condition N that represents the degradation. Why does equation 7 no longer include N? Why can it be considered that the degradation has been removed at this point? 3) In Table 2 and Figure 4, some existing image restoration methods are cascaded in front of the fusion method to promote fairness in comparison, such as low-light enhancement (CLIP-LIT), denoising (SDAP), and white balance (AWB) algorithms. Please explain the choice of the order in which they are connected in series, i.e. why low light enhancement first, then denoising, and finally white balance. 4) Modulating the salience of targets of interest in the fusion process through language is novel. Intuitively, I think the improvement in semantic properties brought about by this modulation is widespread. Currently, the effectiveness of language modulation has only been verified in the semantic segmentation scenario. It is recommended to provide an evaluation in the object detection scenario to further verify its role.

Questions

Please refer to the weaknesses part.

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors have analyzed the limitations and potential negative impact.

Summary

This work focuses on the topic of multi-modal image fusion. Two innovations enhance the performance of the fusion. One is the clever integration of information fusion into the diffusion process. This coupling way enables the fusion function to resist degradation. The other is the introduction of a text-based fusion remodulation strategy. This changes the limitation of previous fusion methods that could only use fixed mappings, allowing for the dynamic adjustment of the fused image based on specific requirements. This remodulation also enhances semantic attributes, improving the scores of the semantic segmentation task.

Strengths

1. Integrating information fusion into the diffusion process is novel. Especially, each sampling step triggers an information fusion, which enhances the sufficiency of information fusion. This coupling can ensure the robustness of information fusion, addressing challenges such as low light, noise, and color cast. 2. The introduction of multi-modal large models is interesting, particularly the ability to remodulate fused images using textual commands. This capability could potentially facilitate the flexible deployment of the proposed method across different application requirements. The demonstration of enhanced semantic attributes and improved semantic segmentation performance is good. 3. Overall, the experiments are relatively sufficient. The comparative experiments include both baseline comparisons and pre-enhancement comparisons, which are important for ensuring fairness. 4. The code is provided, which helps in reproducing the performance.

Weaknesses

1. On page 5, line 174, the source data used for fusion contains degradation, [{Xb,Y}|N]. My question is, in Equations (9) and (10), where do the clean {Xb,Y} used to guide the fusion come from? Is there a multi-modal dataset that contains paired degraded and clean data? The paper seems to lack an explanation for this. 2. The forward process of the diffusion model involves T steps of noise addition, while the reverse process consists of T steps of iterative sampling. Is the Z0 obtained in equation (8) a hypothetical Z0 derived from the diffusion relation at each sampling, or is it the Z0 after completing the full T steps of sampling? This determines the object of the constraints in the loss functions (9) and (10). It would be better to provide a detailed discussion on this. 3. Only after the T steps of sampling can the data without degradation be obtained. So why can Z_{t-1}^b in equation (7) be considered free from degradation N? 4. It's understandable that using textual modulation to control the desired targets of interest can enhance semantic attributes. My question is whether these enhanced semantic attributes can be generalized. In other words, can it also be effective in other high-level visual tasks besides semantic segmentation? 5. Typo: The Zt on the left side of equation (8) seems to have a missing superscript b.

Questions

Please answer the question raised in Weaknesses.

Rating

7

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes, there are discussions about the limitations and potential negative societal impacts.

Summary

This paper addresses two primary challenges in multimodal image fusion: the mixed degradation of modalities and the insufficient salience of target objects. It proposes two methods to tackle these challenges: feature-level fusion diffusion and the re-modulation of fusion rules in target areas using a zero-shot segmentation model. They implement adequate experiments for evaluation, and the results demonstrate this method's advanced performance across various aspects, including the visual and semantic.

Strengths

+ The mixed degradation of modalities and the insufficient salience of target objects are two interesting problems in multimodal image fusion. This paper’s discussion and solution of these two problems may promote the usability of fusion methods in real scenarios. + The information fusion at the feature level is integrated into the diffusion process, which effectively realizes the degradation removal. + The customized object highlighting strategy based on the zero-shot segmentation model is flexible. In particular, its gain in semantic attributes will increase the usability of the fused image in downstream tasks. + This paper conducts lots of comparative experiments and ablation studies on the overall method. + The narrative of this paper is comprehensive and clear. For me, it's easy to follow.

Weaknesses

- This paper mentioned that the diffusion model is pre-trained to enable the denoising network to have the degradation removal function. However, details about the construction of the data used to train the diffusion model are missing. They need to describe this process to make the overall approach clearer. - This paper focuses on multimodal image fusion, being reflected in the title. In the main text, the proposed method is evaluated in two scenarios: infrared and visible image fusion and medical image fusion. In the supplementary materials, they further provide experiments on polarization image fusion. I am curious whether the applicable scenarios of the proposed method can be further expanded, such as the typical fusion of near-infrared and visible bands. - The experiments on polarization image fusion only provide visual results, and it would be better to add a quantitative evaluation. - I noticed that the proposed method separates the chrominance component and the brightness component, and then performs de-degradation on them separately. An explanation of why this operation is needed should be given. Perhaps an ablation experiment could more intuitively show the effect of this operation. - There are some minor typos, such as potential misspellings of dataset names in Tables 1 and 2. In addition, there seems to be a lack of underline on AG's second place.

Questions

1. How were the degradation condition data constructed, were paired supervised datasets used or synthetic datasets? 2. Has there been an attempt to evaluate the fusion on the fusion of near-infrared and visible bands? 3. Could you provide the quantitative results of polarization image fusion? 4. The separation of chrominance and brightness requires more explanation.

Rating

6

Confidence

5

Soundness

4

Presentation

3

Contribution

4

Limitations

Limitations and broader impacts have been included.

Summary

This paper proposes an interactive framework that can exploit the intrinsic connection between image restoration and multi-modal image fusion. The authors embed information fusion within the diffusion process and address the "composite degradation challenge" i.e., multi-modal information integration with effective information restoration from degradation like colour casts, noise, and improper lighting. Particularly, first, independent conditional diffusion models are applied to each modality with compound degradation -- the degradation removal priors are embedded into the encoder-decoder network. A fusion control module (FCM) sits in the multi-step diffusion process to manage the integration of multi-modal features and remove degradation during T-step sampling. Next, to interactively enhance focus on objects of interest during diffusion fusion, the authors designed a text-controlled fusion re-modulation strategy that incorporates a text and a zero-shot OWL-ViT to identify the objects of interest. In other words, this step performs a secondary modulation with the built-in prior to enhance saliency.

Strengths

- It is interesting to see the effect of combining image restoration and multi-modal image fusion in a single framework. - The proposed method is well-motivated and the authors provide a clear explanation of the method. - The Text-controlled fusion re-modulation strategy could be useful in many applications. - The authors provide the code in the supplementary material (although I have only dry run the code and not tested it). - Extensive experiments are conducted to validate the proposed method. - The authors provide ablation studies to show the effectiveness of each component of the proposed method.

Weaknesses

For now, I have minor concerns and mostly questions (as listed in the next section). - The authors should add a brief discussion on the competitors in supplementary material. For example, differences between TarDAL, DeFusion, LRRNet, DDFM, and MRFS. - Typo in Eq. 2: $\Theta_{t}^{X^{B}}$ should be $\Theta_{t}^{X^{b}}$. - Improve the caption of Figure 2. I had to read the entire paper to understand the figure (it should be self-explanatory). - Not much of a weakness, but the authors could improve the clarity of the paper if they added the tensor dimension of each variable in Figure 2.

Questions

- In the proposed method, input visual image X is broken into brightness and chroma components. I wonder if this step is absolutely necessary -- or can we skip $\eta^{c}_{ theta}$ and directly combine both $X$ and $Y$ as three-channel images $\mathbb{R}^{H \times W \times 3}$. - What if I use InstructIR (for image restoration) followed by MaxFusion (for multi-modal fusion) -- how would it compare with the proposed method? - (InstructIR) https://arxiv.org/pdf/2401.16468 | Github: https://github.com/mv-lab/InstructIR - (MaxFusion) https://arxiv.org/pdf/2404.09977 | Github: https://github.com/Nithin-GK/MaxFusion - In the Limitation and Future work section, will a no-training approach be possible? For example, "MaxFusion"-like approach but with the proposed deep integration of image restoration and multi-modal fusion.

Rating

6

Confidence

2

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors discussed limitation section in supplementary A.4 -- Particularly, Table S1 shows number of parameters and runtime. This is highly appreciated.

To author's response

Thanks for the effort to clarify my questions in the provided rebuttal. Using the two-step strategy to address the limitation of data unavailability is clever, and I'm pleased to see the general semantic attributes brought by textual modulation. Therefore, I'm inclined to accept this paper. In the camera-ready version, please include the provided clarifications about the source of Z0 in loss functions (9) and (10) and the degradation N.