SSDiff: Spatial-spectral Integrated Diffusion Model for Remote Sensing Pansharpening

Summary

This paper introduces SSDiff, a model that improves the resolution of satellite imagery by combining high-resolution panchromatic images and low-resolution multispectral images. SSDiff separates image details into spatial and spectral components, and processes them through different network branches, then merges them back together. It utilizes a denoising diffusion model and adopts a designed Alternating Projection Fusion Module (APFM) to complete the spatial-spectral fusion. Additionally, they propose a Frequency Modulation Inter-branch Module (FMIM) to modulate the frequency distribution between branches, as well as a LoRA-like branch-alternating fine-tuning method to further improve performance. The paper provides a solid theoretical foundation, and through experiments on various datasets, SSDiff demonstrates the ability to generate high-resolution multispectral images with minimal spatial and spectral distortion.

Strengths

1. The authors propose an innovative spatial-spectral integrated diffusion model for pansharpening, which is a timely application of diffusion models to this domain. 2. This paper provides a solid theoretical basis for the approach, grounded in concepts from linear algebra and subspace decomposition. 3. The proposed methods in this paper achieve state-of-the-art performance on two datasets. The comprehensive ablation study further adds to the robustness and credibility of the results. 4. This paper is well-organized and the content is clear and easy to read.

Weaknesses

- Efficiency of the method: Evaluating and discussing the efficiency of the DDPM-based method used in SSDiff would help provide a more comprehensive understanding of the practical advantages and limitations of the proposed approach. This information could be valuable for readers to assess the real-world applicability and deployment feasibility of the SSDiff model. - Experimental Section: The proposed method in this paper has been extensively evaluated for its generalization performance, but the experiments are limited to the Pansharpening task. It would be better if there were experiments on other datasets, but it's not essential.

Questions

1. How does the element-wise multiplication operation help preserve the complementary information from the two domains? 2. Could the author elaborate on any challenges faced when integrating information from PAN and LrMSI into a diffusion model? This clarification would help in understanding your methodology.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have provided reasonable Limitations and Broader Impact.

Summary

For the task of multispectral and panchromatic image fusion (Pansharpening), this paper proposes a spatial-spectral integrated diffusion model (SSDiff). The framework is novel, utilizing spatial and spectral branches to learn spatial details and spectral features separately. It introduces an alternating projection fusion strategy and a frequency modulation inter-branch module to enhance fusion quality. Furthermore, designing a LoRA-like fine-tuning strategy for the proposed dual-branch network to capture component-discriminating features more sufficiently. The study demonstrates promising results across multiple datasets.

Strengths

1. The writing of the article is quite clear. 2. The experiments are conducted quite thoroughly. 3. Subspace decomposition to divide the network into spatial and spectral branches has some degree of innovation. And it has reliable theoretical support 4. The alternating projection fusion strategy and the accompanying LoRA-like fine-tuning method can provide insights for future work.

Weaknesses

1. Figure 4 is not clear. It is hard to see how “ the low-frequency components undergo a gradual modulation characterized by a slow and subtle rate of change in the denoising process. In contrast, the modulation process of the high-frequency components exhibits distinct dynamic variation.”. Suggest enhancing the high-frequency image to make it more prominent. 2. To ensure the fairness of the experiment, it would be reasonable to add a control experiment. Continuing to train the 150k base model for 30k iterations using the fine-tuned model's configuration and then comparing the results to the fine-tuned model. This would provide a more comprehensive evaluation of the effectiveness of the fine-tuning strategy.

Questions

1. What does the first row of data in Table 5 represent? The paper states that "the total training iterations of the WV3 datasets are set to 150k. ... and the total fine-tune training iterations are set to 30k." Is the first row's data the result after training for 150k iterations? If that is the case, please address the issue mentioned concern (see weakness 2). 2. Regarding the limitation mentioned about the efficiency disadvantage of diffusion models, can the authors provide more specific comparative results? The paper noted that diffusion models have an efficiency disadvantage compared to other super-resolution methods. To fully evaluate this limitation, it would be helpful if the authors could include more concrete performance comparisons. 3. Why do low-frequency components undergo progressive modulation while high-frequency components exhibit distinct dynamic variations? I don't quite understand this point. 4. What are the specific differences between this method and other DDPM methods in terms of training and sampling processes, e.g. Meng et al.2023.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors provide Limitations and Broader Impact, where limitations are influenced by the nature of the task itself.

Summary

This paper propose a novel spatial-spectral integrated diffusion model called SSDiff based on subspace decomposition, and its main contributions contain: 1) Considering existing DDPM-based methods have not yet designed models for the discriminative features required in the pansharpening task, SSDiff divide the network into spatial and branches based on subspace decomposition. Additionally, an alternating projection fusion module (APFM) is constructed for SSDiff to transform the process of fusing HrMSI into the fusion process of spatial and spectral components. Tests on four widely used pansharpening datasets demostrate SSDiff can achieve state-of-the-art (SOTA) performance. 2) The frequency modulation inter-branch module is used at the junction of spectral and spatial branches to enrich extracted spatial information with more high-frequency information in the denoising process. 3) The proposed L-BAF method is used to fine-tune the network based on the proposed APFM, where the spatial and spectral branches are updated alternately. This design allows alternately fine-tuning the two branches without increasing the parameter count, enabling the learning of more discriminative features.

Strengths

1) Considering existing DDPM-based methods have not yet designed models for the discriminative features required in the pansharpening task, this paper proposes a novel SSDiff method based on subspace decomposition, which leverages spatial and spectral branches to discriminatively capture global spatial information and spectral features, respectively. 2) And from the perspective of subspace decomposition, an alternating projection fusion module (APFM) is constructed to fuse the captured spatial and spectral components, which is concerned with vector projection. 3) To overcome the problem of uneven distribution of frequency information between two branches in the denoising process, a frequency modulation inter-branch module (FMIM) is designed. 4) Through the proposed LoRA-like branch-wise alternating fine-tuning (L-BAF), SSDiff can further reveal spatial and spectral information not discovered in each branch. And L-BAF is inspired by the idea that subspace decomposition can be generalized into the self-attention mechanism. 5) There are abundant experiments to support the effectiveness of SSDiff from various aspects.

Weaknesses

1) Maybe expression should be pay more attention to, both grammatically and logically. For example, the sentences in line 135 and line 140 are incomplete. 2) This paper claims the fusion result images and error maps of all these methods in Fig. 5, yet it's not the truth. And there lacks concerned descriptions about results on GaoFen-2 full dataset. 3) Paragraph from line 162 to line 167 seems not to be expressed clearly about how eq.(3) and eq.(7) can be generalized into matrix form of the self-attention mechanism as shown in Fig.3.

Questions

1) Why introduce another two features from the features of spatial domain and spectral domain separately? 2) Why choose FFT as the specific way to extract the high-frequency information of the feature map obtained from the spatial branch rather than other methods? 3) The design of FMIM?

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

1) This paper evaluate the effectiveness of the proposed SSDiff over pansharpening task and will extend our method to other multispectral fusion tasks. 2) The proposed FMIM can adjust the frequency information between the two branches, but it also introduces additional hyperparameters, increasing the difficulty of fine-tuning during training. 3) The time cost for our approach is higher than other DL-based models, primarily due to the limitation imposed by the large number of sampling steps required in the diffusion model.