Fourier-enhanced Implicit Neural Fusion Network for Multispectral and Hyperspectral Image Fusion

Summary

The paper provides a novel INR-based fusion framework tailored for the MHIF task, effectively capturing high-frequency details and global information through innovative architectural components. The key contributions are transforming the latent features into the frequency domain to enhance high-frequency information and introducing a complex Gabor wavelet activation function to strengthen feature representation and fusion. Experiments on two MHIF datasets demonstrate the state-of-the-art performance achieved by the proposed method, outperforming existing approaches both visually and quantitatively.

Strengths

Originality: This paper provides a novel solution to the problem of MLP-ReLU losing high-frequency information by leveraging domain transformation. The originality is high. Quality: The viewpoints presented in the paper are supported by theoretical proofs and corresponding experimental evidence. Clarity: The paper is well-organized and the content is clear and easy to read. Significance: The proposed solution could have a significant impact on the field of computer vision.

Weaknesses

- Complexity: Some parts of the methodology, particularly the theoretical proofs and implementation details, are complex and may be challenging for readers to fully grasp without additional explanation. - Quality Metrics: The paper does not seem to provide the meaning and calculation methods of the four metrics (PSNR, SAM, ERGAS, SSIM). - Experimental Section: Besides the two benchmark MHIF datasets used in the experiments, how does the proposed method perform in other image fusion tasks or datasets? It would be better if there were experiments on other datasets, but it's not essential.

Questions

1. How does Fourier transformation within the INR framework specifically contribute to observed performance improvements? What are the practical benefits of frequency-domain features? 2. After reviewing the proof of Theorem 1, I don't understand why finding the optimal bandwidth leads to better representation. Could you provide further explanation, especially regarding Fig. 5. 3. For the PSNR curve in Figure 7, on which dataset was this experiment conducted? Was it on the training set, validation set, or test set? The paper does not explicitly specify this. 4. Regarding Fig. 6, were the errors normalized? The more the error maps lean towards red, the greater the difference between the pixel values and the ground truth. Is my understanding correct? The differences between the close-up images seem very small to the naked eye, and I can't discern any differences between the images in the first and third rows.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have included a discussion on the limitations of their work in Section A.1, covering the data simulation, and potential societal impacts.

Summary

The paper introduces a novel Fourier-enhanced Implicit Neural Fusion Network .The core innovation lies in the integration of Fourier transformations within an Implicit Neural Representation framework to address the loss of high-frequency information—a common limitation in existing INR approaches. The effectiveness of FeINFN is demonstrated through experiments on two benchmark MHIF datasets, showing superior performance in terms of PSNR and other metrics compared to state-of-the-art methods.

Strengths

The paper introduces novel components to the INR framework, specifically tailored for MHIF tasks, a creative combination of existing ideas. The proposed modules are theoretically feasible. It provides clear and understandable graphical explanations for complex theories, such as Fig. 5. The paper is generally well-written, with clear explanations of complex concepts and a logical flow of ideas. The method proposed in the article performs optimally on two publicly available datasets, which should contribute significantly to the MHIF field.

Weaknesses

1. Some sections of the paper are overly technical and dense, which might be challenging for readers unfamiliar with the specific domain of Fourier transforms and neural networks. 2. In Table 3, the description of S and F in the last row is incorrect and should be indicated with two check marks. This seems to be an editing error. 3. In the ablation experiments section of the paper, tables 2, 3, and 4 do not provide the corresponding parameter quantities. If available, could you please provide them? Ablation experiments need to compare models under similar parameter conditions.

Questions

1. Network Complexity: The introduction of Spa-Fre IFF and SFID seems to increase the model complexity. Could the authors comment on the computational efficiency of their proposed method compared to the baselines? 2. In the explanation of the compound Gabor wavelet activation function in the article, Theorem 1 mentions optimal bandwidth. What does this refer to, and how does it affect the activation of features? 3. The abstract of the paper claims that the proposed model achieved state-of-the-art performance on two datasets. For the Harvard x4 experiments, how can the fact that the SAM and SSIM metrics did not reach SOTA be explained? 4. “frequency point may correspond to multiple pixels at different positions in the spatial domain”， Why does transforming into the Fourier domain amplify the frequency domain? I'm a bit unclear on this point.

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

4

Limitations

The authors have provided reasonable Limitations and Broader Impact.

Summary

The paper proposes a novel Fourier-enhanced Implicit Neural Fusion Network (FeINFN) specifically designed for the Multispectral and Hyperspectral Image Fusion task. The paper identifies the unique characteristics of the amplitude and phase of the latent codes in both HRHSI and LRHSI, and proposes to enhance high-frequency details and expand the receptive field from the Fourier domain. Additionally, the paper introduces a new decoder to achieve better interaction between spatial domain features and frequency domain features.

Strengths

1. As mentioned in the Summary, the paper's observations on amplitude and phase are intriguing, and the proposed spatial-frequency domain fusion framework aligns with the paper's motivation. 2. The paper presents clear theoretical proofs for its claims, such as the advantage of complex Gabor wavelet activation in finding the optimal bandwidths. 3. The paper is well-written, with clear and organized content. 4. The experimental observations are interesting and can be considered to have a positive impact.

Weaknesses

1. What does 'current neural network-based methods are insensitive to high-frequency information' mean, and how does the proposed model demonstrate sensitivity to high-frequency information? 2. As mentioned in Section 3.4, the Spatial Implicit Fusion Function utilizes frequency encoding in the spatial domain. Does the Frequency Implicit Fusion Function also utilize frequency encoding in the frequency domain? 3. Should the vertical axis scales of (c), (d), and (e) in Figure 5 be consistent? In my understanding, at least (c) and (d) should be consistent to demonstrate that the ground truth (GT) frequencies and the frequencies captured using Gabor activation are similar. 4. This paper proposes a new fusion architecture based on INR. It would be even better if there were discussions and analyses regarding network complexity, such as comparisons in spatial complexity with Transformer methods mentioned in section I. 5. There are some editing errors: the double quotes are displayed incorrectly in lines 307 and 310 as “Fourier Domain" and “chart and stuffed toy". 6. In line 226, “I” should correspond to elsewhere in the article.

Questions

See weaknesses

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

N/A

Summary

This paper proposes a quite interesting hyperspectral and multispectral image fusion framework via implicit representation. Moreover, the authors introduce the Fourier transformation to decouple the amplitude and phase domain.

Strengths

1. Introducing implicit model into the task of hyperspectral multispectral fusion is quite interesting. 2. A novel Gabor wavelet activation function is proposed. Theoratical analysis of Gabor activation is also given; 3. The decoupling process of spatial-spectral domain is reasoable for hyperspectral image; 4. The performance of the proposed method is competitative with SOTA methods; and 5. Paper is easy to follow, figure is clear to vlidate the motivation of designing.

Weaknesses

1. The proposed Fourier decomposition seems to only result in very limited performance improvement. The results from table 3 and figure 7 seem not consistent. 2. Actually, such implicit representation is more plausible to work in a self-regularized overfitting manner than current train-test setting. Authors can make further design or give a unsupervised version. 3. Authors should also compare the computational cost of different methods. 4. Authors should also evaluate the performance with different SR ratios.

Questions

Improvement of Fourier design. Unsupervised training manner. Computational cost. Different SR ratios. As the weakness Sec.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Please refer to weakness sec.

Supplement about the complete version of tables in rebuttal

**A3.** Tab 2,3 updates as shown below： |Methods|PSNR ($\uparrow$)|SAM ($\downarrow$) |ERGAS ($\downarrow$) |SSIM ($\uparrow$)|Params| |:-:|:-:|:-:|:-:|:-:|:-:| |Bilinear|52.23$\pm$4.40|1.92$\pm$0.60|1.03$\pm$0.86|0.997$\pm$0.0021|3.119M| |Bicubic|52.22$\pm$4.31|1.95$\pm$0.61|1.02$\pm$0.82|0.997$\pm$0.0021|3.119M| |Pixel Shuffle|52.26$\pm$4.37|**1.90$\pm$0.59**|1.02$\pm$0.85|0.997$\pm$0.0022|3.057M| |Our|**52.47$\pm$4.10**|1.91$\pm$0.59|**0.98$\pm$0.74**|**0.998$\pm$0.0015**|3.165M| |$\mathcal{S}$/$\mathcal{F}$|PSNR ($\uparrow$)|SAM ($\downarrow$)|ERGAS ($\downarrow$)|SSIM ($\uparrow$)|Params| |:-:|:-:|:-:|:-:|:-:|:-:| |$\checkmark$/✘|52.11$\pm$4.22|1.95$\pm$0.59|1.04$\pm$0.82|0.998$\pm$0.0017|2.869M| |✘/$\checkmark$|47.86$\pm$3.42|3.49$\pm$1.30|1.67$\pm$1.13|0.995$\pm$0.0020|2.940M| |$\checkmark$/$\checkmark$|**52.47$\pm$4.10**|**1.91$\pm$0.59**|**0.98$\pm$0.74**|**0.998$\pm$0.0015**|3.165M| **A6.** Detailed results on the Chikusei dataset and Pavia Center dataset are as follows: | |Chikusei $\times 4$| | | | |Pavia $\times 4$| | | | | |:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:| |**Methods**|**PSNR ($\uparrow$)**|**SAM ($\downarrow$)**|**ERGAS ($\downarrow$)**|**SSIM ($\uparrow$)**|**Params** |**PSNR ($\uparrow$)**|**SAM ($\downarrow$)**|**ERGAS ($\downarrow$)**|**SSIM ($\uparrow$)**|**Params**| |Bicubic|33.35|4.00|7.65|0.815|-|26.65|7.07|8.46|0.614|-| |CSTF-FUS|35.40|5.40|7.88|0.844|-|30.93|11.08|5.74|0.791|-| |LTTR|37.86|3.68|6.27|0.917|-|31.15|6.58|5.50|0.801|-| |LTMR|41.21|2.98|4.84|0.950|-|32.33|6.35|5.10|0.820|-| |IR-TenSR|36.00|5.12|7.86|0.868|-|30.87|6.81|5.82|0.783|-| |ResTFNet|42.33|2.48|3.93|0.950|2.471M|33.56|4.99|4.16|0.885|2.448M| |SSRNet|42.36|2.35|3.92|0.951|0.446M|33.20|5.36|4.36|0.876|0.282M| |HSRNet|42.01|2.33|3.95|0.947|0.633M|32.17|5.60|4.60|0.867|2.061M| |MogDCN|42.21|2.27|3.76|0.936|6.840M|33.84|4.61|4.07|0.889|7.202M| |Fusformer|43.37|2.03|3.49|0.959|0.504M|35.31|4.33|3.37|0.924|0.539M| |DHIF|43.69|1.94|3.33|0.960|22.462M|35.30|4.36|3.35|0.924|38.785M| |PSRT|43.48|2.01|3.47|0.961|0.303M|34.86|4.47|3.54|0.916|0.288M| |3DT-Net|43.53|2.03|3.46|**0.963**|3.464M|35.10|4.44|3.35|**0.927**|3.482M| |DSPNet|43.55|2.03|3.44|0.960|6.138M|35.47|4.26|**3.30**|**0.927**|6.115M| |BDT|43.25|2.09|3.44|0.955|3.263M|34.55|4.66|3.70|0.904|3.056M| |MIMO-SST|43.36|2.09|3.48|0.958|4.983M|35.37|4.48|3.34|0.922|5.227M| |**Proposed**|**43.88**|**1.91**|**3.30**|**0.963**|3.488M|**35.51**|**4.15**|**3.30**|**0.927**|3.399M|

Comments on the rebuttal

The response addressed all my comments well. Therefore, I'll raise the score to ACCEPT.