MambaLLIE: Implicit Retinex-Aware Low Light Enhancement with Global-then-Local State Space

Summary

This paper proposes a Mamba-based framework, namely MambaLLIE, for low-light image enhancement. Specifically, the authors claim that they have two technical contributions: (i) A global-then-local state space block that integrates a local-enhanced state space module and an implicit Retinex-aware selective kernel module to capture intricate global and local dependencies. (ii) An implicit Retinex-aware selective kernel mechanism to guide deeper neural representations and segregate them into independent positive and negative illumination components before integrating them.

Strengths

(i) The idea is novel and interesting. Mamba is a scorching research topic in computer vision now and has shown promising performance in many high-level vision tasks like image recognition, object detection, image segmentation, etc. But until now, there have been fewer efforts dedicated to the low-light image enhancement task, a small low-level topic. Thus, it is very exciting to see this wonderful work. (ii) The presentation is well-dressed. I like the style of the figures in this paper. For example, the teaser figure can clearly show the advantages in larger receptive fields of the proposed MambaLLIE than previous Transformer-based methods and Mamba-based image restoration methods. For another example, the pipeline in Figure 2 can also clearly show the workflow and the details of each submodule of the whole framework. (iii) The performance is good and solid. The proposed MambaLLIE significantly outperforms the state-of-the-art methods on six benchmarks including LOL-v2-syn, LOL-v2-real, SMID, SDSD-indoor, and SDSD-outdoor, as reported in Table 1. The visual comparisons in Figure 3 also show the effectiveness of the proposed method. Looks good.

Weaknesses

(i) The motivation is not clear. To be specific, why use Mamba for low-light image enhancement? The authors claim that the Mamba can help capture long-range dependences. However, Transformer architecture can also model the long-range dependences. Why use Mamba instead of Transformer? Also, why design the space state module like this? The insight of the proposed Mamba should be analyzed in more detail. (ii) A technique should be explained, i.e., what is the scanning operation in the state space module? The authors mention the scan in Mamba but I could not find any mathematical formula or text description in the paper to explain the computation process of this scan. (iii) In table 1, although the performance of the proposed MambaLLIE is better than the state-of-the-art Transformer-based method Retinexformer, its computational complexity and memory cost are larger than those of Retinexformer. For example, the FLOPS and the Params are 33% and 38% higher. In addition, what about the comparison of performance on the MIT-Adobe-FiveK dataset (sRGB output mode)? (iv) In Figure 4, it seems that Retinexformer achieves more visually pleasant results. For example, in the third line of Figure 6, Retinexformer reconstructs the complete edge of the redwood in the left zoomed-in patch but the proposed MambaLLIE fails. Thus, it is better to compare MambaLLIE and the state-of-the-art method on unpaired datasets such as the LIME, NPE, MEF, DICM, and VV datasets. (v) The source code and pre-trained models have not been submitted. The reproducibility cannot be checked.

Questions

I am curious about the training time comparison between the proposed MambaLLIE and other methods.

Rating

8

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the authors have analyzed the limitations in the main paper.

Summary

This paper presents MambaLLIE, an implicit Retinex-awre low light image enhancement framework with modified state space blocks. Specifically, a global-then-local state space block (GLSSB) is designed, which incorporates a local-enhanced state space module (LESSM) and an implicit Retinex-aware selective kernel (IRSK) module. By enhancing the original SSMs with local bias, the proposed MambaLLIE outperforms state-of-the-art CNN and Transformer-based methods.

Strengths

1. The main idea of this paper is well illustrated. 2. Experimental results and user studies demonstrate that the paper achieves better performance than the compared approaches.

Weaknesses

1. The novelty of this paper is limited. Like many Transformer-based works that embed local modeling capabilities in ViTs, the main contribution of this paper is incorporating local dependencies into SSMs. However, the advantages of the proposed GLSSB compared to ViTs and original SSMs have not been thoroughly analyzed. Furthermore, given the authors' design, the proposed GLSSB should have advantages in many other visual tasks, which need to be demonstrated through more experiments. 2. The writing of this paper needs improvement as there are many unclear descriptions. For example, how is the illumination prior in the paper obtained, how does the proposed IRSK module work specifically, and what are the specific settings for the different variants in the ablation study? 3. Some state-of-the-art works have not been compared (e.g., DiffLL[1]), and the experimental results in this paper do not seem to outperform them. 4. As the low-light enhancement task typically encounters issues like color distortion or other visual artifacts, the authors should compare more perceptual evaluation metrics. Merely reporting PSNR and SSIM is not sufficient. [1] Jiang et al, Low-Light Image Enhancement with Wavelet-based Diffusion Models. Siggraph asia 2023.

Questions

Please refer to the `Weaknesses'. Moreover, it appears that the color in the last column of Table 2 is incorrectly marked.

Rating

4

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors have discussed the limitations in the paper.

Summary

The authors proposed a Mamba-inspired method for LLIE, which is designed to address some challenges of the existing method, MambaIR. By integrating GLSSB consisting of LESSM and IRSK, the authors could make the model capture a large local receptive field while preserving global understanding natures. The overall pipeline is explained in detail and motivated by sufficient reasoning. MambaLLIE achieved SOTA performance on several benchmarks, significantly outperforming prior works.

Strengths

The overall pipeline is explained in detail and motivated by sufficient reasoning. MambaLLIE achieved SOTA performance on several benchmarks, significantly outperforming prior works. Considering emerging attention on Mamba for low-level vision tasks, the proposed work is timely and can thus attract attention to the community.

Weaknesses

The proposed method is somewhat incremental in that it makes small (although relevant) changes to MambaIR. Since this is not the first work that introduces Mamba for LLIE, a more in-depth analysis and exploration is expected; however, the proposed is a slightly better engineered network structure. The authors are recommended to discuss MambaIR in more detail and clarify technical contributions over MambaIR.

Questions

Although the ERF visualization in Figure 1 is impressive, the authors are expected to provide some numeric results for quantitative performance comparisons. It is not clear why it is called "retinex-aware selection" in Figure 2(d). There are several methods dedicated for face detection in low light. It will be interesting to compare face detection performance compared to such methods,

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors adequately addressed the limitations; but some complexity analysis will be helpful for readers to understand the method.

Summary

This paper introduces a low-light image enhancement module (MambaLLIE). This module has a U-shaped structure and each GLSSB block follows the Transformer-based design. LESSM is proposed to capture the spatial long-term dependency. IRSK is proposed to introduce large and selective kernels for enhancing feature-capturing ability. IRSK also introduces illumination guidance by utilising attention mechanisms.

Strengths

1. The usage of the state space model and large selective kernel greatly increases the effective receptive fields. By capturing longer dependency on features, the network can generate better outputs. 2. MambaLLIE achieves state-of-the-art results on low-light image enhancement tasks across various datasets and evaluation methods.

Weaknesses

1. Could this paper state the difference between LESSM and the Vision State Space Module in [14]? It seems these two blocks have similar designs. 2. The core block GLSSB appears to be a combination of [14] and [23]. However, since [14] is proposed for image restoration tasks, this may limit the novelty of this paper. 3. The descriptions of baseline-1 and baseline-2 are not clear. 4. In Section 4.4 Ablation Study, which dataset is used for the ablation study? How about other datasets? If the dataset used in the ablation study is SDSD-indoor, then the baseline-1 result (PSNR 28.87, SSIM 0.865) is quite similar to [14] (PSNR 28.97, SSIM 0.884), which may indicate that the better performance of this paper relies on Retinex-aware guidance rather than a larger receptive field. However, Retinex-aware guidance has already been explored in [3] and "Low-Light Image Enhancement with Multi-stage Residue Quantization and Brightness-aware Attention."

Questions

1. Could this paper provide an analysis or explanation of why MambaLLIE has a larger receptive field in terms of global and local?

Rating

4

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

This paper discusses limitation in Sec 5 and give potential solutions.

Response to the author rebuttal

Thanks for your response. Most of my concerns have been addressed except one. You claim that `Mamba over Transformers due to its efficient long-range dependency modeling with linear complexity. Transformers can model long-range dependencies, but they typically involve higher computational complexity and memory costs.` However, this is not true. Many Transformers also enjoy linear computational complexity, e.g., Retinexformer. It takes much shorter time to train than the Mamba-based methods. Anyway, this is a good attempt to explore the potential of Mamba in LLE. Thus, I decided to raise my score to `strong accept` to support you. Thanks for your effort.