Library

MambaSCI: Efficient Mamba-UNet for Quad-Bayer Patterned Video Snapshot Compressive Imaging

Color video snapshot compressive imaging (SCI) employs computational imaging techniques to capture multiple sequential video frames in a single Bayer-patterned measurement. With the increasing popularity of quad-Bayer pattern in mainstream smartphone cameras for capturing high-resolution videos, mobile photography has become more accessible to a wider audience. However, existing color video SCI reconstruction algorithms are designed based on the traditional Bayer pattern. When applied to videos captured by quad-Bayer cameras, these algorithms often result in color distortion and ineffective demosaicing, rendering them impractical for primary equipment. To address this challenge, we propose the MambaSCI method, which leverages the Mamba and UNet architectures for efficient reconstruction of quad-Bayer patterned color video SCI. To the best of our knowledge, our work presents the first algorithm for quad-Bayer patterned SCI reconstruction, and also the initial application of the Mamba model to this task. Specifically, we customize Residual-Mamba-Blocks, which residually connect the Spatial-Temporal Mamba (STMamba), Edge-Detail-Reconstruction (EDR) module, and Channel Attention (CA) module. Respectively, STMamba is used to model long-range spatial-temporal dependencies with linear complexity, EDR is for better edge-detail reconstruction, and CA is used to compensate for the missing channel information interaction in Mamba model. Experiments demonstrate that MambaSCI surpasses state-of-the-art methods with lower computational and memory costs. PyTorch style pseudo-code for the core modules is provided in the supplementary materials.

cs.CVeess.IVComputer ScienceEngineering
View graphSource

Paper

Similar papers

Memory-Efficient Network for Large-scale Video Compressive SensingZiheng Cheng, Bo Chen, Guanliang Liu et al.2021 · Computer Vision and Pattern Recognition · 81 citations In Library
EfficientSCI: Densely Connected Network with Space-time Factorization for Large-scale Video Snapshot Compressive ImagingLishun Wang, Miao Cao, Xin Yuan2023 · Computer Vision and Pattern Recognition · 43 citations In Library
Plug-and-Play Algorithms for Video Snapshot Compressive ImagingXin Yuan, Yang Liu, J. Suo et al.2021 · IEEE Transactions on Pattern Analysis and Machine Intelligence · 111 citations In Library
MetaSCI: Scalable and Adaptive Reconstruction for Video Compressive SensingZhengjue Wang, Hao Zhang, Ziheng Cheng et al.2021 · Computer Vision and Pattern Recognition · 66 citations In Library
Binarized Mamba-Transformer for Lightweight Quad Bayer HybridEVS DemosaicingShiyang Zhou, Haijin Zeng, Yunfan Lu et al.2025 · Computer Vision and Pattern Recognition · 9 citations In Library
Adaptive Deep PnP Algorithm for Video Snapshot Compressive ImagingZong-Jhe Wu, Chengshuai Yang, Xiongfei Su et al.2022 · International Journal of Computer Vision · 40 citations In Library
Towards Real-time Video Compressive Sensing on Mobile DevicesMiao Cao, Lishun Wang, Huan Wang et al.2024 · ACM Multimedia · 7 citations In Library
Unfolding Framework with Prior of Convolution-Transformer Mixture and Uncertainty Estimation for Video Snapshot Compressive ImagingSiming Zheng, Xin Yuan2023 · IEEE International Conference on Computer Vision · 7 citations In Library
MambaVC: Learned Visual Compression with Selective State SpacesS. Joe Qin, Jinpeng Wang, Yimin Zhou et al.2024 · arXiv (Cornell University) In Library
QuadMamba: Learning Quadtree-based Selective Scan for Visual State Space ModelFei Xie, Weijia Zhang, Zhongdao Wang et al.2024 · Neural Information Processing Systems · 34 citations In Library

Peer review

Accept (poster)NeurIPS 2024 · via OpenReview
Reviewer yWFS8/10 · confidence 5/52024-07-06

Summary

The paper proposes a method called MambaSCI for efficient reconstruction of quad-Bayer patterned color video snapshot compressive imaging.This method surpasses state-of-the-art algorithms with lower computational and memory costs, providing improved color accuracy and demosaicing.

Strengths

1) The contributions of this paper are quite novel. The usage of Mamba for video SCI reconstruction has been used for the first time. Moreover, the paper also introduces Quad-Bayer CFA patterned color video SCI reconstruction which is also being done for the first time. 2) The quality of the results produced is quite high. The average PSNR and SSIM scores produced with the method proposed in this paper are both comfortably beating the state of the art results. 3) The paper is well written and the presentation is very clear. The citations, and comparisons with SOTA are sufficient to build confidence in the proposed method. 4) The contributions due to each individual module has been presented which clarifies the necessity for each.

Weaknesses

1) The reconstruction seems to be performed on fairly low resolution images(512 x 512). I would have liked to see the simulations being done on higher resolution. 2) The proposed method is tested only on simulated datasets (since there is no real SCI dataset based on quad-Bayer pattern). To me, it is unclear how the simulated datasets have been created. Was noise added while creating the simulated datasets? My biggest concern is the way the noise has been modeled while creating the simulated dataset might not match the actual data(If there was a way to do it). Therefore, currently the efficacy of the provided solution is somewhat hypothetical.

Questions

1) The paper describes the computational benefits of the proposed approach over SOTA in terms of fewer parameters and FLOPS, however no concrete numbers have been provided around the performance. Is it possible to add some information around the time it would take to reconstruct each frame? 2) There is no mention on how the masks were generated. Or the conditioning of the masking matrix. This very important information should be included. 3) What is the relationship between T (number of frames used during reconstruction) and the amount of motion between consecutive frames? For instance I would expect you could use more frames(T) if the object was moving slowly within the scene, compared to when it is moving fast. This might help in further improving the computational cost. 4) There is no tiling done during reconstruction. I wonder why this is so? Especially if you want to deal with higher resolution images?

Rating

8

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

Limitations have been adequately addressed in the appendix, so nothing to comment upon here.

Reviewer xnVL5/10 · confidence 4/52024-07-08

Summary

This paper investigate video snapshot compression imaging reconstruction task by Quad-Bayer CFA pattern into color video SCI. They design a Residual-Mamba-Block consisting of ST-Mamba, Edge-Detail-Reconstruction module and Channel-wise attention module to enhance reconstruction quality and edge details. Experimental results on many datasets validate its effectiveness.

Strengths

It is interesting to introduce Mamba into a new task. They investigate long sequence modeling to further validate the effectiveness of ST-Mamba. They proposed a novel framework for video SCI reconstruction task, which outperform existing methods not only in effectiveness but also in efficiency.

Weaknesses

The contribution (ii) and (iii) have limited novelty. It seems that the author simply adapt ST-Mamba to video SCI reconstruction task without elaborated designs. The proposed Residual-Mamba-Block is an incremental combination of ST-Mamba from Vivim, DWConv and Channel-wise Attention. The author should further clarify the novelty. More description of EDR module should be given. Why EDR module can provide more edge details? Observed from Table 4, the improvements on large-scale simulation color video are not consistent across datasets. More analysis on such phenomenon should be given. While the author claim the results on Messi and Hummingbird is strained by parameters and FLOPs, specific quantitative comparison should be provided.

Questions

Why we need Mamba instead of self-attention in this task? Why quad-Bayer pattern is better than a single Bayer-patterned measurement?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The author is suggested to compare the efficiency of the proposed method and existing methods on long videos to validate the effectiveness on long sequence.

Authorsrebuttal2024-08-11

Further Explanation

Thanks for your valuable comments. **Q1: Clarification about comparison on long videos.** **A1:** Regarding your questions about run-time and performance, we'll address them as follows: - **Performance:** As the model's FLOPs increase with $T$, our method maintains only **9.8%** of the FLOPs required by EfficientSCI. While achieving comparable PSNR to EfficientSCI, our method significantly outperforms it in SSIM metrics. Additionally, as shown in the visual comparison in **Figure 4** of the submitted PDF, our method provides superior visualization compared to EfficientSCI. - **Run-time:** The reason why our method's runtime is not as competitive as EfficientSCI's can be attributed to two key factors: - **Mamba Characteristics.** Mamba's acceleration is tied to the GPU's hardware characteristics and the underlying CUDA architecture, limiting our ability to enhance its speed directly. - **Network Framework Optimization.** To achieve high-quality reconstruction, our framework currently uses 3D convolution for channel feature interactions at the end of each Residual-Mamba-Block. However, this process becomes more time-intensive as $T$ increases. We plan to optimize the framework further in the future to develop a lightweight, high-quality, and faster reconstruction algorithm. **Q2: Clarification of the innovation of DWConv.** **A2:** The depth-wise convolution used by Vivim [1] is essentially a 3D depth-wise convolutional layer, which focuses solely on spatial feature extraction. **It lacks a pointwise convolution component, meaning it doesn't account for the relationships between channels**. In contrast, the DWConv we designed **integrates both depthwise and pointwise convolution**. First, it processes spatial information through depthwise convolution, then it combines the relationships between channels via pointwise convolution to produce the final output features. This design allows us to effectively extract local detailed features and combinations between channels while reducing computational costs and the number of parameters. Thank you for your insightful feedback. We will update the DWConv section of Figure 4 in the main manuscript accordingly and provide a more detailed explanation of DWConv in lines 186-187. Additionally, we will include further analysis in the ablation study in Section 4.5 to enhance the clarity and robustness of our findings. [1] Yang Y, Xing Z, Zhu L. Vivim: a video vision mamba for medical video object segmentation[J]. arXiv preprint arXiv:2401.14168, 2024.

Reviewer xnVL2024-08-13

Thank you for the rebuttal. Most of my concerns have been addressed. However, I strongly suggest the authors include these additional experiments and analyses in the revised manuscript, and clearly describe the difference against previous Mamba-related works. I tend to maintain my score.

Reviewer zshs3/10 · confidence 5/52024-07-13

Summary

This manuscript introduces the Mamba model and Quad-Bayer CFA pattern into color video snapshot compressive imaging (SCI) for the first time. Specifically, the proposed MambaSCI adopts a non-symmetric U-shaped encoder-decoder architecture, which includes DWConv, Residual-Mamba-Blocks, and ReConv. The Residual-Mamba-Blocks integrate several modules designed to enhance reconstruction quality and edge details. Experiments demonstrate that MambaSCI outperforms comparison methods in both quantitative and qualitative results, with fewer parameters and FLOPs.

Strengths

1. This work introduces a method for quad-Bayer patterned color video SCI for the first time. Quad-Bayer sensors offer better hardware performance than Bayer pattern sensors and represent a promising direction worth exploring in SCI. 2. The Mamba model, as a popular module for extracting data causality, aligns well with SCI tasks. Its incorporation into SCI is well-justified. 3. The manuscript is generally well-structured and well-written.

Weaknesses

+ The proposed MambaSCI takes X_in as input, which is obtained from initialization Block. So the framework can be seen as a simple video enhancement task. Furthermore, MambaSCI does not consider the characteristics of Quad-Bayer pattern like previous work (STFormer, EfficientSCI), but instead directly adopts or integrates popular modules.The in-depth analysis appears to be lacking in certain modules, specifically within the passages from lines 185-187 and 193-197. +The lack of an explanation for the initialization block raises questions about whether special processing was done to address issues introduced by the quad-Bayer pattern. It would be better to add further visualization of the X_in results for comparison. +In Figure 6, the results of proposed method show artifacts and color distortion in the lower left corner of ``Swinger”. More analysis should be included. +The manuscript lacks real experiments on videos captured by quad-Bayer patterns. All the results are obtained on simulated data. Results on real data are expected to evaluate the effectiveness of the proposed method.

Questions

Please refer to weakness.

Rating

3

Confidence

5

Soundness

2

Presentation

3

Contribution

1

Limitations

Please refer to weakness.

Authorsrebuttal2024-08-10

Further explanation

Thanks for your valuable comments. **Q1: Clarification on the lack of real experiments using videos captured by quad-Bayer patterns.** **A1:** As we are pioneering a **new task** in the field of video SCI, which is still in its **exploratory stage**, no public dataset currently exists for this class. Meanwhile, our main innovations include: - **New task:** Quad-Bayer offers significant advantages over traditional Bayer sensors, such as higher resolution and HDR capabilities, and is widely used in smartphone cameras. However, there have been no studies on its application in video SCI. We are **the first** to introduce quad-Bayer into video SCI, aiming to expand the field and achieve higher quality reconstruction. - **New method:** To enable lightweight reconstruction, we are **the first** to utilize an asymmetric U-shaped architecture, employing customized Residual-Mamba-Blocks modules to achieve efficient, high-quality video reconstruction. Furthermore, constructing the corresponding SCI encoding system requires specialized optical encoding hardware and significant time, which is often beyond the capabilities of researchers focused on decoding algorithms. **We are working on building and refining this system** to fully leverage the advantages of quad-Bayer over Bayer, such as capturing video in low-light environments and compressing HDR video SCI by adjusting pixel exposure within the 2x2 color grid. **Q2: Clarification of swinger's visual comparison.** **A2:** As detailed in lines 494-500 of the Supplementary Material of main manuscript, we utilized the high-performance BJDD [1] model for joint denoising and demosaicing to display the RGB images reconstructed by PnP-FastDvDnet. Specifically, **PnP-FastDvDnet only handles Raw domain reconstruction**, while BJDD performs further denoising and demosaicing to obtain the final RGB video. In contrast, **our model integrates both Raw domain reconstruction and demosaicing**. As a result, the visual quality of PnP-FastDvDnet may appear superior due to the BJDD network's influence. Therefore, as described in the main text, to ensure a fair comparison, we converted the RGB video output from MambaSCI back to the Raw domain and calculated metrics such as PSNR and SSIM in the Raw domain. The performance metrics of MambaSCI vs. PnP-FastDvDnet on swinger in Raw domain are compared as follows: |Methods|PSNR (dB)|SSIM| |-|-|-| |PnP-FastDvDNet|28.60|0.887| |MambaSCI|**29.78**|**0.920**| As seen in the table, our method outperforms PnP-FastDVDnet in both PSNR and SSIM in the Raw domain. Additionally, as shown in Figure 6 and Figure 15 of the main manuscript, our approach achieves superior reconstruction in other detailed areas compared to PnP-FastDVDnet. [1] A Sharif S M, Naqvi R A, Biswas M. "Beyond joint demosaicking and denoising: An image processing pipeline for a pixel-bin image sensor." *CVPR* 2021.

Reviewer zshs2024-08-11

Further discussion

I appreciate the authors' clarification and the new result in the Raw domain. However, I am not sure whether my concerns have been addressed. **Q1: Real System.** For simulation as is, changing the Bayer pattern is a simple modofication by easily replacing the old Bayer mask by the new mask in the data generation and reconstruction. Moreover, the simulation is oversimplified without considering the camera response, optical transmission, and mask discretization ($\mathbf{M} \in \mathbb{R}$ in current work), which present significant gap against practical implementation. Glad to know that the authors are constructing a real system for the proposed pipeline, which I feel that would significantly lift the contribution and quality of this research work. **Q2: Reconstruction Quality.** Intuitively, if trained properly, the proposed end2end reconstruction pipeline has a great potential to outperform the cascaded “PnP-FastDvDnet + BJDD” combination. If it is not the case, I feel that it would be worth to further improve the reconstruction net.

Authorsrebuttal2024-08-11

Further Explanation

Thanks for your valuable comments. **Q1: Real System.** **A1:** Since we are exploring a new task in the SCI field, our focus has been on addressing issues like artifacts and color distortion when dealing with quad-Bayer array video. We appreciate your insight into the potential challenges that might arise in real-world scenarios. We will take your suggestions into account and work on developing a quad-Bayer-based coding system as soon as possible. **Q2: Reconstruction Quality.** **A2:** Our proposed E2E network strikes a balance between performance and efficiency. As demonstrated in **Figure 5 of the main manuscript**, at a resolution of 512x512, our method produces superior visual results compared to the cascaded "PnP-FastDvDnet + BJDD" combination. Our approach maintains high color fidelity without introducing additional artifacts, whereas the "PnP-FastDvDnet + BJDD" combination suffers from severe artifacts and shape distortions across multiple datasets. Also, we provide a comparison of the corresponding performance metrics as well as a comparison of the reconstruction times in the table below: *Table 1: A comparison of performance metrics (PSNR (dB), SSIM) and reconstruction time between PnP-FastDVDnet and MambaSCI.* (**Note: The time for PnP-FastDvDnet does not include the additional time required for the demosaicing process performed by BJDD.**) |Method|Beauty|Bosphours|Runner|ShakeNDry|Traffic|Jockey|Running time(s)| |-|-|-|-|-|-|-|-| |PnP-FastDvDnet|34.29,0.967|33.07,0.947|34.18,0.928|30.11,0.883|23.74,0.811|32.70,0.921|14.60| |MambaSCI|**36.95,0.979**|**38.62,0.982**|**40.02,0.977**|**34.55,0.950**|**27.52,0.904**|**36.54,0.960**|**5.12**| Table 1 clearly highlights the superior performance of our approach compared to PnP-FastDVDnet, along with a significant advantage in the time required for reconstruction. Additionally, we are actively exploring improvements to the pipeline and have identified the following areas for enhancement: - **Raw Domain Reconstruction Performance:** While our method performs well on middle-scale datasets, it does not achieve superior metrics on large-scale datasets, primarily due to the significant difference in resolution sizes between training (128\*128, and 256\*256 for finetuning) and testing (1920\*1080). We are trying to further investigate multi-scale generalization approaches to address this issue. - **Improved Demosaicing Algorithms:** Similar to previous Bayer-based reconstruction methods, and for lightweight considerations, our current approach utilizes 3D convolution for the final demosaicing operation. However, the distinct structure of the quad-Bayer pattern makes it more challenging to demosaic compared to traditional Bayer arrays. This complexity can often result in color confusion and artifacts during the demosaicing process. **Recognizing this limitation, we have designed a lightweight demosaicing network that has already shown promising results in preliminary experiments.** We will continue to refine and explore this approach further.

Reviewer zshs2024-08-13

Thanks for the explanation, from which I can feel their frank and honest sense. While this work investigate a topic that might be interested in the area of SCI, it leaves too many aspects unconsidered, such as camera response, optical transmission, noise modelinng (Commented also by Reviewer yWFS), and mask discretization. This suggests that the research is at its early stage, and I would encourage the authors further investigate these issues.

Reviewer M9Ao6/10 · confidence 5/52024-07-15

Summary

This paper presents a method for compressive video image using Mamba-Unet for quad bayer sensors.

Strengths

+ The work seems to be in a less explored area of research.

Weaknesses

- The work is not making significant contribution in terms of method. Directly applying Mamba-Unet to this problem looks unnatural. Why not use a transformer or a CNN? I dont think there will be a much of a difference. - The writing needs improvement and section 3 is not that clear.

Questions

What is the primary motivation to address quad bayer sensors when they are not that prevalent? What is the need of using Mamba-Unet for this work?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are not addressed well.

Authorsrebuttal2024-08-11

Thank you for your review！

We sincerely appreciate your thorough review of our paper, careful consideration of our rebuttals, and raising the score. If you have any further questions or concerns, we are always available to provide additional clarification as necessary.

Reviewer zshs2024-08-08

1. The quad-Bayer pattern has some advantages, e.g., higher resolution and HDR capability, in convential camera imaging. However, these advantages are not necessarily inherited by the SCI configuration due to the mask modulation. Moreover, the current simulation of SCI pipline is oversimplified without considering optical effects in real scenarios. Therefore, it is crucial to have real-scene evidence to verify the significance of the SCI task with quad-Bayer. The manuscript lacks real experiments on videos captured by quad-Bayer patterns. All the results are obtained on simulated data. Results on real data are expected to evaluate the effectiveness of the proposed method. 2. Regarding the result of the "Swinger", I feel that the PnP-FastDvDnet method provides better visual quality than the proposed MambaSCI, given that both introduced some artifacts. While PnP-FastDvDnet presents some ghosting artifacts, MambaSCI brings obvious geometrical distortion: line structures are curved locally with also some ringing artifacts.

Reviewer M9Ao2024-08-10

Thanks for the detailed response. My doubts are cleared and such a revision should help. I have increased my rating after also reading other reviews. Good luck!

Reviewer xnVL2024-08-11

1. About comparison on long videos, when increasing T to 16 and 32, I cannot see the great advantage in run-time and performance over EfficientSCI. Could you please further explain this point? 2. The author claims the global-local design in the proposed framework compared to STMamba. However, as far as I know, there is also Depth-wise Convolution in ST-Mamba. Thus, this cannot be one of the innovations.

Authorsrebuttal2024-08-14

Summarization

As the discussion phase comes to a close, we would like to summarize our key points for the convenience of the esteemed reviewers and ACs. **New Task:** As far as I know, we are the first to introduce quad-Bayer arrays into the field of video SCI, which responds to the trend of smartphones becoming the mainstream video shooting devices and the mainstreaming of quad-Bayer arrays in smartphone cameras, and also compensates for the serious color distortion and ghosting problems that exist in the existing methods when facing quad-Bayer array videos. Meanwhile, numerous literatures have shown that quad-Bayer arrays can provide better image quality compared with Bayer, and we hope to broaden the research scope and reconstruction quality of video SCI. **New Approach:** For lightweight video reconstruction, we have pioneered the use of an asymmetric U-shaped architecture with customized Residual-Mamba-Blocks modules. This design enables efficient and high-quality video reconstruction, achieving superior results with fewer parameters and FLOPS compared to previous transformer and CNN-based methods, as demonstrated in **Table 2** of the main manuscript. **Real Dataset:** As the task is still in the exploratory phase, public datasets of this class are currently targeted. Additionally, constructing the necessary SCI encoding system requires specialized optical hardware and considerable time, which often exceeds the resources of researchers focused on decoding algorithms. We are committed to developing and refining this system to fully capitalize on the advantages of quad-Bayer over Bayer. We also extend our gratitude to reviewers **zshs** and **yWFS** for highlighting potential challenges in real-world applications, which will guide us in building a robust system. Finally, we sincerely thank all reviewers for their time and thoughtful reviews. We will carefully consider your constructive suggestions and continue to improve our work.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC