bit2bit: 1-bit quanta video reconstruction via self-supervised photon prediction

Quanta image sensors, such as SPAD arrays, are an emerging sensor technology, producing 1-bit arrays representing photon detection events over exposures as short as a few nanoseconds. In practice, raw data are post-processed using heavy spatiotemporal binning to create more useful and interpretable images at the cost of degrading spatiotemporal resolution. In this work, we propose bit2bit, a new method for reconstructing high-quality image stacks at the original spatiotemporal resolution from sparse binary quanta image data. Inspired by recent work on Poisson denoising, we developed an algorithm that creates a dense image sequence from sparse binary photon data by predicting the photon arrival location probability distribution. However, due to the binary nature of the data, we show that the assumption of a Poisson distribution is inadequate. Instead, we model the process with a Bernoulli lattice process from the truncated Poisson. This leads to the proposal of a novel self-supervised solution based on a masked loss function. We evaluate our method using both simulated and real data. On simulated data from a conventional video, we achieve 34.35 mean PSNR with extremely photon-sparse binary input (<0.06 photons per pixel per frame). We also present a novel dataset containing a wide range of real SPAD high-speed videos under various challenging imaging conditions. The scenes cover strong/weak ambient light, strong motion, ultra-fast events, etc., which will be made available to the community, on which we demonstrate the promise of our approach. Both reconstruction quality and throughput substantially surpass the state-of-the-art methods (e.g., Quanta Burst Photography (QBP)). Our approach significantly enhances the visualization and usability of the data, enabling the application of existing analysis techniques.

Paper

References (42)

Scroll for more · 30 remaining

Similar papers

Peer review

Reviewer VmGM6/10 · confidence 5/52024-07-13

Summary

1. The paper introduces a novel self-supervised method for reconstructing high-quality image sequences from sparse binary quanta image data. 2. The paper mainly adapt a self supervised denoising algorithm called GAP. Instead of directly adopting the GAP method, the authors extended it to spatiotemporal structures and proposed a novel masked loss to address the correlation issue between input and target images. 3. Experimental results demonstrate that the proposed method substantially outperforms state-of-the-art techniques such as Quanta Burst Photography (QBP) in both reconstruction quality and throughput. 4. Additionally, the paper presents a new dataset and discusses the potential of the method for generalizing to other spatial event point processes beyond the specific application of quanta image sensors.

Strengths

1. The strengths of this approach mainly lie in its ability to effectively utilize spatiotemporal information, its novel masked loss function that addresses the correlation between input and target images, and its demonstrated superiority over existing methods like Quanta Burst Photography in terms of reconstruction quality and throughput efficiency. 2. In addition, as a self supervised algorithm, the training of this scheme does not require the construction of large-scale synthetic datasets, making it more flexible and convenient.

Weaknesses

1. In Equation 1, there is a confusion between x_{in} and x_{inp}. Moreover, the right side of Equation 1 seems to have overlooked x_{tar}. 2. To illustrate the application of QIS in high-speed and low-light scenarios, it would be beneficial for the authors to provide specific data on the rotation speed of the fan and the light intensity of the scene. 3. The authors used a 10-shot approach to calculate the average results. However, methods similar to GAP require many iterations during inference, and the combination of these two factors could lead to significant computational time. The authors should provide more detailed explanations, such as the number of iterations and the computational workload involved.

Questions

1. I still have doubts about the role of the masked loss. The authors state that this loss is intended to address the correlation issue between the input and target images. However, after convolution, the information from locations that were originally 1 in the input image has already spread to other pixel locations. Simply masking the original 1 positions may not fully conceal the information from those positions. 2. I am not quite clear about what the network's output is, given that the input is a t*h*w matrix. Is the output also t*h*w? If so, in the supervised loss comparison experiments, are there 32 grayscale images gt? For the N2N comparison experiment, is the supervision using QIS data with a shape of t*h*w? How was the GAP-2D experiment designed—does the network input consist of the first frame of QIS data from t*h*w? 3. Why does N2N lead to a granular pattern? Why can GAP-like method resolve the granular pattern caused by the N2N method? A good response to the Weaknesses and Questions will improve my initial rating.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

4

Limitations

There is no discussion of potential negative societal impacts.

Reviewer iaWM5/10 · confidence 4/52024-07-13

Summary

This paper presents a self-supervised method for reconstructing high-quality video from sparse binary quanta image data produced by single-photon avalanche diode (SPAD) arrays. The authors propose a novel masking strategy to handle the binary nature of the data and extend their method to 3D to leverage spatiotemporal information. They evaluate their approach on both simulated and real SPAD data, demonstrating improved reconstruction quality and throughput compared to existing methods like Quanta Burst Photography (QBP). The paper also introduces a new dataset of real SPAD high-speed videos under various challenging imaging conditions.

Strengths

1. The paper addresses an important problem in computational imaging, proposing a novel self-supervised approach for reconstructing high-quality video from sparse binary quanta data. 2. The authors' masking strategy to handle binary data is innovative and appears effective based on the presented results. 3. The introduction of a new real SPAD high-speed video dataset is valuable for the research community.

Weaknesses

The paper fails to convincingly demonstrate the effectiveness and advantages of the proposed method over existing approaches. The theoretical foundations are not well-developed, and the experimental results are not sufficiently rigorous or comprehensive to support the claims made. The writing lacks clarity in many sections, making it difficult to fully understand the proposed method and its implications. 1. The discussion on selecting the photon splitting variable p is confusing and lacks practical considerations. The paper doesn't adequately address how to ensure correct signal reconstruction when the signal level is unknown before capture. 2. Many crucial implementation details are absent: - The composition and size of the training dataset are not specified. - It's unclear if the quantitative results for simulated data are computed from only one video (L204). - The number of videos in the real test data is not mentioned. 3. The comparison with existing quanta video reconstruction methods is limited to only QBP (proposed in 2020). There is a lack of quantitative comparisons with other existing methods. 4. The paper lacks a detailed analysis of the method's runtime and memory requirements compared to existing approaches. 5. More extensive ablation studies exploring the impact of various components of the method (e.g., network architecture choices, hyperparameters) are needed to provide deeper insights into the method's performance. 6. Several technical terms and abbreviations are not properly defined or explained: - L1: "SPAD" is not defined on first use. - L15: The unit for "0.06 photons per pixel" is not specified (e.g., per frame or per second). - L22: "QBP" is not defined on first use. - L98: The meaning of s_i and i is unclear (intensity or pixel position?). - L112: The probability of zero photons hitting a pixel being e^{s_i} needs a detailed explanation. - L197: The term "10-shot inference" is not explained. 7. Some experimental details are missing: - L205: More information on the iPhone 15 slow motion mode (e.g., resolution, exact frame rate) would be helpful.

Questions

1. How do you propose to select the photon splitting variable p in practice when the signal level is unknown before capture? Can you provide guidelines for selecting an optimal value? 2. Can you provide a more comprehensive comparison with other existing quanta video reconstruction methods beyond QBP, including quantitative results? 3. Can you provide a detailed analysis of the computational complexity and how it scales with dataset size? 4. Could you elaborate on the runtime and memory requirements of your method compared to existing approaches? 5. Can you provide more extensive ablation studies on the impact of various components of your method, such as network architecture choices and hyperparameters? 6. Can you provide more details on the composition and size of the training dataset used? How many videos were used in both simulated and real tests? 7. How does the method perform on extremely low photon count data, and what is the lower limit of photon count where the method remains effective? 8. How does the method perform on different types of scenes or motion patterns?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors have adequately addressed limitations, discussing the scope of applicability to Poisson noise and computational considerations. They have also provided a thorough discussion of assumptions and potential limitations in the conclusion.

Authorsrebuttal2024-08-14

We thank the reviewer for their comments. Regarding our used datasets, please note that we commit to publishing our datasets. We agree that the suggested clear and to-the-point summary of the used data can be helpful. We will add a sentence, in the form suggested by the reviewer, stating the number of videos and frames in the main paper: "To evaluate our method, we use a total of 7 real SPAD videos, each containing 100k-130k frames, and a synthetic video with simulated noise consisting of 3990 frames. Additionally, we use a real video with 100k frames published in [8]". We additionally will point to the description of the simulation in L199 “Simulated data” section. We will also add a table to the supplementary material describing the characteristics of each video including frame rate, number of frames, frame size (resolution), and subject content. However, we would like to stress that our manuscript includes experiments on a total of 9 (7 real + 1 synthetic + the video from [8]) videos, each showing vastly different content including low-signal, high-signal, high-contrast, high-ambient-light, moving camera, moving object, linear movement, random movement, combined movement, ultra-fast events, and stochastic events. The real data include dynamically moving objects, plasma balls, high-frequency bubble dynamics, histology images, and fluorescence microscopy data. The simulated data was carefully taken to cover a large range of image contrast and structures. We will include results on additional simulated data from other domains, including microscopy, in the supplementary material of the final version.

Reviewer Z4Sm6/10 · confidence 3/52024-07-14

Summary

The paper proposes to extend the Generative Accumulation of Photons that was proposed for Poisson noise to 1 bit Quanta image sensors.

Strengths

The proposed method is novel, and mathematically interesting. The authors have put in a lot of effort to fit GAP for the problem of 1-bit QIS reconstruction.

Weaknesses

The proposed method does not seem to be performing better than a supervised method. It is not clear what is the supervised method that was used in the comparisons. It is not clear what the unique advantages of this method are compared to standard techniques like supervised learning or data simulation based training.

Questions

Wouldnt randomly assigning each photon event to one of the two bins lead to loss of temporal information to some extent in both the input and target? The supervised method seems to be performing better than the proposed method on static simulated scenes? Would it achieve similar fps as the proposed method on the video data too?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes. The authors have made the limitations of the proposed method clear.

Reviewer abBE8/10 · confidence 4/52024-07-15

Summary

This paper introduces a method to reconstruct/denoise high-resolution high-frame-rate videos captured by 1-bit quanta imaging sensors (e.g., SPAD arrays) without heavy spatio-temporal binning. The paper also captures and will release a new SPAD dataset. The proposed method is loosely based on Generative Accumulation of Photons [9], which trains a CNN to reconstruct images in a self-supervised fashion by reconstructing the pdf of the photon arrivals. This paper makes several significant modifications to [9] so that it can work with quantum image sensors: First, in order to account for 1-bit sensors, the proposed method models the photon arrival pdf as a Bournoulli (rather than Poisson) distribution to account for the binary nature of the measurements. Second, in order to improve reconstruction accuracy it incorporates temporal information regularization. The proposed method is tested on experimentally captured data and produces visually compelling results, including in the presence of non-rigid motion (e.g., guitar string).

Strengths

A SPAD dataset would be extremely valuable to the research community. Proposed method is novel and works well. Paper is well written. Denoising quanta image sensor data is an important problem with many scientific imaging applications

Weaknesses

Proposed technique largely follows from GAP. Unclear if Poisson distribution assumption is invalid in the photon starved regime, i.e., would GAP work for dimmer scenes? Figure/table captions could use more info. E.g., state whether Table 1 results are experimental or simulated. Underperforms supervised methods.

Questions

Does a Poisson signal model work in the low-flux regime (where it would be rare for more than one photon to arrive in a pixel)? Can one bin a large # of frames in overlapping windows to ensure Poisson statistics while still recovering high temporal resolution? How important is self-supervised training? Is the simulated noise model accurate enough that one could train an effective QIS denoiser using only simulated data?

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

Well discussed

Reviewer VmGM2024-08-13

Thank you for the kind reply, most of my concerns have been addressed. I am willing to raise the score to match the contribution of this paper.

Reviewer iaWM2024-08-13

Thank you for addressing my concerns. While the reorganization of the response clarifies some aspects, it makes it difficult to follow my original concerns. Therefore, I've outlined my remaining comments below, following the original numbering for clarity. 1. [Comments] I am satisfied with the explanation provided for selecting the photon splitting variable p, which significantly impacts the method's applicability. 2. Many crucial implementation details about the dataset are absent. [Comments] The response lacks crucial implementation details, particularly regarding dataset statistics. In lines 208-211, more specific information about the dataset size is needed, such as: "The proposed dataset contains XX real videos and XX synthetic videos, each with XX frames. Synthetic sequences were simulated from XX dataset videos using XX method." While Figures S5-S12 provide examples of real SPAD data and reconstruction, I am seeking a comprehensive statistical profile, not just illustrative examples. Without this information, the evaluation may be biased if it is based on a single video. 3. The comparison with existing quanta video reconstruction methods is limited to only QBP (proposed in 2020). There is a lack of quantitative comparisons with other existing methods. Can you provide a more comprehensive comparison with other existing quanta video reconstruction methods beyond QBP, including quantitative results? [Comments] The theoretical foundations of N2N, N2V, and GAP do not restrict their application to specific data dimensions. If I understand the proposed method correctly, it shares this flexibility. Therefore, the applicability to 3D data is not a unique contribution of the proposed method. Additionally, QBP can be adapted for 3D data by applying a densely sampled sliding window, which further emphasizes the need for a broader comparison. 4. The paper lacks a detailed analysis of the method's runtime and memory requirements compared to existing approaches. Can you provide a detailed analysis of the computational complexity and how it scales with dataset size? Could you elaborate on the runtime and memory requirements of your method compared to existing approaches? [Comments] To clarify, I am seeking a fair quantitative comparison of inference computational complexity between different methods under the same input data size and hardware requirements, as commonly found in the literature. This typically involves evaluating the minimal memory and runtime required for inference. The current discussion only mentions the maximum GPU memory, total training time, and a rough estimate of minutes used without any constraints. 5. More extensive ablation studies exploring the impact of various components of the method (e.g., network architecture choices, hyperparameters) are needed to provide deeper insights into the method's performance. Can you provide more extensive ablation studies on the impact of various components of your method, such as network architecture choices and hyperparameters? [Comments] As stated in line 57, one of the paper's main contributions is providing insights into network design. However, Table S1 only lists the selected hyperparameters, not the results of ablation studies. The only relevant ablation study is Table S7, which examines the effect of model size. 6. Several technical terms and abbreviations are not properly defined or explained: - L1: "SPAD" is not defined on first use. - L22: "QBP" is not defined on first use. [Comments] I mean the first use in the main text, not the abstract. My suggestion is that the terminologies should be defined in the main text when they are used for the first time, even if they appear in the abstracts. 7. Some experimental details are missing. - How does the method perform on extremely low photon count data, and what is the lower limit of photon count where the method remains effective? [Comments] The statistics provided are about the simulation. The effectiveness means quantitative results demonstrating the method's performance under different illumination levels measured in lux. - How does the method perform on different types of scenes or motion patterns? [Comments] Can you provide an related analysis including insights into potential limitations or guidelines for practical applications?

Authorsrebuttal2024-08-14

Response to the reviewer iaWM's new comments

We appreciate the reviewer’s last-minute response. We want to note another factual error in this comment: 6. **I mean the first use in the main text, not the abstract. My suggestion is that the terminologies should be defined in the main text when they are used for the first time, even if they appear in the abstracts.**\ QBP is also defined on first use in the main text (L291).\ We acknowledge that SPAD is not defined in L25. The initial definition is on the same page. We will fix it in the revision. Please find other responses below: 2. **The response lacks crucial implementation details ...**\ We stated that each real SPAD dataset has 100-130k frames in L208. Each data set has a single video. We can provide other required information in the revision. We are not sure what the term “comprehensive statistical profile” means in this context. 3. **The theoretical foundations of N2N, N2V, and GAP ...**\ We acknowledge that most convolutional neural networks can be applied to multidimensional data. In fact, we tested 3D N2N, N2V, and GAP with the same network architecture as shown in Table 1. However, directly applying these methods in 3D to 1-bit SPAD data produced suboptimal results. We also applied masked loss to 2D GAP (Table 1). It did not produce comparable results either. The combination of the 3D implementation, masked loss, and p-thinning-based sampling strategy, together with important architectural design choices such as group normalization, is the key to high-quality 1-bit quanta image reconstruction. The main contribution of this work is the combined concept and approach. While QBP can be adapted for 3D data by applying a densely sampled sliding window, the original manuscript indicates that speed (30 minutes/frame) is a major limitation as noted in L295. This is not a practically viable approach for reconstructing thousands of densely sampled sliding windows. Additionally, QBP works in a fundamentally different way than our method and we are not aware of how to make fair quantitative comparisons. It registers and bins adjacent frames, therefore the output is an accumulation of aligned images instead of a direct volume prediction. Despite that, we tested real SPAD data from QBP with our method and reported the result in Fig. 5. We are not aware of any method beyond QBP for a broader comparison. 4. **To clarify, I am seeking a fair quantitative comparison ...**\ There are no differences in inference computational complexity for the 3D methods compared in Table 1. The only difference is the self-supervised training strategy. In L214, we mentioned “to ensure fair comparisons, we incorporated the same network architectures, hyperparameters, and training steps into individual baselines…” Therefore, we believe we provided a fair quantitative comparison of inference computational complexity between different methods under the same input data size and hardware requirements. Also, this work highlights a new self-supervised concept for Bernoulli distributed data. Still, we provide a network design that is readily useful within the defined constraints. We acknowledge the network performance can be further improved using optimal architectures and hyperparameters, but it is beyond the scope of this work. The numerical information we provided in Table S1, L192, and L298 can be used to estimate the general runtime requirements in different situations by scaling the parameters. 5. **As stated in line 57, one of the paper's main contributions ...**\ We conducted extensive ablation studies besides Table S7. Table S6 and Fig. 4d compare the initial filter size and indicate both large and small sizes can degrade the performance of the network. Fig. 4a and L261 compare the group normalization size, which is key for quality reconstruction. We also consider p as a highly relevant parameter in our method, and we conducted thorough ablation studies with different p ranges in Fig. 4b, c and Table S2-5, 8. We also included results with many different experiments in hp.CSV as noted in the Table S1 caption. All of the studies include numerical results. It is unclear which specific ablation studies the reviewer wants. \ 6. **See above** 7. **The statistics provided are about the simulation ...**\ We will clarify that the pixel values indicate the absolute illumination level (photon count). SPADs are very sensitive devices and work at the single photon level. Lux is a perceptual measurement of visible illuminance, which is not an appropriate measurement. For example, SPAD measures the spectrum through UV-NIR. Lux only accounts for visible light.\ The requirements/assumptions for applying this method are discussed in L49-51. In extreme cases, as shown in Fig. S4, we applied the method to a random binary array with 1% of the pixels occupied with ones. The result is an expected output of a blank image. One limitation of the method is that if the pixels are not independent, the predictions can be erroneous.

Reviewer iaWM2024-08-14

Thank you for addressing my concerns and providing clarifications. The term 'comprehensive statistical profile' means exactly 'more specific information about the dataset size is needed, such as: "The proposed dataset contains XX real videos and XX synthetic videos, each with XX frames. Synthetic sequences were simulated from XX dataset videos using XX method." ' Evaluating a method on a single video can lead to limited generalizability and biased results, which is unusual in the community. Therefore, I still have concerns on the evaluation conducted on a single video and the proposed datasets that each contains a single video. Robust evaluation necessitates using diverse datasets with multiple videos to demonstrate a method's effectiveness adequately. I would appreciate the authors' response to this specific concern. How do you plan to address this potential limitation and ensure the generalizability of your method? I am satisfied with your responses regarding the remaining points. I will raise my rating accordingly based on your responses and the promised incorporation of clarifications in the revision.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC