Transient Neural Radiance Fields for Lidar View Synthesis and 3D Reconstruction

Summary

The paper brings NeRFs to a new dimension of imaging at transient timescales enabling the rendering of transient imagery. It incorporates the underlying image formation model of the lidar and recovers improved geometry and conventional appearance when training on a few input viewpoints. The paper also proposes a first-of-its-kind dataset of simulated and captured transient multiview scans from a prototype single-photon lidar and tests the transient NeRF on the dataset.

Strengths

The paper brings NeRFs to a new dimension of imaging at transient timescales and proposes a first-of-its-kind dataset of simulated and captured transient multiview scans from a prototype single-photon lidar. To deal with the new data type, HDR-Informed loss function and Space carving regularization are proposed to further improve the performance. The experiments are sound compared with other NeRF-based methods. The full paper has a clear structure and a clear narrative.

Weaknesses

It only tests in a dark environment because of the single-photon Lidar. It is hardly used in broad application scenarios.

Questions

Do you do the ablation study of HDR-Informed loss function and Space carving regularization? Can the method work under natural light?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

n/a

Summary

The paper describes a variant of NeRF specially designed for the image formation model of a time-of-flight imaging sensor. In contrast to earlier works using ToF sensors in a neural radiance field setting, the method is not designed to merely take depth maps or point clouds from the ToF sensors as input. Instead, the formulation goes one level deeper and presents a NeRF and image formation model on the basis of the time resolved photon count histograms that are used as basis for the final depth measurement by the ToF camera. To this end, the authors propose a time-resolved version of the volume rendering equation for image formation, as well as adapted HDR aware loss functions for training the neural radiance fields on photon histograms. The approach is tested on simulated data as well as real data captured with a ToF prototype setup. The authors show that in particular in a sparse view setting, the new photon count based formulation has advantages over depth-supervised or image-based NeRFs. Overall, I think this is a strong paper showing very innovative ideas. I enjoyed reading it.

Strengths

Overall, I think this is a strong paper. It introduces quite a few clever ideas on how to adapt the neural radiance fields concept to the image formation model of a time-of-flight camera. This ranges from adapting the image formation model to a time-resolved version, to introducing a HDR aware loss and a space-carving regularization tailored to the peculiarities of the imaging properties of the employed sensor. To my knowledge, this is the first paper approaching neural radiance fields in this way. As the authors suggest in their discussion the formulation on the basis of time resolved photon histograms may enable additional improvements in 3D reconstruction, e.g. of complex shapes, material properties etc.

Weaknesses

Not much to say here. One point that could have been discussed more is from what number of viewpoints on the traditional or depth-supervised approaches catch up. At the moment, advantages of the approach are mostly shown in the sparse view setting, which is fine and showcasing an advantage of the approach.

Questions

see the weaknesses section One issue with time-of-flight sensors in general is their often non-trivial noise characteristics. Would explicitly incorporating the noise model into the formulation help here ?

Rating

9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The discussion in the paper is adequate.

Summary

NeRF is a method that has recently become popular for view synthesis. It allows estimation of a 3D scene density and 5D radiance map using a few intensity images and their camera poses. These can then be used to render intensity and depth images from any novel 2D view. This paper extends NeRF to allow estimation of scene density and radiance from “scene transients” instead of intensity/depth images. Previous methods have tried to incorporate depth information into NeRF in the form of point clouds, but the proposed method directly uses raw scene transients (histogram of photon counts) captured by a SPAD sensor. This method can then be used to render scene transients given a novel 2D view. The authors' main contribution is to extend the volume rendering equation of NeRF to produce time resolved measurement instead of integrated over time. Additionally their training/rendering takes into account the shape of the laser pulse, spatial footprint of the laser spot etc. And lastly they add modifications to the NeRF objective to ensure their method works well for the high dynamic range SPAD data (predict radiances in the double log space, and apply loss in the log space). The authors also apply a regularization (space carving) to encourage the density estimates to be sparse (similar to what other NeRF variants have done e.g urban NeRF). The authors also capture a real world multiview LiDAR dataset with their hardware prototype. Using this dataset and another simulation based dataset, they provide quantitative and qualitative comparison of their method with other baselines for intensity and depth reconstruction from novel views. The results from simulation data show their method’s superior performance. The results from captured data also show that their method is superior, but by a smaller margin (which they attribute to alignment imperfections in the captured dataset).

Strengths

1. This is a nice extension to the NeRF setup that allows directly leveraging the raw lidar measurements to learn a 5D representation instead of learning from intensity/depth images which won't contain all the information available in the raw measurements. The improvement in estimation accuracy is evident in the results/experiments shown. 2. The paper is written very well, with good attention to details (e.g. modeling assumptions, experiment setup). Background work is explained well. 3. Authors perform exhaustive experiments with baseline methods using both simulated and captured data, and results look impressive.

Weaknesses

From a novelty of algorithm standpoint, the contributions of the paper are limited. From what I understood, It is a straightforward application of the NeRF framework to a new domain, i.e, the authors represent the scene density/radiance just like in NeRF, the only difference is that they modify the rendering logic to produce 3D histograms instead of 2D images. The image formation model of SPADs (which is used in the rendering equation) is also not novel.

Questions

1. You mention that you model the spatial footprint of the laser/sensor spot when rendering the transients. Is that necessary? Does performance deteriorate if you just assume an ideal spot? 2. I'm not convinced about the utility of rendering raw lidar measurements from novel viewpoints. Other than generating simulation data, are there any practical applications?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

Yes

Summary

This work proposes a novel-view synthesis method for active sensors (single-photon LiDAR sensors) based on the Neural Radiance Field formulation. To this end, the volume rendering formulation for active sensors is derived, properly taking into account the measurement formation process (two-way distance, intensity fall-off) of LiDAR sensors. Along with that, a new loss functions for empty-space carving is proposed. The proposed method is implemented based on the INGP NeRF model and evaluated on both synthetic dataset of transient multiview scans (one of the contributions of this work), as well as on real-world data that was captured using a prototype single-photon LiDAR system. In these experiments, the proposed method outperforms SoTA NeRF methods (also with depth supervision) across all metrics.

Strengths

- Original and important problem formulation. Novel-view synthesis for active sensors is an under researched, but a very important problem. - A volume rendering formulation for active sensors that properly takes (at least some of the) the properties of the measurement formation process into account. These include, two-way path when computing the transmittance, intensity falloff, beam divergence. - Simulated dataset of transient multiview scans that will be made publicly available (along with the scripts used to generate it) - A dataset of real-world captures that were acquired using a prototype single-photon LiDAR which, if I understand correctly, will also be made publicly available

Weaknesses

There are two main weaknesses in my view: - **Clarity**: The clarity of the paper could in my opinion be improved, and the main confusion stems from including the RGB images from simulated data in figure 1 and using **c** to denote number of photons (?) in Figure 2. I might have misunderstood something, but if I am not wrong, the actual LiDAR system can only measure the photon count and depth (time-of-flight) and the volumetric rendering formulation should be tied to that. However, in the Equation 3 the radiance is a vector quantity (RGB color?) same as in L150? The simulated data seems to be full RGB (what is the reason for this?), does that mean that for the simulated data the evaluation is actually similar to the original NeRF (PSNR and LPIPS computed over the RGB images?). - **Experimental evaluation**: In my perception, the experimental evaluation seems somewhat biased. The proposed method is compared to the following baselines *INGP*, *DS-NeRF* and *UrbanNeRF* which are supervised through color/intensity supervision (and *DS-NeRF* and *UrbanNeRF* also using the depth). However, the depth supervision only constrains the integral over the density and the supervision signal might even be very noisy (log-matched filter in case of real world data are usually very noisy). On the other hand, the proposed method is supervised with the full histograms per each ray (before integration) which provide additional supervision and helps to constrain the empty space. This is especially important in the low-view setting that was for some reason selected in the evaluation (2-5 views on synthetic data?). I am wondering why the low-view setting was selected for simulated data? In my opinion this pronounces the bias as it emphasizes the differences in the supervision signal between the methods

Questions

- What exactly does **c** denote in Equation 2 and L150? - I like that the formulation tries to follow the measurement formation process, but currently only the intensity fall-off and round-trip are considered. Would it make sense to add at least the effects of the incidence angle? - Similar to the comment above: why does simulated data simulate RGB and why is the evaluation considering low-view setting? - I also like that the beam divergence is modeled through sampling multiple rays (L180), but it would be good to clarify at which point in the volume rendering formulation the contributions are averaged? - The depth of the proposed methods is obtained as the argmax across the histogram, however for the baselines the integral along the ray is used. Does that yield better results as taking the argmax of the volume density profile?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations are discussed in the conclusion section, while broader societal impacts are not (but I also don't believe that there are any significant ones).

Response to the rebuttal

I would like to thank the authors for providing a detailed response. Some of my concerns were addressed, but I would still like to note a couple of things. **Clarification on dimensionality of symbol in rendering equations.** It would be good to make this clear in the paper (for other scalars you use non-bold font, so this might be confusing to others as well). If I understand things correctly (it might be that I don't, I am not an expert on the photon lasers), the photon count output of your network is strongly correlated with the density (module the differences in material and translucence). It would be good to discuss this in the paper and provide some histograms of both density and photon count ($\textbf{c}$) outputs of your network. **Comparison to baselines.** I would respectfully disagree that the comparison to the baselines is completely apples-to-apples. While the raw data is indeed the same, the baselines methods only have access to the data after the log-matched filter, which reduces a histogram to a single value representation (obviously a lot of information is lost here). The histogram data is especially useful in the sparse view setting as it provides **spatial supervision** that helps to constrain the solution (empty space supervision) which other methods don't have access to. I would even go as far as to argue that the main insight ***raw measurements provide better supervision*** was "known/suspected" before. Baseline methods do not use spatial data as it is simply not available by most consumer grade LiDAR systems (full-waveform LiDAR are rare in robotics, AV and other typical use cases). In fact, due to the missing spatial data, previous methods try to reason about it using heuristic observations, e.g. UrbanRadianceFields introduce line-of-sight priors about empty space and center a Gaussian at the measured depth to obtain additional supervision. Do not get me wrong, I think that this work is good, and the insights are interesting, but I would wish that they would be presented in a bit more "humble" (for the lack of better term) way and also discuss the availability of such data to prior methods and its effect on the evaluation. I am convinced that the previous works would also try to use full histogram data if it would be available to them (incorporating manually defined spatial priors based supports this). Nevertheless, the proposed formulation is interesting and might spark new ideas in the future.