Summary
The present paper proposes a novel view synthesis approach that reconstructs 3D scenes from event camera streams without precise camera poses. The main goal is to handle the unique asynchronous and irregular data characteristics of event cameras, which pose significant challenges for traditional 3D reconstruction methods. By utilizing a tracking and mapping approach inspired by SLAM pipelines, the method estimates camera motion based on prior reconstructed 3D-GS scenes and incrementally refines both the camera motion and scene representation. It's capable of handling real-world scenarios with no gt. poses, offering improved performance compared to NeRF-based methods and event-based visual odometry. It efficiently renders high-quality brightness images, outperforming baseline methods in terms of novel view synthesis and motion estimation.
Strengths
Overall, this paper presents an incremental 3D Gaussian Splatting reconstruction algorithm from a single event camera, without requiring the ground truth camera poses. It has a motivation and is adequate for the audience and also solid on the technical side and adds the event-based VO tricks and off-the-shelf depth model for re-initialization to 3DGS. Thus, this work is interesting for readers working at the intersection of novel view synthesis and neuromorphic vision.
Weaknesses
1. The setup of the real-world experiments lacks validity. Specifically, for the evaluation on the TUM-VIE dataset, the qualitative results alone are insufficient. The authors should also include quantitative analysis using no-reference image quality assessment metrics to provide a more comprehensive evaluation.
2. Although the proposed method can operate using an event stream as input to reconstruct 3D Gaussians, it still relies on uniform event stream as input. The proposed method is, therefore, limited by the density of event data streams, which restricts its practical applications.
3. Despite the detailed comparison of the quality of rendered images, the efficiency of the training and rendering process is not included, which is an important metric of NVS methods. Extra comparisons with other methods on training time and inference FPS would help better evaluate the proposed method.
4. This method is valuable for addressing event-based visual odometry. However, the authors focus more on the NVS task, and using Gaussian functions to reconstruct grayscale scenes seems less relevant, as they are mainly suited for head-mounted devices, which reduces the method’s rationale.
Beyond this I have mainly minor comments and nitpicks:
l.117, the sentence contains a grammatical error and should be revised. Specifically, "IncEventGS conduct..." should be corrected to "IncEventGS conducts...".
l.142, the expression should be standardized by changing "se3" to "se(3)" for clarity and consistency.
l.162~186, I think the re-initialization process is vital to the method, but the main figure of the pipeline does not reflect this which may generate some confusion with readers not familiar with the method.
Questions
1. The re-initialization using the pre-trained depth model for regularization with the proposed Gaussian model is not clear. Can the authors provide more details about it? Especially regarding the visualization of the intermediate process.
2. For SLAM or VIO, the accuracy of the trajectory is crucial. However, for NVS (Novel View Synthesis) tasks, the proposed method merely reconstructing a gray map of the scene can diminish the significance of the task to some extent. It is not enough to work only on the gray map. Could we perform the NVS task on the RGB event dataset? For example, the dataset from [1] or [2] or the event-based color synthetic Replica dataset.
3. What's more, I noticed that the authors did not provide any supplementary materials. Could the authors provide some visual demos to better observe the overall effect of this method?
[1] Robust e-NeRF: NeRF from Sparse & Noisy Events under Non-Uniform Motion
[2] EventNeRF: Neural Radiance Fields from a Single Color Event Camera