Weaknesses
- Statement
- Line 27-31
- Although the manuscript does acknowledge certain NeRF-based SLAM methods for their ability to provide dense 3D reconstruction, relevant references [1,2,3] are conspicuously absent. Additionally, some NeRF-based SLAM methods that utilize prior for pose estimation have been glossed over.
- Line 34
- The cited works do not adequately represent the body of literature since both methods focus exclusively on depth map estimation. This oversight should be rectified by including relevant citations [4,5].
- Line 143
- wrong format
- Motivation
- The motivation for introducing single view generalizable nerf for relative pose estimation is unclear and wired.
- Line 142
- the definition of the x_t. In equ2, x_i is the sampled points along the ray. Does x_t is the S(r) in equ2?
- while pixelnerf can use only a single view to render a novel view, it is much like a mono-depth estimation module. Without optical flow, an alternating solution uses iNeRF to align two single views point cloud/nerf, minimizing photometric loss. However, even most SOTA mono-depth estimation methods are insufficient to estimate accurate relative pose due to cross-view scale misalignment. Some unposed/posed Nerf only use mono-depth as a reg term. This is why the proposed method **HAS TO** introduce the out-of-shelf optical flow estimation for pose estimation. If the single view pixelnerf is removed from the proposed method, optical flow, as the correspondences, is enough to estimate the essential matrix/relative pose between two neighbor frames. Furthermore, the deep-sfm module, like banet, droidslam, deepv3d, all of these methods take multiview images to estimate both multiview depth map and the camera pose at the same time. The author should carefully address this issue during the rebuttal.
- the motivation of flow confidence weights.
- Line 143
- This part could be easily solved by bi-direction consistency check with optical flow. Introducing another pretrained network for weight prediction will make the generalization issue worse.
- Figure 9 does not show the results with dynamic objects.
- Line 178
- RGB VO/SLAM confronts scale ambiguity problem. The proposed method doesn't handle it at all.
- Line 194
- where is the definition of N, J?
- Line 203 Equ8
- The mono-depth is enforced by this loss term during fine-tune. As mentioned above, the optical flow for relative pose estimation is enough.
- This loss term does not handle singular solutions.
- Experiment
- Line 211
- the training detail is missing.
- the proposed method cannot run on a long video clip.
- Most of the experiments show the object-center case. It will be great to see the results on the inside-out case either.
- Although the authors mention that they do not claim sota accuracy of pose estimation, the metrics of the pose are still necessary to be reported on the main paper, including droid slam, droid vo, orb slam, etc.
- Line 224
- test-time optimization on the weight of CNN and MLP on a single video clip in a self-supervised manner is extremely hard. The problem will become worse if the precomputed optical flow is removed. The author should provide more detail about this section.
- Line 250
- more baseline methods are required, such as nicer-slam[2], dim-slam[3], droid-slam.
- some unposed nerf methods with mono depth estimation module, such as nope-nerf[6]
- and another test-time optization method, such as CVD[4], RCVD[5].
- The pose trajectories in the figure are misaligned in both the main paper and the supplementary, which are very confusing.
- reference
- [1] Sucar, E., Liu, S., Ortiz, J., & Davison, A. J. (2021). iMAP: Implicit mapping and positioning in real-time. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 6229-6238).
- [2] Zhu, Z., Peng, S., Larsson, V., Cui, Z., Oswald, M. R., Geiger, A., & Pollefeys, M. (2023). Nicer-slam: Neural implicit scene encoding for rgb slam. arXiv preprint arXiv:2302.03594.
- [3] Li, H., Gu, X., Yuan, W., Yang, L., Dong, Z., & Tan, P. (2023). Dense RGB SLAM with Neural Implicit Maps. The Eleventh International Conference on Learning Representations(ICLR)
- [4] Luo, X., Huang, J. B., Szeliski, R., Matzen, K., & Kopf, J. (2020). Consistent video depth estimation. ACM Transactions on Graphics (ToG), 39(4), 71-1.
- [5] Kopf, J., Rong, X., & Huang, J. B. (2021). Robust consistent video depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1611-1621).
- [6] Bian, W., Wang, Z., Li, K., Bian, J. W., & Prisacariu, V. A. (2023). Nope-nerf: Optimising neural radiance field with no pose prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4160-4169).