A Global Depth-Range-Free Multi-View Stereo Transformer Network with Pose Embedding

Summary

This paper proposes a depth-range free MVS network that considers the information within the MVS framework. The initial depth is derived from the epipolar geometry of the reference image and the source images. In addition, the disparity features are enhanced by an uncertainty encoding module and a multi-view disparity attention module. The proposed method is assessed on several benchmarks compared to existing works. It shows promising results regarding the robustness to different depth ranges and the overall performance compared to RNN-based methods.

Strengths

1. The evaluation of the proposed method shows better results than the existing RNN-based methods on several public benchmarks and it is more robust to different depth ranges.

Weaknesses

1. The methodology lacks some clarity. For example, a) no explanation for $\vec{f}$ in equations 3 and 4, b) no explanation for $0$ and $1$ in $p^{0}$ and $p^{1}$ for the search range, c) the dot before $\varphi4$ in equation 8, d) how $H_{di}$ is initialized in equation 8, e) in equation 10, is $R^{0,i}$ the relative rotation between two images, or generated by the intersection angle $\theta^{0,1}$, f) no explanation for $t_{c}$ and $t_{f}$ in equation 11. Readers in the same research field might get the actual meaning without indication, but the broader audiences need to be considered. 2. There are problems with writing. For example, a) wrong reference for DispMVS, b) table 2 is indexed before tables 3 and 4 but it is mentioned after them in the main text, 3) missing reference of a section in the last paragraph in section 1.

Questions

1. In Figure 1, when the depth range is clipped within the actual depth range, e.g., the first column, how can you still get the depth for the foreground (425mm -- 552.5mm) and background (807.5mm -- 935mm) points? It seems that the proposed method yields the same results for all configurations as the four figures look the same. 2. When sampling M points to construct the cost volume, what is the interval between the points and how it is decided? Is it adaptive during the iterations?

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

1. It is suggested that the authors refine the writing of the methodology section and make clarifications regarding W1 and Q2.

Summary

This paper proposes a depth-range-free Multi-View Stereo (MVS) method, which iteratively updates the depth using a GRU-based approach. To eliminate the dependency on depth priors, the paper improves the depth initialization method of DispMVS. To fully utilize multi-frame information, the paper encodes the observation information between multiple source frames and the reference frame into features and proposes a Multi-view Disparity Attention module for multi-frame information fusion. To enhance the preservation and utilization of geometric information, the paper introduces 3D pose embedding, uncertainty estimation, and disparity hidden states. The paper has been tested on the DTU and T&T datasets, and the experimental results show that the method exhibits robust depth estimation results in the absence of depth range priors.

Strengths

-The motivation for the model design is clear, and the experimental results to some extent reflect the effectiveness of the model. -The experimental results demonstrate that the method can achieve good predictive results without relying on depth priors.

Weaknesses

-Poor writing quality: The consistency of the symbols in the paper is poor, such as the left side of Eq. 5 should be V_i(p0); The Fd_i in Fig. 2, Eq. 8, and F^d_i in Sec 3.4, the H_i in Fig. 2 and Hd_i in Eq. 8; the use of symbols is not standardized, such as the representation of matrices and vectors, the authors are advised to carefully check the entire text; In Eq. 8, there is an extra dot multiplication symbol before phi_4. -Missing method description: For the final GRU update part, the paper lacks a complete description. My current guess is that the authors use Fd_i and context features as input information to update the hidden state not mentioned in the paper, and decode the residual from this hidden state. -Incorrect key citation: There is an error in the citation of one of the main comparative methods, DispMVS[5], and the authors are advised to carefully check the reference list. "Rethinking the multi-view stereo from the perspective of rendering-based augmentation." -> "Rethinking Disparity: A Depth Range Free Multi-View Stereo Based on Disparity" -The application of cross-attention is strange: The cross-attention in this paper is only calculated at the same pixel positions in different frames, and the paper emphasizes that there are few continuous smooth trajectories between different frames in MVS, making this design very illogical. The authors are expected to provide a detailed explanation. -Missing comparisons: CER-MVS [Ⅰ], as an early GRU-based MVS work, should be included in the comparison. Other works such as MVSFormer series[Ⅱ, Ⅲ], GeoMVSNet [Ⅳ] should not be ignored either. -Incomplete ablation experiments: There is a lack of separate ablation experiments on uncertainty and disparity feature hidden states, and the existing ablation experiments couple the two together. There is a lack of ablation experiments on the MDA module. [Ⅰ] Ma, Zeyu, Zachary Teed, and Jia Deng. "Multiview stereo with cascaded epipolar raft." ECCV 2022. [Ⅱ] Cao, Chenjie, Xinlin Ren, and Yanwei Fu. "MVSFormer: Multi-view stereo by learning robust image features and temperature-based depth." arXiv preprint arXiv:2208.02541 (2022). [Ⅲ] Cao, Chenjie, Xinlin Ren, and Yanwei Fu. "MVSFormer++: Revealing the Devil in Transformer's Details for Multi-View Stereo." ICLR 2024. [Ⅳ] Zhang, et al. "Geomvsnet: Learning multi-view stereo with geometry perception." CVPR 2023.

Questions

-The number of Fd_i encoded by the authors is related to the number of source frames. From Fig. 3, it seems that during the GRU update process, they are directly concatenated, which will cause the channel number to be related to the number of source frames. So, if the model uses 5 frames in training and then the number of frames cannot be changed during testing, is that correct? -From the numerical results and method description, the paper has completely abandoned the use of depth priors now. But if the depth prior is known and the prior is more compact than the depth range calculated in Sec 3.2, can using the depth prior improve the performance of this method? -As a GRU-type method, how does the accuracy of the estimated results change with the number of iterations?

Rating

5

Confidence

5

Soundness

2

Presentation

1

Contribution

3

Limitations

-Limited performance: From the experimental results, the method in this paper is not outstanding on the benchmark, and the gap with the state-of-the-art methods is quite obvious, even only compared with GRU-based methods (such as CER-MVS). -Limited novelty: The method in this paper is more like a combination of existing modules.

Summary

- The author proposes a depth-range-free multi-view stereo framework that simultaneously takes into account all the source images. - The author has specially designed a 3D pose embedding to better encode specific geometric information. - The Multi-View Stereo method proposed in the paper achieves more robust depth estimation by cascading Disparity Feature Encoding, Multi-view Disparity Attention, and GRU Updating, and has demonstrated good performance across multiple datasets.

Strengths

- The depth-range has always been an issue affecting the robustness of Multi-View Stereo algorithms. Although previous methods have attempted to address it, they are still imperfect. The method proposed by the author exhibits technical novelty. - The iterative updating approach proposed by the author can simultaneously take into account the information of all source images. - The method presented in the article has shown good performance on the dataset's benchmark.

Weaknesses

- The visual results of the method proposed in the article exhibit many floater artifacts. - The overall description of the network architecture needs further refinement. The writing need to be improved. The current introduction is not well-balanced in detail, and there are several typographical errors with symbols. - The use of symbols like $F^d_i$ is inconsistent in terms of superscripts and subscripts, making it difficult to understand and appearing unprofessional in typesetting. This is not the only symbol with usage issues; multiple symbols have similar problems. It is recommended to provide a comprehensive explanation of the symbols in the supplementary material and to revise the symbols used in the figures and text.

Questions

- Are the training results of the epipolar disparity flows truly meaningful? Are there any visual results to demonstrate this? - How is $D_0$ constructed? During construction, were all source images initialized with the reference image? Was $D_0$ calculated as an average of initial values obtained from each source view? The explanation could be clearer. - How is the depth updated through the GRU? The section 3.5 is not clearly explained. - The ablation study is not sufficient: - What are the different initializations of depth, such as random or all zeros? - How is the convergence? - In Figure 2, what is the relationship between the $e_i$ output from the GRU and the $e_i$ after fusion? How is $D_i$ obtained from $e_i$, and how is $U_i$ combined? - As a method that also uses GRU for updates, there is a lack of reference to and discussion of "Multiview Stereo with Cascaded Epipolar RAFT."

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The author has already addressed the limitations of the method.

Summary

The paper proposes a depth-rage free method for MVS. It leverages the transformers for designing a global-aware model, with using pose positional embedding to guide the model and also predict the uncertainty at the same time. The methods demonstrates good performance on diverse datasets and is also robust to different depth range.

Strengths

- The idea of building global-aware models using transformers makes sense. - I like the idea of injecting inductive bias using pose positional embedding and the uncertainty method. - The proposed method demonstrates satisfying experimental results and robustness to different depth range.

Weaknesses

- Related work. Using transformers for building global-aware model is very common for 3D reconstruction now [1,2,3]. I would be good if the authors could discuss these related works. [1] Wang, Peng et al. “PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction.” ICLR 2024. [2] Jiang, Hanwen et al. “LEAP: Liberate Sparse-view 3D Modeling from Camera Poses.” ICLR 2024. [3] Zhang, Kai et al. “GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting.” ArXiv 2024. - Dense view inputs. The experiments are performed in the sparse-view setting where the computation of transformers is not a big problem. Is the method also efficient in the dense view setting? If not, I hope the authors could have more discussions in the paper. - I would say that the gains in the ablation experiments are not that significant.

Questions

Please see the weakness section.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The scale of the experiment is not big enough. It would be great if the authors could perform larger-scale experiments, ideally Dust3r-level scales, to understand the scaling capability of the proposed method. I can understand it is not easy to acquire more resources for training, but it would be great if the authors could have more discussions on this.

Summary

The paper describes an MVS approach that does not depend on given depth ranges, which MVS algorithms typically require when building 3D cost volumes. The solution, previously proposed in DispMVS, is to perform searching and iterative updates in disparity space. Compared to DispMVS, the authors propose several novel designs to further eliminate the dependency on depth range during initialization, as well as to allow synchronous updates from all source views.

Strengths

**Significance**: The complete elimination of the dependency on depth ranges brings convenience and avoids performance degradation due to improper ranges. These benefits may enable wider adoption of MVS in real-world applications. **Originality**: Although the paper largely follows the foundation built in DispMVS, the several improvements are novel and nontrivial. The combination of 3D pose embedding and cross-attention as a solution for synchronous updates among all source views is quite interesting and proven effective. **Quality**: The results show consistent improvements over other depth-range-free options. The ablation studies and the analysis of impact from varying depth ranges help make the overall design more convincing.

Weaknesses

**Clarity** The paper has some rather noteworthy issues regarding writing. The method section needs to have the right level of detail. The approach is built on top of DispMVS and shares many concepts and details, and the authors' choices to keep and omit details seem somewhat arbitrary, resulting in a non-self-contained method section. For example, there are also some concepts used without introduction, e.g. "sampling range" on L.168, "t_c, t_f" in Eq.11. The GRU updating process isn't defined. The relationship between epipolar disparity flow and depth is a critical concept that the paper does not explain. The resulting poor readability is potentially a significant issue, but unfortunately, it is hard to address in a rebuttal. I'd like to see how other reviewers weigh its impact. **Performance** While beating other depth-range-free solutions, including DispMVS, on the two benchmarks, results still need to catch up on cost-volume approaches like GBi-Net, often by a large margin. It'll be quite valuable to understand what causes such discrepancy. Is it due to the limited receptive field, search gratuity, or else? Can improving on any of these help close the gap? **Generalization** There is no discussion of generalization capability in the paper. Conceptually, it seems a depth-range-free approach would be more generalizable due to being invariant to object/scene scales. The ability to train a single model that operates in various use cases (indoor, outdoor, etc.) is arguably the most desirable feature of such approaches.

Questions

I look forward to answers to the questions above regarding clarity, generalization and performance.

Rating

5

Confidence

4

Soundness

3

Presentation

1

Contribution

3

Limitations

The authors recognize speed as a limitation. The performance gap behind cost-volume counterparts is also worth mentioning.

Rebuttal

Dear Reviewers, please provide feedback about the authors' rebuttal; the deadline is approaching. Thnak you

The claim and comparison with CER-MVS is uncrediable

The authors claim that >CER-MVS does not directly sample within the depth range but requires the use of depth ground truth (depth gt) to scale the entire scene to a range with a mean of 600. When we introduce a certain amount of noise to the depth gt, thereby altering the scale, the performance of CER-MVS declines sharply. However, in CERMVS paper, it is claimed that >To pair neighbor views with reference views, we use the same method as MVSNet. In BlendedMVS, **which is used for training only**, the scenes have large variations in the range of depth values, we scale each reference view, along with its neighbor views, so that its ground-truth depth has a median value **600 mm**. When we evaluate on Tanks-and-Temples, due to lack of ground-truth and noisy background, **we scale each reference view, along with its neighbor views**, so that its minimum depth of a set of reliable feature points (computed by COLMAP as in MVSNet) is 400 mm. To stitch the predicted depth maps from multiple reference views, we simply scale back each depth map to its original scale. For evaluation and test, CERMVS do not rely on the ground truth. When I check the source code of CERMVS again, there is even no depth ground truth loaded in the test dataloader. Why dose the author claim CERMVS rely on depth gt during inference?

About the limited novelty

I appreciate for the authors' response. Now, I think I understand this work. It is heavily based on DispMVS, but there is a main improvement. 1. Replace the vanilla feature correlation (named SIMI) in DispMVS with Disparity Feature Encoding combining Multi-view Disparity Attention Besides, there are several less important improvement 1. The variance-based uncertainty, which replaces the sum weight in original DispMVS and is also used in disparity feature encoding. 2. The initialization method for each flow map It shows effectiveness of the proposed modules, although being with more compuration costs. I still have concerns about the model performance (especially when being compared with CERMVS, which is also a GRU based method), and therefore I decide to keep my rating

Comparison with CER-MVS

We respectfully argue that there are some misunderstandings in your interpretation of the scaling operations in the code. CER-MVS uses depth gt priors to calculate the scale of scene and the way varies across different datasets. CER-MVS primarily uses three datasets: BlendedMVS dataset, Tanks-and-Temples dataset, and DTU dataset. BlendedMVS dataloader (https://github.com/princeton-vl/CER-MVS/blob/main/datasets/blended.py) provides two scaling methods: the default method (self.scaling == "median") uses depth gt to scale the scene to a median of 600 mm on Line 72, while the alternative method scales the scene using the depth range prior provided by the dataset (labeled as 'depth range gt') to achieve a minimum of 400 mm on Line 75. Tanks-and-Temples dataloader (https://github.com/princeton-vl/CER-MVS/blob/main/datasets/tnt.py) uses depth range gt prior to scale the scene to achieve a minimum of 400 mm. In the code, these depth range gt priors are loaded and represented by the 'scale_info' variable on Line 74 and 75. For DTU dataloader (https://github.com/princeton-vl/CER-MVS/blob/main/datasets/dtu.py), the depth range gt is not directly loaded in dataloader because DTU dataset already has a depth median of 600 mm and a minimum depth of 400 mm. The scene scale meets the network's requirements. CER-MVS performs uniform sampling on inverse depth, fixing the depth sample range (https://github.com/princeton-vl/CER-MVS/blob/main/core/corr.py). In the code (https://github.com/princeton-vl/CER-MVS/blob/main/core/raft.py), the maximum disparity $d_{max}$ is set to 0.0025, and the disparity increments of stage1 and stage2 are set to $d_{max}/64$ and $d_{max}/320$ on Line 81. By scaling the dataset, CER-MVS can obtain more accurate depth increments and maintain updates within the inverse depth sampling interval. To validate the robustness of CER-MVS to depth gt priors, we have designed experiments by altering the median or minimum depth values of the scene. Specifically, we have introduced noise perturbations to change the median of DTU dataset during rebuttal. When there is no per-pixel depth gt, CER-MVS uses the depth range to scale the scene depth to a minimum value of 400 mm. It is worth noting that the depth range provided by the dataset is very accurate. For instance, the ground-truth data for Tanks-and-Temples dataset is captured using an industrial laser scanner. However, in practical applications, the depth range obtained through COLMAP are often inaccurate due to the sparsity of feature points and issues such as occlusion and suboptimal viewpoint selection. To verify the robustness of depth range priors, we use the depth range obtained from COLMAP to replace the depth range gt. | Depth Range | Method | Acc.(mm)↓ | Comp.(mm)↓ | Overall(mm)↓ | | :---------: | :-----: | :-------------: | :-------------: | :-------------: | | GT | CER-MVS | 0.359 | **0.305** | **0.332** | | GT | Ours | **0.338** | 0.331 | 0.335 | | COLMAP | CER-MVS | 0.816 | **0.326** | 0.571 | | COLMAP | Ours | **0.338** | 0.331 | **0.335** | From the table, it can be seen that CER-MVS exhibites a certain degree of decline due to the noise in the depth range caused by COLMAP. In contrast, our method, which is independent of depth range, maintained consistent performance regardless of changes in depth range. This further demonstrates the necessity of eliminating the depth priors.