Large Spatial Model: End-to-end Unposed Images to Semantic 3D

Summary

The authors proposed the Large Scene Model (LSM), a novel 3D scene understanding framework that unifies multiple vision tasks within a single model. LSM represents a scene using pixel-aligned point maps, integrating geometric, appearance, and semantic information into a unified representation. By leveraging a Transformer architecture with cross-view and cross-modal attention, the model effectively incorporates multi-view cues and semantic knowledge from a 2D vision model. LSM's design enables efficient scene-level 3D semantic reconstruction and rendering in real time on a single GPU. The model's integration of a 2D semantic model allows for open-vocabulary understanding, extending its applicability to diverse real-world scenarios. Furthermore, by consolidating multiple tasks within a single model, LSM minimizes error propagation, leading to more robust and accurate results compared to state-of-the-art baselines.

Strengths

* The described technical approach in this work is sound and clearly presented. The contributions from the various proposed modules are well ablated and investigated in the experiments (Table 4). * The model demonstrates high inference efficiency compared to other approaches. with reconstruction time of 0.1s and rendering at 270 due to the underlying 3DGS representation that is being generated. * I like that the model reconstructs the underlying 3D representation in a single feedforward pass, as compared to multiview + test time optimization for fusion approaches. This improves the speed and efficiency for inference. It is good to see compelling quality based on the novel view synthesis.

Weaknesses

* I think the main contribution of this paper is the unification of the various scene modeling tasks into the same model, including geometry, color and semantics. The authors further claimed in the abstract and introduction that multitask training end-to-end allows LSM to outperform state-of-the-art baselines. However the paper did not ablate the multi-task learning design choice. For instance, what if some of the tasks are removed (e.g., semantic feature prediction). How does that affect the performance of the other tasks? * A suggestion is that for Figure 5, it is unclear how much pose divergence there is between the input source view and the synthesized novel view. It would be helpful to also show the source view supplied as input to the model. * The paper is named Large Scene Model, which seems to suggest something to do with model parameter scaling, hence large. However the paper does seem to do much scaling on model size. So perhaps a more accurate terminology would be Multitask or Unified Scene Model? Nits. * Line 153: Typo: to serve as input? * In Tables 1-4, I suggest highlighting the best (and possibly second-best result) for easier comparison of the various experiments. * In Table 4, why is + Multi-scale Fusion indented?

Questions

- For the quantitative comparisons given in Figure 3, are they predicted from the input view, or are they for a novel view?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

- My understanding is that in this method, all the Gaussians being generated are pixel aligned with the original input images. Is that a limitation of the method, since that would make the model unable to model large pose divergences that require rendering regions not originally visible in the input view, for instance, the back side of a sofa etc.

Summary

This paper presents the Large Scene Model (LSM), which generates semantic radiance fields from uncalibrated RGB images using a unified Transformer-based framework. LSM can infer geometry, appearance, and semantics simultaneously and synthesize label maps in real-time. The model integrates multi-scale fusion and features from 2D models to enhance accuracy and efficiency.

Strengths

1. Unified Framework: LSM combines multiple 3D vision tasks into a single framework, streamlining the process and reducing complexity. 2. Real-time Performance: The model achieves real-time 3D reconstruction and rendering, suitable for applications needing fast processing. 3. Enhanced Feature Fusion: By incorporating 2D model features, LSM improves the quality of feature lifting and semantic understanding, enhancing overall performance.

Weaknesses

1. Dataset: I recommend the authors organize the training and testing phases in alignment with previous methods (NeRF-DFF and Feature-3DGS) and provide results on the Replica Dataset. The authors have not sufficiently justified deviating from the baseline evaluation split. Furthermore, an explanation is needed for the significant performance discrepancy of the baselines between the Replica Dataset and the authors' setup. Additional training details may also be necessary. 2. Writing: The paper's abstract, introduction, and methods sections require improvement. Specifically, the methods section should introduce each module and their interconnections from a high-level perspective rather than presenting them as isolated components. 3. Method Details: Do the authors use camera parameters? If so, why are camera parameters mentioned in line 117? If camera parameters are used, the model cannot be described as "unposed." 4. Visualization: In Figure 4, there are category colors that are not listed in the legend. Additionally, a more diverse set of results should be displayed, as the current experimental set predominantly features sofas.

Questions

1. Module Timing: I am curious about how the authors manage to use eight cross-attention modules and still achieve reconstruction in 0.1 seconds. Please provide the time consumption for each step. 2. Image Resolution: What is the resolution of the images? More details regarding the inference process should be provided, especially concerning the time comparison.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors have discussed limitations.

Summary

The paper aims to train a network that takes in a set of unposed images and directly produces a semantic radiance field. The method utilizes a single Transformer-based model that learns the attributes of a 3D scene represented by a point-based radiance field. A decoder produced 3D Gausians that can be splatted to make novel images, depth estimates, and semantic segmentations.

Strengths

The paper provides a transformer architecture for producing 3D gaussians with rich features from unposed images, which seem very valuable. The design choices in the proposed system are well-chosen from methods available at this time, leading to a system that has a good combination of little-compute and competitive-accuracy on three different tasks (nvs, depth, semantics).

Weaknesses

The paper shares goals and ideas with "Scene Representation Transformers" (Sajjadi et al., CVPR 2022) and its follow up work Object Scene Representation Transformer (NeurIPS 2022) and RUST: Really Unposed SRT (CVPR 2023). This paper is different, because it ultimate produces a set of gaussians rather than a LLFF or NeRF volume, and it distills features from 2D foundation models. However, it is similar in that a transform encoder and decoder produces a scene representation directly from a set of images, which is then used for novel view synthesis, depth estimation, and semantic segmentation. In any case, those paper seem fairly similar in concept and so I think they should be discussed in the related work, and possibly approach sections. The ablation study in table 4 suggests that the key methods in the paper have little impact on the results of NVS.

Questions

It is interesting that the 3D methods, which have access to multiple views of the same scene do not perform as well as LSeg in Table 1. This is counter-intuitive. Can you please explain why? The results on multiview depth accuracy are kinda amazing. Why is the proposed method better than ones that take the camera parameters? Is it due aligning the scene scale (do all the methods get the same method for scale alignment? The novel view synthesis images look very good. Can you please provide some info about how close the novel cameras are to the reference ones provided at inference time? Is there a way for you to quantify and compare to PixelSplat the NVS results as the novel cameras deviate further and further from the reference ones?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are discussed briefly in the second paragraph of the conclusion.

Summary

This paper solves the sparse-view scene reconstruction problem by Large Scene Model, a unified scene reconstruction model via unposed RGB images. The model utilizes a ViT backbone for extracting the feature and uses cross-view attention to align the multi-pose feature for consistent features. The 3D scene is further rendered from the 3D semantic field derived by the multi-view features. The unified model is capable of multiple 3D-based tasks including novel view synthesis and 3D language-based segmentation. Experiments showed that the work achieves better results with limited performance sacrifices in the NVS task and higher performance in the multi-view language-based segmentation task.

Strengths

1. The model is general and multi-purpose in sparse-view scene reconstruction. 2. The model can achieve better results while still obtaining lighting-fast rendering speed and can be applied to real-time reconstruction.

Weaknesses

1. The technical contribution is limited. The model is generally designed via multi-purpose modules glued attention and Transformers, which is a straightforward and widely applied idea. There is no significant new problem has arisen and novel solutions proposed. 2. The performance comparison with NVS-related works is limited. Firstly, the authors train and run comparison experiments on the same dataset, which can be biased. Secondly, several popular scene datasets incorporated in similar works (such as RealEstate10k) are not utilized in this work. Thirdly, methods similar to pixelSplat such as Splatter Image[1] are not included in comparison. 3. The presentation can still be improved. Firstly, the authors titled their work “Large Scene Model”, while the design is more similar to the idea of pixel-based Gaussian splatting (such as pixelSplat and GaussianImage). Secondly, each module's input and output data type cannot be directly recognized from the pipeline graph. 4. The bibliography of this paper lacks some related works, such as Splatter Image[1], which is also an image-space Gaussian splatting method. Reference: [1] Szymanowicz, Stanislaw, Chrisitian Rupprecht, and Andrea Vedaldi. "Splatter image: Ultra-fast single-view 3d reconstruction." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10208-10217. 2024.

Questions

1. Did the authors try replacing the designed module with large-scale pretrained models, such as using pretrained monocular depth estimation model? 2. The design philosophy is similar to multi-view image generation works. Can this model output high-quality and consistent multi-view images, as the fashion of Free3D? 3. The term of “language-driven segmentation” is not quite clear to me. Does it mean semantic segmentation?

Rating

5

Confidence

3

Soundness

1

Presentation

2

Contribution

3

Limitations

The authors claimed that the major drawback of the model is the VRAM consuming. The social impact of this work mainly related to potential misuse of 3D assets and can be solved via integrating watermarks into the generated result.

Response to rebuttal

Thanks for responding to my questions in the rebuttal. The answers are pretty-much as expected. I will leave my review score unchanged.

Respectfully Requesting Comments from the Reviewer

Dear Reviewer K6pm, Thank you once again for your review. As the deadline for the author-reviewer discussion approaches, we noticed that we haven't received any further comments from you. We have addressed all your questions with additional experiments and clarifications: - We have demonstrated the importance and novelty of the problem: "Semantic 3D reconstruction directly from unposed images using a **single differentiable model**." - We provided comparisons showing that our method is not only pose-free but also enables **more 3D tasks** than other methods. - We plan to adjust the title to better emphasize the unification of 3D tasks that our approach offers. - We clarified the necessity of a model that maps global 3D points for input images. - We provided further clarification on how our method compares with Free3D. - We explained the terminology you requested. As the discussion period is coming to an end, we would greatly appreciate any additional feedback you might have. If our responses have clarified your understanding of our paper, we sincerely hope you might consider raising the rating. Thank you again for your effort in reviewing our paper. Best regards, Authors of Paper 3523