Real-time novel-view image synthesis on mobile devices is prohibitive due to the limited computational power and storage. Using volumetric rendering methods, such as NeRF and its derivatives, on mobile devices is not suitable due to the high computational cost of volumetric rendering. On the other hand, recent advances in neural light field representations have shown promising real-time view synthesis results on mobile devices. Neural light field methods learn a direct mapping from a ray representation to the pixel color. The current choice of ray representation is either stratified ray sampling or Plucker coordinates, overlooking the classic light slab (two-plane) representation, the preferred representation to interpolate between light field views. In this work, we find that using the light slab representation is an efficient representation for learning a neural light field. More importantly, it is a lower-dimensional ray representation enabling us to learn the 4D ray space using feature grids which are significantly faster to train and render. Although mostly designed for frontal views, we show that the light-slab representation can be further extended to non-frontal scenes using a divide-and-conquer strategy. Our method offers superior rendering quality compared to previous light field methods and achieves a significantly improved trade-off between rendering quality and speed.
Paper
Similar papers
Peer review
Summary
This paper describes a novel representation for learning view synthesis from a set of input images with known camera poses. They parameterize a classical two-slab 4D light field using a K-Planes representation (using 6 feature planes). Feature queries are processed through many layers of 1x1 convolutions before being post-processed by a super-resolution network to produce the output image. Similar to previous neural light field methods they augment the training data using virtual views rendered using a trained NeRF model. In experiments on standard benchmark datasets they achieve good rendering quality compared to previous methods capable of rendering on mobile devices, and also achieve a fast training time and compact representation.
Strengths
The proposed approach of parameterizing a neural lightfield using the K-Planes concept is reasonable and novel to the best of my knowledge. They show convincingly that this approach leads to faster convergence and a more compact representation compared to MobileR2L. They also achieve high-quality rendering with fast rendering speeds even on mobile devices. In some cases they actually outperform the teacher model. They present an extensive evaluation using several benchmark datasets, including both synthetic and real data. They also include an ablation study to consider the effect of virtual views and decoder network size. They also compare implementations on different mobile processors in terms of latency. The presentation is clear and easily understandable.
Weaknesses
The results on unbounded scenes especially are lower quality than state-of-the-art (non-real-time) methods such as Mip-NeRF 360. Artifacts such as blurriness and inconsistent shape are clearly evident in the result videos for unbounded scenes (but not for the object-centric scenes). These limitations deserve discussion. The approach of rendering a low-resolution image and then upsampling would seem to be a limiting factor in terms of rendering quality. It would be informative to see what quality is possible with this representation when directly rendering the full-resolution image (without the super-resolution network).
Questions
How do the results compare to state-of-the-art (regardless of rendering speed)? What is the effect of directly rendering a full-resolution image (rather than using the super-resolution network)?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
4 excellent
Contribution
3 good
Limitations
They do describe limitations but I think more discussion of the 360 unbounded results would strengthen the paper.
Summary
This paper presents the LightSpeed for real-time rendering on mobile devices. The approach involves replacing the commonly used Plücker coordinates with a light slab representation and implementing multi-level grids similar to Instant-NGP and k-planes. Additionally, It introduced a divide-and-conquer strategy to address the issue of light slab's inability to represent 360-degree objects effectively.
Strengths
The proposed approach effectively improves the efficiency with several interesting design. Firstly, it uses light-slab to parameterize ray, which shows efficiency compared to the commonly used Plücker coordinate. Besides, the practice of compressing 4D space through six sub-planes can also improve efficiency. Finally, the divide-and-conquer strategy to solve the problem that traditional light-slab method fail to represent 360 degree objects.
Weaknesses
1. In the abstract and introduction sections, the authors repeatedly claim that existing methods overlook the light-slab parameterization. However, research has already been conducted on novel view synthesis based on light-slab parameterization. Besides Attal et al. mentioned in line 69, there are some other literatures that are related but ignored, such as NeuLF: Efficient Novel View Synthesis with Neural 4D Light Field NeLF: Practical Novel View Synthesis with Neural Light Field Progressively-connected Light Field Network for Efficient View Synthesist All these papers all based on light-slab parametrization and its variants, and thus should be carefully discussed or compared with. 2. Some claims are hard to understand. For example, in lines 51-52, current ray parametrization will encounter issues with the introduction of grid-based representations. However, no specific problem is identified, and the subsequent description only explains that the existing method is "redundant." Also, in line 57, “the high-dimensional stratified-point representation is not feasible for grid-based discretization.”, it is unclear why it was not feasible. In my understanding, nerf is also a neural representation that needs sampling, and it can use the grid-based representation. 3. In the ablation section, the authors only ablate the data requirements and the decoder network size, which is not sufficient. More in-depth discussions are needed to demonstrate the effectiveness of each part of the proposed method. For example, it would be great if the authors could make comparison between the grid-based approach and the pure MLP approach within their own framework, such as replacing the Ray-space Grid Encoder with a traditional frequency encoder or so on. Please carefully check the claimed contributions and ensure that all of them are supported by the experiments.
Questions
Please address my questions in the weakness section.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
3 good
Limitations
Yes
Dear Reviewer 8eAH, We sincerely thank you again for your thoughtful suggestions and valuable feedback to improve our work. We provide additional explanations to help clarify our work. As the deadline for open discussion is soon, we sincerely hope to use this opportunity to see if our responses are sufficient and if any concern remains. It would be our great pleasure if you would consider updating your review or score. Thanks again for your time. Best, Authors
Thanks for providing the response! All my concerns are addressed. And thus I would like to raise my rating to "weak accept.
Dear Reviewer 8eAH, Thank you so much for checking our responses and raising the score. It is our great pleasure to know our efforts have helped address your concerns! We appreciate your time and reviewing efforts to help improve our work. If you still have questions or concerns, we would sincerely like to know and will make the best of our efforts to resolve them within the open discussion period. Best, Authors
Summary
This paper introduces LightSpeed, which uses traditional 4D light-slab representation and merges the super-resolution network proposed by MobileR2L. LightSpeed uses the NeLF method and will be primarily implemented on mobile.
Strengths
Originality: Utilize the overlooked method of 4D light-slab representation and merge the good method of saving memory and time from other NeLF methods (The super-resolution network proposed by MobileR2L). Extend the application scenes of light-slab representation to non-frontal scenes using the divide-and-conquer strategy. Quality: Greatly saves the storage compared with other methods and balances the reconstruction quality and storage well. Clarity: Clearly show the results and the advantages of LightSpeed. Significance: Advance the general application of the NeLF method on mobile.
Weaknesses
Since LightSpeed was proposed to solve the reconstruction problem in mobile, this paper does not show enough data for different mobile platforms. This paper will be more convincing with more data in different chips or multi-platform. This paper only shows the examples of unbounded datasets in Fig.4, and more comparisons of other datasets should be shown to be more convincing. The data demonstrated now keeps me in doubt about the effects. Table 3 should be placed in section 4.2 rather than 4.3.
Questions
I think the storage problem is not the most significant in the rendering problem in mobile. The mobile device can upload data to the cloud to solve this problem. RealityScan adopts this method and can obtain similar results based on NeRF method. So what is the most significant strength of LightSpeed for a user who can use cloud storage? Moreover, since MobileNeRF can achieve real-time manipulation, can LightSpeed achieve similar effects? Besides, demonstrating more data will make this paper more convincing, like experimenting on more different chips.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Limitations
please see weaknesses.
Summary
Real-time novel-view image synthesis on mobile devices is challenging due to limited computational power and storage. Volumetric rendering methods are unsuitable due to their high computational cost. The authors propose using the efficient light slab representation for learning a neural light field, which achieves better rendering quality and a favorable trade-off between quality and speed compared to prior light field methods.
Strengths
The paper proposes to use the light slab representation for learning a neural light field, which has not been used significantly in the literature before. The proposed method using light slab presentation is shown to perform better than the SOTA while providing a computational advantage over the existing methods.
Weaknesses
NA
Questions
N/A
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Yes
Summary
The paper introduces LightSpeed, a method aimed at simplifying real-time novel-view image synthesis on mobile devices, which typically face constraints related to computational power and storage. By adopting the traditionally underutilized 4D light-slab (two-plane) representation for learning a neural light field, LightSpeed offers a more compact and efficient ray representation. While the light-slab representation has its limitations, mainly being designed for frontal views, the paper demonstrates a way to extend it for non-frontal scenes using a divide-and-conquer strategy.
Strengths
- The paper proposes a promising approach that integrates neural light field representation with grid representation.
Weaknesses
- The Contribution Context: While the methodology primarily integrates existing techniques like the neural light field method and grid representation from k-plane[9] and tensorf[5], it lacks a direct comparison or detailed discussion on the light-slab representation versus the Plücker coordinate representation[26] for non-frontal scenes. Given that the Plücker coordinate representation might achieve similar results, an experimental comparison would provide a more definitive understanding of the actual performance improvements, if any, achieved by their proposed method. Although such an integrated approach has its merits, the overall performance improvement appears to be marginal without these comparative analyses. - Coverage of Related Work: The paper mentions another neural light field method using light-slab representation. However, other related and potentially influential works such as "Neulf: Efficient novel view synthesis with neural 4D light field, EGSR 2022" and "Signet: Efficient neural representation for light fields, ICCV 2021" have been overlooked. - Limited Results: The experimental results presented have some limitations. Notably, the occurrence of the 'jelly effect', particularly in unbounded scenes, implies that the proposed method could benefit from further optimization. Furthermore, the synthetic scene experiments were conducted at a resolution of 400 x 400, as highlighted in Figure 1 and the ablation study. It remains uncertain how the proposed method would perform at the dataset's original resolution of 800 x 800. This higher-resolution evaluation could provide a more thorough understanding of the method's capabilities. - Feasibility for Real-Time Rendering: The paper claims real-time rendering feasibility on mobile devices, but it does not provide sufficient evidence such as measured MACs or a real-time rendering video demonstration. The computational cost of the 30-layered decoder plus 6-plane feature query might prove too expensive for the intended mobile applications. Further investigations should be conducted to substantiate these claims.
Questions
-In line 175, it is unclear how the authors decided on the locations for the two planes, P1 and P2. Could the authors provide clarification on their choice of plane locations? -It is also not mentioned whether positional encoding was utilized before feeding data into the network. Could the authors specify if this step was incorporated in their method?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
The paper presents a noteworthy approach by merging neural light field representation with grid representation. However, it primarily rehashes existing methods, with limited novel contribution. Experimental results exhibit limitations, and concerns about the feasibility of real-time rendering on mobile devices persist. Therefore, considering these constraints, I suggest a borderline reject for this paper in its current state.
Dear Reviewer bQpx, We sincerely thank you again for your thoughtful suggestions and valuable feedback to improve our work. We provide additional explanations to help clarify our work. As the deadline for open discussion is soon, we sincerely hope to use this opportunity to see if our responses are sufficient and if any concern remains. It would be our great pleasure if you would consider updating your review or score. Thanks again for your time. Best, Authors
Thank you for taking the time to check our response. We try our best to clarify the reservations you may have about the paper. *** **Q1. Light Slab Representation** We provide a detailed discussion of the related works pointed out by reviewer and Reviewer `8eAH` in Q1 of the common response to all the reviews (`Author Rebuttal by Authors`). We will ensure that a discussion is added to the revised paper along with proper references. We humbly think we have addressed the concerns for Reviewer `8eAH` in this regard. *** **Q2. Regarding the Plucker representation** We appreciate the reviewer's questions about the efficiency of a grid-based discretized Plucker method over our light-slab based method. We experimentally try to discretize the 5D Plucker space and model the ray-space using $5 \choose 2$ 2D feature grids using our framework. All our efforts towards this approach fail. The main issue we encounter is that removing the projective ambiguity requires that we fix one of the coordinates to a constant value. However, we encounter *degenerate cases* where the coordinate to be normalized becomes 0. The best performance with this approach is compared with our LightSpeed model in Tab. A. These experiments are conducted for 200k iterations on a Lego scene. The grid-based Plucker representation isn’t able to learn anything compared to our method. > Table A: **Grid-based Plucker Representation**: Plucker-grid representation fails to learn the scene compared to our method. >| Method | PSNR $\uparrow$ | >| :------------ | :-----------: | >| Plucker-Grid | 13.36 | >| Ours | 31.20 | We hypothesize that this poor performance stems from the projective nature of the Plucker representation that *hinders* the effective discretization of the corresponding ray space and hence grid-based learning. To our knowledge, we are unaware of any works that offer an efficient way to discretize a projective space. On the contrary, the light-slab representation is compact and offers an easy discretization of the *Eucledian* ray space enabling grid learning. *** **Q3. Concerning the real-time mobile rendering claim** (a) We would like to draw attention to the fact that all our latency numbers are actually computed on mobile devices themselves (Tab. 3 main paper) leaving *no room for infeasibility on mobile devices*. (b) We agree with the reviewer that our `60-L W-256 LS` achieves 40 FPS on an iPhone 13, and this might be challenging to run on devices with lesser computational capabilities. We would kindly like to point out that this is a competitive FPS as compared to prior works. Further, this is exactly where our method comes into play with our `30-L W-256 LS` variant in rebuttal (which *also surpasses the MobileR2L visual fidelity*), delivering *~50 FPS* with *almost half FLOPs* than both `60-L W-256 LS` and `60-L W-256 MR2L`. Additionally, an even lighter variant, `15-L W-256 LS` in rebuttal with ~3.3x fewer FLOPs than full-sized models delivers a similar visual fidelity as that of the full-sized `60-L W-256 MR2L`. (c\) To numerically support the claim of our method's real-time performance on Android phones, we show the *on-device latency* on the **Snapdragon SM8450 chip** used in various **Android devices**, including **Huawei Honor Magic 4, OnePlus 10 Pro, Oppo Find X5 Pro, vivo iQOO 9, and Xiaomi 12**. We obtain competitive latency numbers (Tab. B) for our full-sized LightSpeed network and *much lower latency for our 30-layered network* which has better rendering fidelity than full-sized MobileR2L as shown in Tab. C of the original rebuttal. > Table B: **Rendering Latency Analysis on various scenes**: LightSpeed maintains a competitive rendering latency (ms) to MobileR2L on the Snapdragon SM8450 chip. >| Scenes | MobileR2L| Ours | Ours (30-L) | >| :------------ | :-----------: | :-----------: | :-----------: | >| LLFF | 39.14 | 45.65 | 32.89| >| Synthetic $360^\circ$ | 40.86 | 41.26 | 33.87| (d) We are building a mobile application following MobileR2L for real-time demonstration that requires substantial software engineering efforts and time to build (please kindly notice that the code for the application from MobileR2L has not been released). We would be happy to share a real-time demonstration video; however, the application is not ready right now and given the *limited time remaining* for the discussion period, it is not feasible to build the on-device application on such a short notice. We are certain that the latency and FPS numbers we provide reflect the on-device performance of the method (we could provide CoreML benchmark reports for different chips in the anonymous link if the reviewer prefers). That being said, we would like to assure the reviewer that we will release a full-fledged demo once the mobile application is ready. *** We sincerely hope that our response answers your questions. It would be our great pleasure if you would consider updating your review or score. Best, Authors
I want to express my gratitude to the author for their thorough response to the concerns I highlighted. I am confident that the forthcoming revisions will address the areas of related work and representation selection. Additionally, I acknowledge the integration of grid representation and light slab representation as a meaningful contribution. **My chief reservations lie in the paper's somewhat audacious claims, particularly the assertions of "Light Speed" and operability "on mobile device.** The detailed performance metrics on both Qualcomm and Apple chips are indeed valuable, but I firmly believe that a video demonstration is indispensable to validate such bold statements. My assessment would be more favorable with further empirical evidence, perhaps at this juncture a benchmark on CoreML as proposed by the author. I'm aware of the time constraints and would like to clarify that my 'borderline' rating is not an outright call for rejection. Instead, it's a nudge for the author to ensure these highlighted areas receive the necessary emphasis in the revised manuscript.
We thank the reviewer for their effort and the opportunity to further provide supporting evidence for our work. We share benchmarking images of LightSpeed CoreML packages done on Apple A15 and A16 chips as empirical proof of our method's real-time capabilities via an anonymous link shared to AC only (required by the NeurIPS PCs). We kindly ask the reviewer to get the link from AC. Best, Authors
Thanks to the authors for their responses. I have read over the other reviews and the authors' responses. I think they have sufficiently addressed the concerns raised in the reviews and I will maintain my recommendation of acceptance.
Thank you so much for checking our response and other reviews as well. We appreciate your time and reviewing efforts. If you have any other questions or concerns, let us know and we will make the best of our efforts to resolve them within the open discussion period. Best, Authors
Thank you to the authors for the detailed rebuttal and additional experiments provided. The response has indeed addressed some of my initial concerns, though I still have noteworthy reservations about the paper. - **Light Slab Representation**: As also highlighted by Reviewer 8eAH, this parametrization has been explored in several pieces of literature. Please ensure you verify if the references are from arXiv preprints or are already published. A revision of the claims and a detailed discussion comparing to these methods is necessary. - **Regarding the Plucker representation**: The authors have responded by suggesting that a 5d space occupies more space than a 4d one. However, in the context of a 360-degree scene, the 4d representation would necessitate multiple light slab representations, which could also be space-consuming. While I appreciate the inclusion of a table that quantitatively compares the light slab representation, demonstrating its superiority, I feel that the advantage might be marginal. - **Concerning the real-time mobile rendering claim**: The title **"lightSpeed"** suggests an extremely efficient and low power consumption solution. However, according to the rebuttal, the configuration "60-L W-256 LS", which surpasses the quality of mobileR2L, achieves just under 40 fps on the iPhone 13 with an A15 chip. This might be even more challenging for Android devices with lesser computational capabilities. Moreover, the authors have mentioned providing a real-time demo only post the code release. Given the bold "lightSpeed" claim and considering NeurIPS's stature, I believe a video demo should be available for review prior to the review process conclusion. Could the authors provide a live demonstration video through an anonymous link? In conclusion, I believe the paper presents an interesting contribution. However, I'd urge the authors to revisit some of their claims and experimental evidence. My reservations, especially concerning the real-time rendering on mobile devices, remain. I am open to reconsidering my evaluation if further convincing evidence is presented.
Decision
Accept (poster)