PAPR: Proximity Attention Point Rendering

Summary

This paper proposes a neural point renderer that improves the existing differentiable point rasterizers using a learned attention block, which improves the gradient behavior alleviating the "vanishing gradient" issue that comes with many existing differentiable point rasterizer. The method compares favorably with prior point-based NeRFs, jointly optimizing the point features and point locations, and achieving impressive 2D visual and 3D reconstruction quality even when using only poorly initialized sparse points. Additionally, it demonstrates useful applications thanks to the explicit point representation.

Strengths

- The proposed method uses a learned attention block to determine the blending weights. It engineers the query, value and key embeddings using suitable geometric entities, which seems oddly effective (see my questions below) to emulate the similar effect of a fully deterministic point rasterizer in terms of the blending behavior w.r.t. point relative distance and depth. - The learning is further assisted with progressive point growing and pruning, which, as demonstrated in the ablation experiments, are effective techniques to achieve high quality reconstruction. - The visualization of point features through clustering is very interesting and insightful. - The paper is well-written, carefully explained, and the proposed components are thoroughly evaluated. - The improvement over existing point-based NeRF papers are significant.

Weaknesses

1. No discussion about complexity. This is important, since the proposed method boasts to be able to compute gradient for all points regardless of their absolution distance to the target pixel in the pixel-space. This requires computing the attention with *every point* for every ray. Existing methods using RBF or other definition rasterization gradient (e.g. [44]) utilize handcrafted rules to reduce the impact of a pixel to a point as the point-pixel increases. While these rules may have the vanishing gradient problem the authors identified, they can serve as hard constraint to limit the number of target pixels required to evaluate the gradient for a specific point, i.e. pixels outside a search window can be ignored. With the proposed approach, and the claimed benefit, that each point receives gradient from all pixels, the computation is significantly higher. It's therefore important to breakdown the algorithm complexity of each step. 1 (cont). Besides the theoretical complexity analysis, the authors should also provide the training and inference speed compared to a standard NeRF that achieves similar visual quality. 2. the remark to [44] is not correct. [44] *is* able to render colored and shaded points. Besides, SynSin [Wiles et.al. CVPR 2020], Pulsar [Lassner, Christoph, and Michael Zollhofer. CVPR 2021] and Adop [35] are three relevant papers that should be cited too. In addition, I would argue that [44], Pulsar and [35] are more relevant than the baselines that are based on MVS initialization, and should be compared too. 3. The necessity to use a convolutional U-Net in the end is disappointing, as the convolution is likely to induce flickering and other inconsistencies when moving the camera, which could be the cause of the flickering on drum surface in the video?

Questions

I'd like the authors to address my concerns in the "weakness" section. Besides, please also help me understand the following questions in the rebuttal. 1. Since the relative distance is completely learned without any prior, why would the network learn to produce higher weights for closer points. Could you visualize the learned weights for some sampled rays? Please also clarify whether Fig 3c is computed from a real learned result or plotted as the "desired behavior"? In addition, why do the curves go up again when the points move further away from the pixel (the right-most part of the blue curve and the left-most part of the orange curve). 1. (cont) The inputs to the key and query embeddings don't share any similarity at all. The MLPs need to learn to map them to compatible space and ensure that keys (3) of points that are close to the rays (which would correspond to the Fig 3c gradient behavior) are more similar to the query (5). How does the network learn this without any prior? Do you need special initialization for the MLPs? And what is the convergence rate. 2. Ideally the learnable foreground score should be related depth and occlusion. But without constraint, this learned foreground score can be abused to label "outlier points" that are far from any target pixels due to optimization problems, which can be conveniently pruned and therefore ignored in the optimization. Please visualize the foreground score before point pruning to ensure that this score indeed correspond to visibility.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Summary

This paper presents a novel view synthesis and 3D reconstruction framework via a point-based representation and a jointly learned transformer-based differentiable renderer. Given multi-view images of a foreground-centric scene, the method can effectively deform a point-cloud from scratch to match the surfaces, with an optional pruning and growing mechanism. Each point is associated with a learned feature, which allows novel views of the scene to be rendered through a transformer-based point renderer. Experiments show that this method leads to high-quality geometry and novel view synthesis results, significantly outperforming existing point-based NVS methods as well as the vanilla NeRF.

Strengths

### S1 - A solid pipeline for point-based reconstruction and rendering - The proposed pipeline is very well designed and validated. In particular: - The design of the transformer-based point renderer is interesting. Ray direction is treated as the query feature, point position the key, and point features the value, which make a lot of sense. - The transformer directly outputs the weights of each point for integration, without a separate volume rendering function as in Point-NeRF that would introduce another order of computation. ### S2 - Great results on geometry reconstruction and novel view synthesis - The method effectively deforms a point cloud from a initial sphere to complex surfaces, which is a very compact representation of the scene geometry, and allows for various scene editing. - The NVS results are also quite impressive, even outperforming NeRF (although I guess one might be able to further tune NeRF to achieve even better numbers). ### S3 - Good demonstrations of scene editing with points - The paper presents a number of scene editing applications, including deformation, object removal and appearance editing. Having an explicit point-based representation makes such editing much easier than implicit NeRF/volumes. ### S4 - Good presentation - The paper is well written, with clear motivation and illustrative figures.

Weaknesses

### W1 - Renderer seems to be learned jointly on one scene only - In theory, the transformer-based renderer should be generic and can be shared across any scene? Does it generalize to other scenes? ### (minor) W2 - Technical details - There seems no explicit inductive bias in the renderer that would encourage it to actually put higher weights to points closer to the ray and to the camera (within the K nearest points). Does the learned renderer actually utilizes the information of $\mathbf{s}_{i,j}$ and $\mathbf{t}$ as we expect? It would be helpful to visualize the weights. - Sup. mat. Sec. B.3 Eq (10): Is the $w_{i,j}$ here taken before or after the softmax in Eq. (6)? - If before, $w_{i,j}$ is not guaranteed to be positive, and Eq. (7) and Eq. (10) would be problematic, as both $w_{i,j}$ and $\tau_i$ can be either positive or negative. - If after, there would be double softmax / exponential. Does that cause any issue, eg. vanishing gradients? - Pruning points with $\tau_i \lt 0$ seems quite arbitrary. I suppose this is related to the choice of the predefined background weight scalar $b$. Is the threshold $0$ chosen empirically? (why not, say, $\tau_i \lt 1$ or $\tau_i \lt -2.64$?) - What is the run time? How does it compare to say NeRF? - What happens if it applied to forward facing scenes with complex background, eg. LLFF? The limitation discusses the failure in modeling background in 360° scenes, but what about forward-facing scenes, where points can be legitimately assigned to "background"? ### Additional comments - Fig. 4: When the pruning is disabled, it is not surprising at all that Point-NeRF fails to represent the surface using the points, as it is still using a volume rendering function where the underlying geometry is ultimately still represented by the density values. Although, I agree it is great that the proposed method is able to deform the points correctly around the surface without pruning, thanks to the distance-aware weighting in the rendering function I suppose. - Only one attention layer? - Any points remain inside the object? - Line 109: minor grammatical issues.

Questions

No critical concerns. It would be helpful to elaborate on the generalization of the jointly learned transformer renderer.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The paper includes a nice discussion on the limitation regarding background modeling.

Summary

This paper proposes a novel method that consists of a point-based scene representation and a differentiable renderer. The proposed representation utilizes relative distance to characterize the contribution of each point, preventing the vanishing gradient issue. The method outperforms baselines in terms of optimizing point cloud positions, rendering. And it enables interesting applications.

Strengths

- Overview figure is clear, well-conveying the idea of the paper. - The representation is simple, reasonable and well-desinged. - The proposed method outperforms the baselines.

Weaknesses

The paper is sound in general. However there are several points that can be clarified. - Feature vector: learned or deterministically determined? If it is the latter case, how it can be determined? - In the figure 3-(c), why does the gradient scale increases again when the target pixel position deviates much from the point cloud? - In L148-151, it is mentioned that using the relative distance guarantess that there are always some points contributing to each pixel, even if their absolute distances are large. However, if the distance is too far, wouldn't this design enlarge the error, i.e., generating the color of the unrelated point cloud? I think setting a threshold would be helpful to prevent error, or is this thresholding same as the filtering process in L164?. Also, if the filtering is applied (L164), wouldn't the points that are far away be removed and therefore cannot contribute to the pixel? And how is this filtering threshold decided? Would the value of it affect the performance? - How can the view dependent effect be modeled? From my understanding, different viewing direction results in different s,t value of each point, and thus it is possible to model it? - What is the fixed background token b in L189? - In Figure 5, the result of proposed method on the Mic sample shows very fine-grained detail similar to GT. However, the result on the Materials sample is relatively blurry. What is the reason behind the quality difference? - It would be great if the visual ablations can be done when using the different number of points.

Questions

- How can the transparent material modeled with the proximity attention? - How this approach can be extended to complicated scene with lot of occlusions? - This approach can work on rigid deformation for some extent. How can this approach be extended to non-rigidly deforming scene?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- Clarifying some details (as mentioned in the weakness section) could further improve the paper.

Summary

This paper presents a novel point-based rendering approach, PAPR, to learn a point cloud for neural rendering under the supervision of posed RGB images. At the core of softening the gradient flow by using an attention mechanism in a novel and elegant way, the proposed PAPR enables high-fidelity rendering while explicitly learning the parsimonious point-based scene representation from scratch. Based on this, the proposed PAPR enjoys flexibility for geometry editing, object manipulation, texture transfer, and exposure control, all with impressive results.

Strengths

1. Learning a parsimonious point cloud from scratch in the form of neural rendering without using the 3D scene information is very interesting and has many potential applications. 2. The core module, proximity attention, is well-designed and should be computationally efficient. Instead of sampling points along rays, the proximity attention module proposed in this paper uses the projection relationship between rays and points to aggregate the proximate points in an attention form and feed the point features in a U-Net for rendering. The overall pipeline is neat and technically sound. The computation should be efficient as well.

Weaknesses

1. Because the authors emphasize the parsimony property of the point cloud, the number of points should be the key to be studied. Although the authors have provided an ablation study in Fig. 6 on the effect of the number of points on rendered image quality, some in-depth discussion would further strengthen this paper. Because the authors mentioned that the proposed method is currently suitable for the near-constant background color, I am curious about the parsimony of the point cloud concerning the non-constant background color. 2. The growing scheme is somewhat heuristic with predefined hyperparameters such as 500 iterations, top-10 nearest neighbors, etc. There should be some ablation study for the effect of these hyperparameters. Furthermore, because proximity attention could capture the relationship between points, is there a way to use the learned affinity to identify the sparse regions?

Questions

Because this paper has shown their ability to learn a parsimonious point cloud from scratch, what will be if we have some inductive initialization of the point cloud? For example, could the proposed method refine the sparse reconstruction quality if we have a sparse reconstruction result for the scene by COLMAP? I am also curious about the potential of doing bundle adjustment with the proposed PAPR.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have discussed their limitations. I don't have any concern on this point.

Summary

This paper tackles the problem of point-based rendering and proposes a new point-based differentiable rendering pipeline with proximity attention. With an attention-based point rendering, the proposed pipeline avoids the tuning of points' radius or splatting sizes that typical point rendering requires. Such a design enables efficient and high-quality rendering. The effectiveness is verified on NeRF synthetic and Tank and Temple datasets.

Strengths

1. The proposed proximity attention is neat and effective. 1. The paper is well-written and easy to follow. 2. The experiments are solid.

Weaknesses

### 1. Choices of query, key, value for attention a. I think the paper misses an important ablation of various representations for query, key and value for the proximity attention. Currently, query is represented as ray direction, key is used as $(t, s, \mathbf{p})$, and value is represented as $(t, s, \mathbf{u})$. A natural question is what if 1) we remove $s$ in both key and value; 2) we have key and value are both represented as $(t, s, \mathbf{p}, \mathbf{u})$ (adding feature vector)? b. What is related is Fig. 7. Can authors clarify: for "incrementally remove key components in the point embedding" (Fig. 7's caption), does it mean removing $t_{i,j}$ in both key and value representation? ### 2. Relationship to [24] a. I feel like the proximity attention is quite similar/related to the **Ray-centric Point Encoding** in [24]. Essentially, [24] also proposes an attention-based point embedding, where "query" is the ray direction same as this paper; "key" and "value" are concatenations from points's positions, distance between point and ray, angle between points and ray (also similar to the paper). b. Therefore, to better position the paper, I want to understand the difference between this paper and [24]. This is one of the major motivations why I am curious about the ablations about the choices of query/key/values. c. One difference I can see is that [24] uses an MLP as the renderer while PAPR uses a UNet. Can the author provide an ablation where an MLP is used for $f_{\theta_\mathcal{R}}$ (L181)? If PAPR does not work well with MLP, then it seems like the major improvement may come from the renderer's capabilities instead of the proposed attention-based point rendering. d. Another difference is that [24] uses an image-based feature as the point's feature. Such a design does not allow for changing points's positions if the points's features are not updated later. For Fig. 4, authors state "we made modifications to their official code to enable point position optimization" (L214-215). Can the authors clarify whether the features will be updated or not? If the features are not updated, then this needs clarifications and it may be better to replace [24] results in Fig. 4 with initializing from a sparse point cloud of SfM or MVS. This will not shade the advantage of PAPR. e. Similarly to d above, do the results of [24] in Tab. 1 come from initializing from a sphere? If so, I think it is better to replace it with initializing from a sparse point cloud and give a note in the table about this. ### 3. Speed a. An advantage of traditional point-based rendering [15, 53] is the speed. Can authors also provide the inference speed of PAPR? b. What is related is the training speed. Can authors also provide the training time of PAPR?

Questions

See above.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The authors provide discussion of limitations in the paper.

Thanks for the detailed responses

I appreciate the immense effort the authors put in the rebuttal to provide the additional results and elaborate technical explanations. In particular, the visualization of the weight distributions and the ablation studies on various token design are very helpful. I think this is a very solid implementation for a learning-based point renderer and my rating remains. I would very much look forward to a version of this renderer being trained at a larger scale that can generalize across arbitrary scenes.

Response to the author rebuttal

I would like to thank the authors for their time and efforts dedicated to rebuttal. All of my concerns have been addressed.

Rebuttal Reply

I appreciate authors's time and effort in addressing my questions. My major concern about the relationship with [24] gets resolved and I am happy with the additional information provided. I think this is solid work and I stay positive on the acceptance. Therefore, I raise my score.

Response to the author rebuttal

I wish to express my appreciation for the authors' dedicated efforts in addressing my queries. I am satisfied with the responses provided and am impressed by that PAPR achieves high-quality point-based rendering without initialization.

Discussion Ending Soon (less than 14 hours) : Would appreciate any feedback from Reviewer U5yB

Since there are less than 14 hours remaining in the discussion period (deadline is on Aug 21st at 1 pm EDT) and we haven’t received your response for our rebuttal, we thought we’d reach out to you directly. We hope our response addressed your concerns. Your input is valuable to us, and we would be happy to address any remaining concerns if you still have any. If all your concerns have been addressed, we would be grateful if you would kindly consider updating your rating. Thank you again for taking the time and effort to review our paper.