We introduce ViCA-NeRF, the first view-consistency-aware method for 3D editing with text instructions. In addition to the implicit neural radiance field (NeRF) modeling, our key insight is to exploit two sources of regularization that explicitly propagate the editing information across different views, thus ensuring multi-view consistency. For geometric regularization, we leverage the depth information derived from NeRF to establish image correspondences between different views. For learned regularization, we align the latent codes in the 2D diffusion model between edited and unedited images, enabling us to edit key views and propagate the update throughout the entire scene. Incorporating these two strategies, our ViCA-NeRF operates in two stages. In the initial stage, we blend edits from different views to create a preliminary 3D edit. This is followed by a second stage of NeRF training, dedicated to further refining the scene's appearance. Experimental results demonstrate that ViCA-NeRF provides more flexible, efficient (3 times faster) editing with higher levels of consistency and details, compared with the state of the art. Our code is publicly available.
Paper
Similar papers
Peer review
Summary
This paper addresses the task of editing 3D scenes using text prompt. The method is based on performing 2D edits on different views using a diffusion model and then optimizing the 3D representation on the edited images. In particular, the authors propose two regularization methods to ensure the 3D consistency of the 2D edits: geometric regularization based on depth-aware warping of of the edited images to other views, and learned regularization by aligning the latent codes in the diffusion model. The proposed method is compared to two main baselines for editing of several 3D scenes.
Strengths
-- Addressing the 3D consistency of text-based 3D editing is valuable, and is currently missing from the existing methods. -- The proposed method has sufficient novelty containing different new components compared to the baseline. -- The proposed regularization based on depth-aware warping is reasonable. -- Not requiring the iterative editing and optimization used in the main baseline, Instruct-NeRF-to-NeRF (IN2N), enables a more efficient editing (3x faster based on what authors report). -- The experiments show more promising results than the baseline in terms of 3D consistency.
Weaknesses
-- I am mainly not convinced about the visual quality proposed methods, as the visual results seem to be quite blurry and lack details. Although IN2N also suffers from some blurry results, it seems the problem is intensified in the proposed method due to the averaging of the latent codes. I am not sure if the proposed averaging is a good solution, considering the blurry outcomes. -- The method section is not well-written and it is quite hard to follow. I find some details of the proposed method to be either missing or not well-explained. Also, the diagrams visualizing the method are too general and not helpful enough in understanding the components of the proposed method. I am afraid the current representation of the paper might not be ready for publication.
Questions
-- I did not fully understand how the key frames are selected, and how many key frames are used in the proposed method. I would appreciate if authors could further clarify that. -- Are the key frames edited independently and then propagated to non-key frame views, or are the key frames also edited sequentially using the previously edited ones? -- Is there any ordering when the edited key-frames warped to other views sequentially, or is the key frames are selected randomly and independently? -- Some of the limitations of the method are discussed in the limitation section. It would have been more helpful to also include some visualizations of the failure cases, especially the ones which do not happen in IN2N.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
2 fair
Contribution
2 fair
Limitations
The limitations and social impact have been addressed by the authors.
Summary
The paper proposes a method to edit a NeRF using natural language. The authors achieve this by first “generating” an edited version of the input dataset and then training a NeRF on the edited dataset. To generate the said dataset, the authors train a NeRF on the original dataset for depth, with which they then propagate their keyframe edits (made with Instruct-Pix2Pix) to other views. Because NeRF depth is noisy, the authors use a diffusion-based blending network to refine those images into a cleaner dataset for the next-stage NeRF, which simply learns the radiance field from the new, edited images.
Strengths
The paper tackles the difficult problem of editing 3D scenes in a multi-view-consistent way. The proposed method is general, being applicable to humans, objects, and even unbounded scenes. It also takes advantage of physics-based operations such as projections and combines them with learning to handle noise and uncertainty (e.g., in NeRF depth). The paper produces plausible results (only up to a low resolution though).
Weaknesses
The major weakness, I think, is the complexity of the pipeline that requires the user to train a NeRF for depth, run inference on keyframes with Instruct Pix2pix, project edited frames to other views, run a diffusion model to perform blending, and finally train another NeRF on the edited images. It does not sound like an easy-to-use pipeline that can be flexibly extended or built upon. A complicated pipeline like this has many points of failure, too, as compared with a simpler pipeline. The question then becomes which part of the pipeline we may cut to achieve similar results. The authors claim it’s crucial to perform diffusion-based blending to get around the artifacts due to direct mixup. Indeed, the results look quite blurry. Given (the 2nd-stage) NeRF also blurs the results, I think the authors should ablate the blending module, i.e., directly feeding the mixup output into the 2nd-stage NeRF. My hunch is NeRF’s blurring effect may already achieve what the additional diffusion model is doing, To be clear, I do think blurry results are a weakness of this paper, too. I think one crucial experiment on the number of edited keyframes is missing: how many keyframes does one need to edit to achieve good results? Why not just run Instruct Pix2pix on all views instead of doing mixup and then blending?
Questions
Because I don’t expect the authors to change their method, I’m primarily interested in hearing about points #2 and #3 from the Weaknesses section above.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Limitations
Yes
Post-Rebuttal Update
I read the authors' rebuttal, which does address some of my concerns. Most importantly, I agree with Reviewer EWGc (and the authors) that putting together a system like this, albeit complex and difficult to build upon, has its own merits. The rebuttal explained in more detail how the different components of this pipeline are essential. As such, I am willing to improve my rating to BA given the task difficulty, the topic timeliness ("NLP x NeRF"), and the authors' rebuttal.
Follow-up on reviewer’s response
We wholeheartedly thank the reviewer for the encouraging response to our rebuttal. We are glad that the reviewer appreciates the merits of our work. We thank the reviewer for improving the rating. We also notice that there is some back and forth in the reviewer’s rating from weak accept to borderline accept. If the reviewer has any remaining concerns, please let us know. We are more than happy to address them.
Summary
This paper introduced ViCA-NeRF, a method for 3D-consistent NeRF editing using diffusion models. The proposed method decouples NeRF editing with two steps. In first stage (i.e. dataset editing), key views are sampled and edited via Instruct-Pix2Pix model and propagated to other views using the NeRF depth maps, obtaining a mixup dataset. In second stage (NeRF training), the edited dataset can be directly used for training the NeRF model. The authors proposed a blending refinement model to enhance the quality that enables consistent NeRF editing results.
Strengths
The proposed method demonstrated 3D consistent NeRF editing results and often achieves better editing image quality than baseline Instruct-NeRF2NeRF and NeRF-Art. Concretely, here are some strengths of the proposed method: - Guided Editing: The method uses key views for consistent and detailed 3D scene manipulation; - Depth-guided Correspondence Matching: Allows projecting one edited view onto all others for consistency and detail; - Key Frame Sampling and Mixup: This reduces the computational load by representing the entire scene with a few key frames; - Blending Refinement Model: Reduces noise in depth information and resolves incongruities between 2D edits and their 3D context; - Warm-Up and Post-Refinement Strategy: Enhances editing quality and efficiency; - Semantically Meaningful Edits: The method can make significant edits based on textual prompts. In summary, the method is capable of performing 3D consistent NeRF editing with text-guided diffusion pipeline. The quality of the results demonstrate improvements over previous approaches, sufficient ablation and comparisons are shown.
Weaknesses
There are some limitations and concerns. - Firstly, I was wondering what happens if the NeRF depth or geometry is bad, such as on glossy or specular surfaces. In such case, the depth-guided correspondence matching would decrease its accuracy. Considering most scenes shown in the paper are diffuse, it'd be interesting to have one such example as an ablation. - Results are rather blurry, such as Flowers in Fig.1, Cyberpunk Style in Fig. 4 and Fireworks in Fig. VI. In contrast, the "original NeRF" on the left exhibits pretty sharp image details. What's the source of blurriness --- Is it due to key frame selection, depth quality, or diffusion model? Some analysis and (at least) discussions are necesssary and helpful. - How's Equation (2) proposed? And the hyperparameter phi = 0.3. - Can local edits be enabled? Currently the pipeline is, similar to Instruct-NeRF2NeRF, heavily based on Instruct-Pix2Pix. This is typically diffusion on the whole image (i.e. background is also changing and being modified). Would be nice to support locally editing of a person without touching the background in some cases.
Questions
Some related works on 3D consistent NeRF editing: - PaletteNeRF: Palette-based Appearance Editing of Neural Radiance Fields. CVPR 2023. - Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field. CVPR 2023. - ARF: Artistic Radiance Fields. ECCV 2022. - SNeRF: Stylized Neural Implicit Representations for 3D Scenes. SIGGRAPH 2022. - NeRFshop: Interactive Editing of Neural Radiance Fields. I3D 2023.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
Discussed in weakness section.
Summary
This paper presents a method for editing Neural Radiance Fields using text instructions. Starting from an initial trained NeRF, the method samples a set of key views which are then edited using Instruct-Pix2Pix. Given the depth maps rendered from the NeRF, the method computes correspondences between different views and propagates the edits to the remaining views. To minimize the artifacts from warping the paper proposes to use a blending procedure that involves 2 Instruct-Pix2Pix models.
Strengths
1. The paper identifies a known issue with performing per-frame edits of Neural Radiance Fields (inconsistencies between views) and proposes an intuitive method to address them. This involves selecting key frames and then use the NeRF geometry to propagate changes to the remaining views. The approach overall is technically sound. 2. Both the qualitative and quantitative results are good, and are on-par with Instruct-Nerf2Nerf. Given that the proposed method is about 3 times faster than Instruct-Nerf2Nerf then it is a valid, more lightweight alternative. 3. The paper is well-written and easy to follow. The figures are clear and most components of the method are adequately explained in the text.
Weaknesses
1. The paper fails to show that all components of the method are needed. For example by looking at Figure 6, I cannot seem to see any difference between the "blended" and "post-refined" images.
Questions
1. It is very hard to tell if the method is significantly better than Instruct-Nerf2Nerf. The quantitative evaluation in the Supplementary shows very similar numbers. The text-image direction score and the temporal consistency score are correlated with the perceptually better results but there is no direct relationship, so it's hard to quantify which method is better. At the same time, it's hard to tell from a handful of qualitative results which method is better. This is not a problem of this paper, it's generally an issue with editing papers where there is no "correct" solution. I do not have a good solution for this. 2. The keyframe selection criterion seems a little bit arbitrary. How does the value of $\phi$ affect the final result?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors have an extended discussion on the limitations of the method which I appreciate. There is also a discussion on the societal impact in the supplementary material.
I read the other reviews and the rebuttal. The rebuttal overall was positive, as the authors attempted to clarify the questions raised by the reviewers, and they provided additional quantitative and qualitative results. Currently the reviewers are split, with 2 giving WA and 3 BR. The main concerns and questions that the reviewers raised are: 1) The reliance of the method on possibly non-accurate depth estimates. 2) The not-so-clear keyframe selection strategy. 3) The overall complexity of the method 3) The evaluations. My overall take is that the method is not necessarily more complicated than other competing methods like Instruct-N2N. From the papers that I've read in the subfield of NeRF editing, it looks like there is a lot of engineering and tricks put into the methods to make them work, with limited "pure" research contributions. I believe however that such methods, if well put together still have merit and can be useful. I agree with Reviewer CQtY in their remark that probably it will be hard to build on top of this method because it's already fairly complex in terms of the total number of components stacked together. Regarding the user study, I would kindly ask the authors to provide more details about the protocol used to conduct it. In the supplementary material it says that the number of raters is 16, but the other details were missing. For example, how many examples were shown in total? How many renderings for each scene? Is it just the 10 scenes that were already used? I appreciate the fact that the authors did a user study, but if the number of examples is small, it's hard to draw statistically significant evidence. Obviously I want to mention here that the authors are not expected or required to perform a large scale study in the 1-week rebuttal timeframe. I would like to here the other reviewers' thoughts before committing to a final score.
Follow-up on reviewer’s response
We wholeheartedly thank the reviewer for the insightful and constructive comments. Below we address the reviewer’s follow-up questions. - more details about the user study Given that Instruct-NeRF2NeRF evaluates 6 scenes in their work, we conducted the user study on the same 6 scenes from Instruct-NeRF2NeRF for a fair comparison. The number of edits for each scene follows the size of their instruction pool. Specifically, we use 3 edits from the IN2N face, 2 edits from the bear, 2 edits from the NeRF-Art face, and 1 for the other 3 scenes (campsite, farm, and person), resulting in 10 samples. For each scene, the instructions are randomly sampled from the instructions shown in our work. With each instruction, we create a pair of edits with different random seeds, and each edit is rendered from 2 views. During the measurement, human raters are asked to determine which pair is more diverse and more in line with the description. The results are averaged over 16 human raters. We agree with the reviewer that our current user study might not represent a large-scale evaluation, primarily due to the time constraint during the rebuttal period and the lack of a large-scale benchmark with numerous scenes (which is a common issue in NeRF-based 3D editing research as explained in the previous response to Reviewer x1ev). On the other hand, we would like to kindly emphasize the **substantial disparities in user preferences** that emerged in the study: our method achieved scores of 0.773 compared with 0.227 by Instruct-NeRF2NeRF for text faithfulness, and our method achieved scores of 0.875 compared with 0.125 by Instruct-NeRF2NeRF for diversity. These results lead us to believe that our current user study already demonstrates a significant superiority of our method over Instruct-NeRF2NeRF. To provide a larger-scale evaluation, we are starting to undertake a more comprehensive user study, which features more scenes, samples, and raters, and we will include the results in the revision. - Complexity of the method We thank the reviewer for highlighting the intrinsic complexity of the task and the corresponding methodological complexity that is commonly found in the subfield of NeRF editing. Indeed, primarily due to the scarcity of large-scale annotated 3D datasets, 3D editing is challenging and needs to leverage different components like 3D modeling from NeRF and image editing from 2D diffusion models, address the associated gaps between 2D and 3D domains, and integrate them into a working pipeline. We also thank the reviewer for highlighting the merit and usefulness of our method. Here, we would like to further explain how subsequent works can build upon our method. (1) In the previous rebuttal (Response to Reviewer CQtY), we clarified that the structured, modular design of our pipeline facilitates its extensibility, so subsequent works can improve each component separately. (2) Our method brings insight into *explicitly* emphasizing 3D consistency during editing. This vital aspect has been overlooked in prior research and holds potential benefits for future studies. (3) By using the components from our method, **we offer an optional interactive editing interface for users** based on the selection of specific 2D editing. This enhancement significantly boosts the controllability and practicality of the editing process. We believe that such an interactive editing feature can be incorporated into upcoming research and will be advantageous for applications in this domain.
Summary
The paper proposes a method for 3D editing via text instruction that is view consistent. It leverages on the idea from Instruct-NeRF2NeRF, but additionally propagates the edits into multiple views in order to achieve multi-view consistency. To achieve such, a geometry regularization and a learned regularization were used, where the former uses depth for correspondence matching and the latter uses the typical 2D prior as used by Instruct-Pix2Pix but instead uses the average of the latent codes. Experiments show the effectiveness of the proposed method on 10 scenes.
Strengths
The paper proposes a way to enable 3D editing through text instruction that is multi-view consistent. The motivation of the proposed pipeline is sound and makes sense. It is also simple, which makes it easy to follow. Experiment demos show nice qualitative examples as well as qualitative ablations.
Weaknesses
The main concern I have with the paper is the thoroughness of the experiments. Specifically, in the main paper only qualitative results are included, and the scenes and examples presented are limited. The supplementary included a quantitative metric, but this is also only on 10 scenes. It also mentioned that the pipeline is heavily reliant on depth, which makes one wonder whether the limited scenes shown is because of the failure cases with depth. Including failure cases might be helpful.
Questions
A critical question I had is the details of the 10 training scenes, specifically, how many input views are used for training? The number of input views would give an idea on how critical the edit propagation to the different views is. Moreover, both the edit propagation and the geometric regularizer rely on depth inferred from the NeRF, and the correctness of NeRF depth is highly dependent on the number of training views. The failure case of the original Instruct-NeRF2NeRF is when the 2D diffusion samples is not consistent in 3D, which is also dependent on how many views are there. Perhaps an ablation on this would also be meaningful and make the contribution more convincing.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The authors included a limitation section in the supplementary material.
Additional clarification on certain blurry results
In addition to clarifying the reason behind certain blurry results as above, we would like to emphasize that **our model can work effectively in many cases where Instruct-NeRF2NeRF fails to converge**. For example, as shown in **Fig. 4** of the paper, our model is able to edit scenes under challenging instructions, as long as the 2D diffusion model remains functional. This is in contrast to previous methods like Instruct-NeRF2NeRF, which completely fail or produce insufficient edits in such situations. Our method also clearly demonstrates more details in some cases, e.g., as shown in **Fig. 5** of the paper. While our enhanced capability for broader editing and improved 3D consistency is notable, it does entail a trade-off by introducing noise/artifacts to the images, addressing which in our model causes blurriness in certain cases. However, based on the analysis presented earlier, we believe that using a diffusion model with improved controllability to 2D editing (e.g., only editing the desired objects/regions without changing the scene layout or other areas) would serve as a solution to mitigate the blurriness issue. We have validated this by showing an example of local editing in **Fig. R2** of the rebuttal PDF (using the Segment Anything Model to focus on the specific object to edit without changing the background). More generally, this blurriness issue can be mitigated by using an improved 2D diffusion model, which we leave as interesting future work.
Additional clarification on certain blurry results
In addition to clarifying the reason behind certain blurry results as above, we would like to emphasize that **our model can work effectively in many cases where Instruct-NeRF2NeRF fails to converge**. For example, as shown in **Fig. 4** of the paper, our model is able to edit scenes under challenging instructions, as long as the 2D diffusion model remains functional. This is in contrast to previous methods like Instruct-NeRF2NeRF, which completely fail or produce insufficient edits in such situations. Our method also clearly demonstrates more details in some cases, e.g., as shown in **Fig. 5** of the paper. While our enhanced capability for broader editing and improved 3D consistency is notable, it does entail a trade-off by introducing noise/artifacts to the images, addressing which in our model causes blurriness in certain cases. However, based on the analysis presented earlier, we believe that using a diffusion model with improved controllability to 2D editing (e.g., only editing the desired objects/regions without changing the scene layout or other areas) would serve as a solution to mitigate the blurriness issue. We have validated this by showing an example of local editing in **Fig. R2** of the rebuttal PDF (using the Segment Anything Model to focus on the specific object to edit without changing the background). More generally, this blurriness issue can be mitigated by using an improved 2D diffusion model, which we leave as interesting future work.
Post-Rebuttal Comments
I appreciate the author's response to my questions. Here is my overall opinion about the work after reading the rebuttal and other reviews: The authors argue that the blurriness caused by the 2D diffusion model, and not the averaging component. However, looking at Fig R1 in the rebuttal PDF, it seems to me that the blurriness has clearly intensified compared to the blended image without averaging (E.G. the details of the hair, the ears, ...). In terms of the complexity of the method discussed by other reviewers, I also don't think the complexity on its own is a problem here, considering the difficulty of the task. However, one would expect the quality of the results to match the complexity of the method. Although the proposed pipeline is reasonable to me, I am still not convinced by the quality of the results obtained by the proposed method. I acknowledge the better 3D consistency of the method and its ability in converging in scenarios challenging for Instruct-Nerf2Nerf. However, such advantages are obtained at the cost of very blurry and unrealistic results. Regarding my concern about the quality of the presentation, the authors' response clarified the ambiguous parts for me. However, I believe the method section needs a significant amount of refining to be also clear for the readers, and I cannot defend the current version. Overall, I am reluctant to change my score for the same reasons as before: the quality of the results and the presentation of the paper are not convincing and might not meet the bar for NeurIPS. However, I also acknowledge the difficulty of the task and that the proposed method is reasonable and of enough novelty.
Clarification on remaining concerns (Part 1/2)
We thank the reviewer for recognizing the rationality and enough novelty of our work. We are glad that our response has clarified the ambiguous parts for the reviewer. We would like to provide further clarification regarding the remaining concerns raised by the reviewer. >Q1. It seems to me that the blurriness has clearly intensified compared to the blended image without averaging We would like to further clarify and explain why we believe it is the 2D diffusion model that caused some blurry results. Conceptually, two potential factors could contribute to the blurriness observed in 3D NeRF editing results: 1) inconsistent 2D edits from different views, so that when the NeRF aggregates the 2D edits from these different views, the resulting 3D edits become blurred; or 2) the loss of high-frequency information during 2D editing, which means that the 2D edits are already smoothed. In our case, because we have specifically improved the multi-view consistency of 2D edits across different views, the blurriness in 3D edits primarily stems from (2) the smoothed 2D edits, rather than (1) the inconsistent 2D edits. This is also evident in the fact that inconsistent edits tend to produce irregular texture on surfaces, but our results show much better regularity compared with the inconsistent edits from Instruct-NeRF2NeRF, e.g., Iron Man’s clothes shown in Fig. 1 of the main paper. Now the question is casted as investigating when the smoothing occurs in images, and we provide the corresponding analysis experiment in the rebuttal PDF. The result in Fig. R1 shows that the smoothing happens when the mixed image is passed through the diffusion model. We can see that in both cases (with or without averaging), most high-frequency information is lost, e.g., texture on the face, beard, and hair. As a result, the 3D edits via NeRF will be blurred. Particularly, the smoothing happens since the IP2P diffusion model tries to reduce the noise in the mixed image introduced by projection. Despite being provided the clear original image as guidance, the IP2P diffusion model fails to create very clear edits. In comparison, Instruct-NeRF2NeRF does not have this issue in its single update, since its input image does not contain such noise from projection. However, we would like to point out again that Instruct-NeRF2NeRF also faces the blurriness issue from iterative updates on the same view. Therefore, both Instruct-NeRF2NeRF and our method face the blurriness issue, but for different reasons. As for the difference between using and not using averaging, we would like to first clarify that **the results in Fig. R1 are intermediate image edits from the diffusion model, which cannot reflect the clarity in 3D after NeRF training**. In practice, though using averaging introduces minor blurriness to the 2D image, it actually reduces the blurriness and improves details **to the NeRF training**. Particularly, the averaging greatly improves the consistency as it creates similar and more stable edits among different views. Please note that the example with Instruct-NeRF2NeRF in Fig. III of the supplementary uses a single pass from consistent images but results in very inconsistent edits (e.g., the clothes are changing rapidly for different views), and this is a similar scenario if we do not use averaging. We will include the comparison of the final rendered results from NeRF between using and not using averaging in the revision (which was not included in the rebuttal PDF due to limited space), where using averaging clearly shows better clarity. In summary, (1) both Instruct-NeRF2NeRF and our method face the blurriness issue, but for different reasons. (2) Some blurry results in our method are caused by the smoothing and loss of high-frequency information during 2D editing via the IP2P diffusion model.
Clarification on remaining concerns (Part 2/2)
> Q2. However, such advantages are obtained at the cost of very blurry and unrealistic results First, we respectfully disagree with the reviewer that our results are very blurry and unrealistic. Please note that **the blurriness happens for certain results, but not all**. A case in point is Fig. 5 in the main paper, where our results demonstrate better clarity, such as the wall and clothing for Vincent Van Gogh. Importantly, as shown in **Fig. R4** of the rebuttal PDF, the **user study involving 16 human raters** validates that our method produces significantly more realistic results than the state-of-the-art Instruct-NeRF2NeRF, with **substantially higher scores in user preferences**: our method achieved scores of 0.773 compared with 0.227 by Instruct-NeRF2NeRF for text faithfulness, and our method achieved scores of 0.875 compared with 0.125 by Instruct-NeRF2NeRF for diversity. Second, we would like to argue that for **3D editing tasks**, being able to achieve faithful 3D consistency should outweigh the occurrence of some blurriness. This is because ensuring 3D consistency represents a fundamental, imperative, yet a relatively unique challenge within 3D scenarios. However, it is under-explored in the field of NeRF-based 3D editing. Our work is the pioneering effort to explicitly address this issue and propose an effective strategy. In addition, as the reviewer recognized, our method is able to consistently converge and function even in challenging cases, in contrast to complete failure or insufficient edits as exhibited in Instruct-NeRF2NeRF. This property is desirable and very important in practical applications, where an editing system should always respond to the text instruction; an editing system that frequently fails to converge like Instruct-NeRF2NeRF makes it unusable in practice. Third, in our previous response, we have also suggested solutions to mitigating the blurriness issue. In particular, in **Fig. R2** of the rebuttal PDF, we have shown that by using SAM to generate a segmentation mask for the object of interest and then using our depth-guided propagation to create masks for all views, we successfully achieve local editing. Doing so reduces blurriness and greatly improves the clarity of background compared with previous methods. Note that this strategy cannot be accomplished by previous works, since it leverages depth-guided propagation which is our novel contribution. Finally, as mentioned in our previous response to Reviewer EWGc, by using the components from our method, **we offer an optional interactive editing interface for users** based on the selection of specific 2D editing. This enhancement significantly boosts the controllability and practicality of the editing process. We believe that such an interactive editing feature can be incorporated into upcoming research and will be advantageous for applications in this domain. Moreover, our method achieves significantly improved efficiency, with about 3 times faster than Instruct-NeRF2NeRF as highlighted by Reviewer EWGc. > Q3. I believe the method section needs a significant amount of refining to be also clear for the readers We greatly appreciate the reviewer’s suggestion on improving the quality of the presentation in the method section. We are committed to improving the presentation quality to the highest level in the revision. However, we respectfully disagree with the reviewer that the method section needs a significant amount of refining. First, we believe that most parts of our method section are clear and easy to read. As noted by Reviewer EWGc: “The paper is well-written and easy to follow. The figures are clear and most components of the method are adequately explained in the text.” In the meanwhile, we realize that some details of the proposed method are unclear to the reviewer. However, we believe that all these issues have been well explained in the rebuttal. As acknowledged by the reviewer, “the authors' response clarified the ambiguous parts for me.” Given that the detailed explanations have already been provided in the rebuttal and that these explanations have proven to be effective for clarifying confusion, we believe that integrating these explanations into the revision should be a straightforward task, requiring a relatively small amount of effort. Finally, we would like to express our sincere appreciation to the reviewer for the valuable comments, insightful suggestions, and considerable time and effort invested in evaluating our work. We hope that our clarifications provided in this response have addressed the reviewer's remaining concerns. Once again, thank you for your valuable feedback.
Thank you for your detailed responses and for the additional experiments. I appreciate the effort put into the rebuttal. I fully agree that there is a lack of datasets and evaluation scheme in general for the task at hand. I also acknowledge that it is a difficult problem.
Dear Reviewer, We wholeheartedly thank you for the encouraging response to our rebuttal. We are excited to know that all your concerns are well addressed. Considering your latest positive comment, we kindly request you to reconsider the rating of our paper, if possible. We would be more than happy to provide additional clarification if you have further questions. Thank you once again for your valuable feedback.
Decision
Accept (poster)