Geometry Cloak: Preventing TGS-based 3D Reconstruction from Copyrighted Images

Summary

This paper proposed a new image cloaking approach, which adds adversarial noise on single-view image and makes TGS-based 3D reconstruction fail. This can be served as a watermark for protecting copyright image assets.

Strengths

The topic is popular and needs more investigation by the community. The paper writing is clear.

Weaknesses

1. The main weakness lies in the scope of this work is too limited. The motivation is to protect copyright images from unauthorized 3D reconstruction, however the work only targets at TGS-based reconstruction, which is too limited to have practical effects. I suggest the authors should at least do experiments on other single image 3D reconstruction works, for example LRM[1], Gamba[2] or LGM[3]. This work shows fair results on TGS, but TGS itself is not representative among 3D reconstruction works, and does not yield the best single image 3D reconstruction results. In practice, we cannot assume unauthorized users will use which image-to-3D model, so it only makes sense when the watermarking / cloaking technique can effectively generalize to shield all those image-to-3D reconstruction algorithms. 2. Some typos. Line 138, "preventing copyrighted"? Reference [1] LRM: Large Reconstruction Model for Single Image to 3D. ICLR 2024 [2] Gamba: Marry Gaussian Splatting with Mamba. https://arxiv.org/abs/2403.18795 [3] LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation. https://arxiv.org/abs/2402.05054

Questions

1. How to specify the observation viewing direction? Is it arbitrarily selected? How will the specified viewing direction affect reconstruction effects? (e.g. when the observation viewing direction is not good, will the algorithm fail?) 2. The example images shown in the paper use only one character or number for pre-defined patterns. If the watermark information is more complexed and needs multiple characters / numbers, how will the results be like? 3. Regarding weakness, can you discuss how this method can generalize to other single image to 3D reconstruction algorithms? Is there any more generalized way to do watermarking/cloaking ? 4. In method description, the authors mentioned they use "a mask version of PGD", but details on how to generate the mask is not given in the text.

Rating

3

Confidence

4

Soundness

3

Presentation

2

Contribution

1

Limitations

The author has discussed the social impacts. Overall this is a work for copyright protection, which does not have direct negative social impacts.

Summary

This paper proposes a novel method to protect copyrighted images from unauthorized 3D reconstruction using Triplane Gaussian Splatting (TGS). This topic is very interesting and highly valuable for preventing the misuse of copyrighted images. Their proposed method achieves protection by incorporating invisible geometry perturbations into the image, which is easy to deploy with less cost. The extensive experimental results have demonstrated the effectiveness of their approach in preventing unauthorized 3D reconstructions while embedding a verifiable pattern. This paper can provide a contribution to addressing the growing issue of image abuse and raise the community's attention to this issue.

Strengths

1. The motivation of this paper is well-stated and significant, and the paper is well-written. The idea of using geometry cloaks to protect images from unauthorized 3D reconstruction is novel. This paper is an attempt to address the issue of image abuse in 3D reconstruction. Such an issue has not received sufficient attention, but it is very important, especially when 3D reconstruction technologies are becoming more accessible. This paper can also raise the community's awareness of this issue. 2. Besides significantly distorting the 3D reconstruction from the protected images, the idea of embedding identifiable patterns into output renderings is interesting. This allows the image owners to determine whether the generated 3D models have used their copyrighted images. This traceability property can enhance the practicality of the method for digital copyright protection. 3. The paper provides extensive experimental results, validating the effectiveness of the proposed approach across different datasets and perturbation strategies. This thorough evaluation helps in building confidence in the robustness and reliability of the proposed method. The approach is scalable and can be applied to a wide range of images without requiring significant computational resources. This enables artists and content creators to safeguard from being misused illegally.

Weaknesses

1. The paper lacks quantitative metrics comparing the similarity between protected and unprotected images, such as PSNR, which would provide a more comprehensive evaluation of the method's impact on image quality. 2. The importance of the view-specific PGD angle has not been thoroughly explored. It would be beneficial to investigate whether the method is effective from different angles, not just the top view. Besides, the effectiveness of combined attacks on point cloud and triplane latent features should be explored to determine if they provide better protection. 3. Minor Issues: a). Add a period at the end of "Quantitative comparison of perturbation strategies" in the Table 1 caption. b). Replace "no perturbation" with "not perturbed" (L255 P7). c). Correct the citation format "et al" to "et al." (L95 P3).

Questions

Please see the weakness.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

The authors have adequately addressed the limitations

Summary

The paper introduces a novel approach to protect copyrighted images from unauthorized 3D reconstructions using Triplane Gaussian Splatting (TGS). The method involves embedding invisible geometry perturbations, termed "geometry cloaks," into images. These cloaks cause TGS to fail in a specific way, generating a recognizable watermark, thus protecting the original content.

Strengths

1. The concept of protecting images against unauthorized 3D reconstruction using geometric perturbations is novel. 2. This paper is well-detailed and well-written.

Weaknesses

1. The approach is tailored specifically to TGS-based 3D reconstruction. Can the geometry cloak technique be adapted or extended to protect against other 3D reconstruction methods beyond TGS? 2. What are the potential impacts on image quality for legitimate uses when these cloaks are applied? 3. The optimization process for generating geometry cloaks might introduce significant computational overhead, which is not thoroughly discussed in the paper.

Questions

See Weaknesses

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Mentioned in the manuscript.

Summary

The paper introduces a novel image protection approach called "Geometry Cloak" to prevent unauthorized 3D model generation from copyrighted images using single-view 3D reconstruction methods like Triplane Gaussian Splatting (TGS). The Geometry Cloak embeds invisible geometry perturbations into images, which are revealed as a customized message when TGS attempts 3D reconstructions, thus acting as a watermark for copyright assertion.

Strengths

1. This paper raises a novel question, namely how to protect the copyright in the process of image to 3D, which is very meaningful. 2. The presentation of this paper is very clear and easy to understand. 3. A view-specific PGD strategy is proposed to optimize geometry cloak, which is simple but effective. 4. The authors conduct experiments on two 3D datasets and various types of patterns, and verify the effectiveness of the experimental results via sufficient ablation experiments and visualization results.

Weaknesses

1. Can the proposed method be extended to other image to 3D models, such as LRM [1], LGM [2]? The author could introduce the advantages of using TGS instead of other image to 3D models in terms of generation speed and quality, so that the readers can better understand its task scenario. [1] Lrm: Large reconstruction model for single image to 3d. In ICLR 2024. [2] Large multi-view gaussian model. In ECCV 2024. 2. The robustness of the proposed method should be verified. For example, when Gaussian noise and JPEG compression are added to the protected image, can it still resist illegal theft?

Questions

Please refer to the weakness.

Rating

8

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors have clearly presented their limitations.

Summary

This paper presents a method for copyrights in 3D reconstruction, specifically targeting novel-view synthesis, rather than traditional 2D images. Recently, advancements in 3D reconstruction have been driven by neural radiance fields (NeRFs) and 3D Gaussian splatting (3D GS), both of which maintain 3D consistency. These methods primarily focus on learning RGB values from multiple posed images. Notably, recent developments have shown that single-view 3D reconstruction can be achieved with the support of pre-trained diffusion-based generative models. Unlike 2D images, privacy preservation in 3D reconstruction has received less attention due to its recent development. This paper introduces a geometry cloak that perturbs 3D point cloud representations instead of 2D images. This perturbation prevents 3D reconstruction from 2D posed images without degrading the visual quality of the 2D images. The technique is particularly supportive of 3D GS, enabling real-time novel-view synthesis, a capability not supported by existing NeRFs. To the best of our knowledge, this paper is the first to propose an adversarial attack to preserve copyright in 3D GS from a 3D reconstruction perspective.

Strengths

1. This paper addresses contemporary issues regarding copyright protection in 3D reconstruction. Utilizing the framework of Tri-Plane Gaussian Splatting (TGS), which encodes posed images into tri-plane representations and 3D point clouds before generating 3D Gaussian splatting, the paper proposes a novel perturbation method. This method degrades the quality of 3D geometry without affecting the visual quality of 2D images. 2. This demonstrates that the proposed perturbation method significantly impacts the performance of 3D reconstruction, regardless of the degree of perturbation, unlike simple noise injections. It also shows that this method is robust and not sensitive to hyper-parameter variations.

Weaknesses

1. This paper heavily relies on the prior work of Tri-Plane Gaussian Splatting (TGS), which explicitly represents 3D point clouds to enhance geometric properties and employs tri-plane representation to encode 3D Gaussians. Without leveraging TGS, this study would not effectively address copyright issues in 3D reconstruction. This dependency indicates that the proposed approach is not applicable to a wide range of novel-view synthesis techniques, highlighting a limitation in its generalizability. 2. While this paper demonstrates that the proposed method is insensitive to the degree of perturbation, it does not adequately explain this phenomenon in the context of privacy preservation. In the past, understanding the tendency of performance degradation is crucial and be well-explained through probabilistic analysis for the extent of perturbation. However, the experimental results presented in this paper do not support this concept.

Questions

1. Could you show the slope of performance deviation when $\epsilon$ is less than 2? When $\epsilon = \{0, 1\}$, does the proposed method also exhibit a negative slope of performance in terms of PSNR, SSIM, and LPIPS? 2. Could you show the performance degradation of random noise when $\epsilon$ increases beyond 8? It should demonstrate that the proposed approach is more beneficial under strong perturbation. While $\epsilon=8$ indicates color disturbance, the context and shape in the 2D images do not change. 3. Could you present the ablation study on the influences of PGD?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Please refer to weakness.