Learning Disentangled Representations for Perceptual Point Cloud Quality Assessment via Mutual Information Minimization

Summary

This paper explores the method for No-Reference Point Cloud Quality Assessment. The key idea is to involve the disentangled representation learning to minimize mutual information between representations of point cloud content and distortion. The authors conduct experimental performance comparisons on three public databases and compare their proposed method with 15 existing models. The proposed method achieves optimal or suboptimal results in most of the metrics. The ablation study demonstrated the necessity of each design part.

Strengths

The paper is clearly written and provides a detailed formulation of the approach.

Weaknesses

The weak point of the paper is its presentation. Many terms are introduced without explaining them properly, e.g. "the tight upper bound" or "the masked autoencoding strategy”. Furthermore, the Fig 2 is not well-designed with content aware branch. The Fig2 splits the proposed architecture and the content-aware pretraining and masked autoencoding strategy, which does not help the reader to understand architecture with figure. Lastly, it would be highly beneficial to make the code publicly available to enhance collaborative efforts and facilitate the sharing of this work. Apart from this, there are quite a few typos and grammar mistakes that should be corrected, such as “the can be”, “masked auto-encoding / autoencoding

Questions

See Weaknesses.

Rating

4

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

See Weaknesses.

Summary

This paper proposes a novel no-reference quality assessment model tailored for point-cloud data. A disentangled representation learning strategy is leveraged to account for both content-aware information and distortion-aware information. Comprehensive experiments are conducted and the effectiveness of this proposed model is well verified.

Strengths

The proposed framework is novel and the paper is easy to understand.

Weaknesses

1. The idea of framing the quality assessment into content-/semantic-aware and distortion-aware aspects is actually quite common in IQA/VQA on 2D visual data. It’d be better to review some related QA metrics for 2D data, and analyze the key difference on the implementation of this idea between point-cloud data and 2D data quality assessment. 2. Lack of comparison on computational complexity.

Questions

Please see the Weaknesses.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Please see the Weaknesses.

Summary

This paper proposes a novel disentangled representation learning framework called DisPA to decouple the representation learning process of point cloud content and distortion. To sufficiently disentangle these two representations, the DisPA uses two branches to learn them and adopt different training philosophies separately. For the content-aware branch, DisPA pretrains one encoder using a proposed masked auto-encoding strategy, which partially masks the images projected from distorted point clouds and reconstructs the corresponding patches of the images projected from pristine point clouds; For the distortion-aware branch, the DisPA integrates the mini patches of the rendered multi-view images into a mini-patch map, which can focus on local distortions and ignore the global point cloud content. Furthermore, to disentangle the learned representations, the DisPA uses a trainable mutual information estimator to estimate the mutual information (actually the tight upper bound) between these two branches and further minimize it alternatively along with the training of the main network (i.e., the two encoders and regression layers). Finally, the experimental results demonstrate the superior performance of DisPA in terms of both prediction accuracy and generalizability.

Strengths

+ This is the first paper exploring disentangled representation learning for PCQA. The disentanglement is reasonable and even necessary because the point cloud content and distortion are differently perceived by humans. + The proposed methodology is well-motivated. The masked auto-encoding strategy can intuitively make the encoder capture the point cloud content information. + The paper is written well. The motivation of why disentangled representation learning is essential for PCQA has been introduced clearly from the observation of human vision systems. + The mathematical derivations are detailed and easy to follow, including the approximation of MI in Section 3 and further proof in Appendix A.

Weaknesses

1. From my viewpoint, the DisPA can be totally used for IQA, or even more suitable, since this work does not process 3D native point cloud data but just projects point clouds into images and uses 2D networks. Have the authors tried to introduce unique attribute information that is closely related to point clouds? 2. The whole DisPA is based on projections of point clouds, so the number of viewpoints is very important to the quality score prediction. However, the authors did not discuss the impact of the number of viewpoints. 3. How much time does it take to pretrain the content-aware encoder? The authors did not discuss the pretraining details in the implementation details part. 4. Why does not the LS-PCQA follow the K-fold data splitting? The authors should explain this for the loss of a unified experiment setting. 5. What is the actual architecture of the MI estimator? And what is the relation between the MI estimator and the lightweight neural network Q_phi? The idea of alternative training of the MI estimator and the main network is easy to understand, but the components of the MI estimator need more clarification. Is it just simple MLPs? 6. The mini-patch map generation has been used in many papers [1,2,3]. However, considering this is a minor contribution and the mini-patch map is actually effective for learning disentangled representations, this limited contribution is acceptable. 7. A minor weakness: There are some small flaws (gray bounding box) with the presentation of Figure 5 when zooming in. Please ensure all figures can be presented clearly without error. [1] Wu, Haoning, et al. "Exploring video quality assessment on user generated contents from aesthetic and technical perspectives." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023. [2] Wu, Haoning, et al. "Fast-vqa: Efficient end-to-end video quality assessment with fragment sampling." European conference on computer vision. Cham: Springer Nature Switzerland, 2022. [3] Zhang, Zicheng, et al. "Gms-3dqa: Projection-based grid mini-patch sampling for 3d model quality assessment." ACM Transactions on Multimedia Computing, Communications and Applications 20.6 (2024): 1-19.

Questions

1. Why the conditional distribution p(y|x) is unavailable in your case? This needs more explanations, because the mathematical analysis is somewhat abstract so more details are needed to make it more understandable. 2. What is the advantanges of the differential ranking loss function? 3. Can this work be trained end-to-end for IQA or video quality assessment (VQA) without modification of specific modules (just replacing the input/output)? 4. Can the masked auto-encoding strategy use natural images for pretraining?

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the authors have adequately addressed the limitations and discussed the potential negative societal impact in the appendix D Limitations and Future Work.

Summary

This article's motivation is interesting. It combines content-aware and distortion-aware characteristics to train a 3D quality assessment network. Additionally, it employs a MAE-based method to train a content-aware encoder, uses patches to focus the network on learning distortion, and applies an MI module to integrate both features.

Strengths

1. The authors first analyze the shortcomings of existing networks, specifically that data imbalance leads to overfitting in current methods, resulting in poor processing of other content images with the same degradation. Therefore, the authors aim to use an MAE-based method to learn content-related features. 2. The proposed key MI-based regularization is effective. 3. The presented methods can obtain impressive results on multiple datasets.

Weaknesses

1. Regarding the masked part, there is a gap between the first and second stages because the input in the first stage is a partially masked image, while the input in the second stage is the complete image. 2. I am also a bit confused about why the constraint in the first stage is a clean/reference image. This would give the masked encoder the characteristic of restoration, while the core of quality assessment is to evaluate the quality of the image. If the image features are restored, will it affect the accuracy of the image quality assessment? 3. The motivation for the MI part should provide more details and explanations, which leaves me somewhat puzzled. 4. I am not quite sure what the distortion-aware encoder has learned. Is it truly related to degradation features? This might require some verification.

Questions

This section corresponds to the Weaknesses: 1. How does the network address the gap between the input images in the first and second stages? 2. Why is a clean image used as a constraint? This results in learning the restoration characteristics, which is not very helpful for quality assessment. 3. The motivation for MI needs further explanation. 4. Visualize or analyze the distortion-aware features.

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

No potential negative societal impact. Please address the questions/suggestions above.

Response to the authors.

Thank you for your detailed responses. I apologize for missing the fine-tuning process in the first question. In fact, applying the fine-tuned operation for MAE is general, and I thought this step was omitted in the paper, which caused my confusion. Other doubts have also been resolved, and I will improve my score to weak accept.