Unsupervised Graph Neural Architecture Search with Disentangled Self-supervision

Summary

This paper proposed a method DSGAS to automcatiicaly design the architectures in an unsupervised manner. It discovers the optimal architectures by learning the latent graph factor.

Strengths

It is interesting and novel to automate the design of GNNs in an unsupervised manner, which holds significant potential.

Weaknesses

1. The necessity of designing disentanglement is not clear. It is true that GNNs may preferred different graph factors. What is the difference between the proposed method and the baseline that: using one latent factor design one architectures, and then run multiple times and ensembles the search architectures? The motivation for disentangled architectures is not well justified. While the evaluations on K are provided, it would be more effective to achieve K=2 by ensemble two K=1 results. 2. The experiments are insufficient to justify the effectiveness of the proposed design method. Since it designs GNNs in an unsupervised manner, it would be better to conduct fair comparisons with existing unsupervised methods rather than methods designed under a supervised manner. 3. Apart from the comparisons of performance, the comparisons of search cost, the parameters of the searched architectures should be provided as well.

Questions

Please check the weaknesses.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Please check the weaknesses.

Summary

This paper addresses the problem of unsupervised graph neural architecture search, which has received limited attention in existing literature. The authors propose a novel approach called Disentangled Self-supervised Graph Neural Architecture Search (DSGAS) to discover optimal architectures capturing latent graph factors in an unsupervised manner. The DSGAS model consists of a disentangled graph super-network that incorporates multiple architectures with factor-wise disentanglement, self-supervised training to estimate architecture performance under different factors, and a contrastive search method to discover architectures with factor-specific expertise. Extensive experiments on 11 real-world datasets demonstrate that the proposed model achieves state-of-the-art performance compared to several baseline methods.

Strengths

- The paper addresses an important and underexplored problem in the field of graph neural architecture search, i.e., scenarios where supervised labels are not available. - The proposed DSGAS model introduces a novel approach to discovering optimal architectures by leveraging latent graph factors in a self-supervised fashion based on unlabeled graph data. - The disentangled graph super-network and the self-supervised training with joint architecture-graph disentanglement are novel contributions that enhance the understanding and performance of the proposed model. - The extensive experimental evaluation on 11 real-world datasets demonstrates the superiority of the DSGAS model over several baseline methods, showing its effectiveness in unsupervised graph neural architecture search.

Weaknesses

- While the paper presents a novel approach, further details regarding the implementation of the proposed DSGAS model in the main paper would be beneficial for the readers to understand the design clearly. For example, it is better to illustrate the details of the supernet construction in the main paper. - Some typos should be corrected, e.g., heterogenous -> heterogeneous on line 323.

Questions

Please see Weaknesses.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

N/A

Summary

This paper gives a pioneer solution for graph neural architecture search with limited labels. The key idea is to train a super-network containing disentangled factor-wise architectures by a self-supervised learning specially designed for graph neural architecture search. In this way, the paper addresses the key problem of finding the optimal architectures capturing various graph factors in an unsupervised fashion. The extensive experiments show that the method achieves signifiant improvements over the state-of-the-art GraphNAS methods under unsupervised settings. Detailed ablation studies also show the effectiveness of each proposed components.

Strengths

Unsupervised GraphNAS is an important and valuable problem which remains unexplored in the literature. This paper is a timely work to introduce GraphNAS in the scenarios where labels are scarce, and these scenarios are actually quite common in practice. The method design is novel, where the contrastive search module is especially interesting that it pushes together architectures with similar factor expertises and pull away unsimilar architectures by architecture augmentations to explore the search space. The extensive experiments verify its capability of automating the architecture design for the data from various fields.

Weaknesses

1. Although it may fall outside the scope of this paper, i'm curious about whether the model can be applied into areas like AI4Science where labels are quite scarce? 2. Pretraining techniques have shown promising applications in many scenarios like NLP and CV, and they usually have a close relationship with unsupervised or self-supervised techniques. It would better to discuss the relationship between this method and GraphNAS pretraining. 3. The framework diagram (figure 1) is a bit small, especially that the font size which is too small to read clearly. It would be better to enlarge the diagram and increase the font size a bit.

Questions

1. Although it may fall outside the scope of this paper, i'm curious about whether the model can be applied into areas like AI4Science where labels are quite scarce? 2. Pretraining techniques have shown promising applications in many scenarios like NLP and CV, and they usually have a close relationship with unsupervised or self-supervised techniques. It would better to discuss the relationship between this method and GraphNAS pretraining.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I don't see any concerns regarding the societal impact in this work.

Summary

This paper mainly focuses on the problem of graph neural architecture search without labels. The authors find that the key problem is to discover the latent graph factors that drive the formation of graph data as well as the underlying relations between the factors and the optimal neural architectures. To this end, the authors propose the disentangled super-network, and design a self-supervised training and searching paradigm to automate the architecture design. The experiment results are significant and the deeper analyses are convincing.

Strengths

- This paper mainly focuses on the problem of graph neural architecture search without labels, which is important yet underexplored in the literature. - The proposed methods sound reasonable. - The experiment results are significant and the deeper analyses are convincing.

Weaknesses

- Can the authors explain the details of the intuition for the proposed contrastive search? - The experiments include both node classification and graph classification tasks. Do these tasks share the same search space? - As the existing graph NAS methods are in need of labels for training and searching, how do the authors modify them to compare as baselines? - What about the complexity of the method?

Questions

Please see the weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

None.

Summary

In this paper, the authors study the problem of unsupervised graph neural architecture search, which remains unexplored in the literature. The authors propose a novel Disentangled Self-supervised Graph Neural Architecture Search (DSGAS) model, which is able to discover the optimal architectures capturing various latent graph factors in a self-supervised fashion based on unlabeled graph data. Extensive experiments on 11 real-world datasets demonstrate that the proposed DSGAS model is able to achieve state-of-the-art performance against several baseline methods in an unsupervised manner.

Strengths

1. I believe the problem of unsupervised graph neural architecture search is important, and this work is significant that it may unlock more NAS applications in practice. 2. The writing is clear and well-organized in general. 3. The experments and analyses are extensive.

Weaknesses

Please see my comments in the questions part.

Questions

1. What is the relationship between supervised and unsupervised nas paradigms in terms of the optimization problem, e.g., line 84 in the main paper? 2. In Figure 1, the super-network seems a simple DAG, why do the searched architectures in Figure 4 seem kind of complex in terms of the operation connections? 3. In Figure 1, it seems that the mixed operation has different colors for different $\alpha_i$, is it possible that some factor-wise architectures share the same operation? 4. In Figure 1, is it required that all the architectures should choose the same augmentations? In other words, is it possible that some architectures adopt operation choice perturbation, while others adopt weight perturbation in the same batch for contrastive learning?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

N/A

Thanks for your reply.

Thanks for your detailed response. All my concerns have been resolved. Besides, after reading other reviews and your reply, I found the NAS disentangled mechanism is pretty interesting and sounds good. I would like to raise my score.