DisenGCD: A Meta Multigraph-assisted Disentangled Graph Learning Framework for Cognitive Diagnosis

Existing graph learning-based cognitive diagnosis (CD) methods have made relatively good results, but their student, exercise, and concept representations are learned and exchanged in an implicit unified graph, which makes the interaction-agnostic exercise and concept representations be learned poorly, failing to provide high robustness against noise in students' interactions. Besides, lower-order exercise latent representations obtained in shallow layers are not well explored when learning the student representation. To tackle the issues, this paper suggests a meta multigraph-assisted disentangled graph learning framework for CD (DisenGCD), which learns three types of representations on three disentangled graphs: student-exercise-concept interaction, exercise-concept relation, and concept dependency graphs, respectively. Specifically, the latter two graphs are first disentangled from the interaction graph. Then, the student representation is learned from the interaction graph by a devised meta multigraph learning module; multiple learnable propagation paths in this module enable current student latent representation to access lower-order exercise latent representations, which can lead to more effective nad robust student representations learned; the exercise and concept representations are learned on the relation and dependency graphs by graph attention modules. Finally, a novel diagnostic function is devised to handle three disentangled representations for prediction. Experiments show better performance and robustness of DisenGCD than state-of-the-art CD methods and demonstrate the effectiveness of the disentangled learning framework and meta multigraph module. The source code is available at \textcolor{red}{\url{https://github.com/BIMK/Intelligent-Education/tree/main/DisenGCD}}.

Paper

References (49)

12Evolutionary Neural Architecture Search for Transformer in Knowledge Tracing2023

Scroll for more · 37 remaining

Similar papers

Peer review

Reviewer 53z45/10 · confidence 4/52024-07-10

Summary

The paper studies cognitive diagnosis based on graph learning. The proposed method utilizes multiple graphs to assist the representation learning. Specifically, the authors further disentangle another two graphs from the most comprehensive student-exercise-concept interaction graph by removing certain node and edge types. Therefore, three kinds of disentangled representations are learned based on the three graphs. Lastly, a diagnostic function fuses the three representations for final predictions.

Strengths

1. The problem that the paper aims to solve is well formulated and it is of practical value. 2. The authors give very detailed illustrations of the methodology part. 3. Comprehensive experiments are conducted to demonstrate the effectiveness of the proposed method.

Weaknesses

1. Since the proposed method learns the graph representations based on disentangled graphs, the authors are encouraged to add a section about disentangled graph representation learning in the related work part to summarize the recent works on this problem and illustrate the relation of this work with previous ones. 2. A more detailed introduction to incorporated datasets should be given. Specifically, both accuracy and RMSE are employed as evaluation metrics, the authors should provide a clearer illustration of the target tasks of those datasets. 3. The authors should include more mode variants to demonstrate the effectiveness of the added graphs (e.g., w/o G_R, w/o G_D, and w/o both). 4. GAT is used in this work to learn exercise and concept representations. The authors are encouraged to introduce the reason for this choice or conduct experiments based on other graph model backbones. 5. There are many typos in the paper, e.g., line 16 and line 254. The authors should check the grammar carefully.

Questions

Please refer to the weakness part.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

1

Limitations

N/A

Reviewer 7tDb7/10 · confidence 4/52024-07-11

Summary

In this paper, the authors introduce a meta multigraph assisted disentangled graph cognitive diagnosis model. Its main contribution is to propose a disentangled graph framework, which disentangles the student-exercise-concept dependency graph into an exercise-concept interaction graph and a concept-concept relationship graph to improve the robustness of representation modeling. Additionally, the concept of meta multigraph is introduced in the student-exercise-concept dependency graph to assist in better modeling of student representation. DisenGCD has demonstrated good performance compared to previous models in multiple datasets, and has shown good robustness in the presence of noise interference or sparse datasets. Finally, the authors also discussed some limitations of DisenGCD in terms of computational complexity and task transferability.

Strengths

1. The paper is well-organized and clearly written, making it accessible to readers. Besides, the authors also present mathematical formulations, detailed explanations of the proposed models. 2. The authors focus on improving the robustness of the CD model and provide a detailed analysis of some problems found in previous graph cognitive diagnosis works. 3. The experiments are very thorough. The authors demonstrate the robustness of the model through decoupling experiments, sparse dataset experiments, and noisy dataset experiments

Weaknesses

1. Although DisenGCD has demonstrated excellent performance across multiple datasets, it still appears to have the drawback of high computational complexity, which may limit its application in practical scenarios. 2. Although the model has shown improved performance, its interpretability may be lacking. In real-world applications within the education field, transparent and interpretable models are more likely to gain acceptance and trust. However, the paper does not address methods for enhancing the interpretability of the model. 3. The paper lacks detailed discussion on the challenges and potential issues that may arise when applying the model in practical educational settings. For instance, it does not address the strategies for handling data privacy and security concerns, which are critical considerations in the implementation of such models.

Questions

1. KaNCD seems to have been found to work equally well in some papers. Can you compare DisenGCD and KaNCD? 2. The paper utilizes graph attention networks (GAT) to learn representations of exercises and concepts. However, it would be valuable to explore and compare alternative graph neural network (GNN) models, such as graph convolutional networks (GCN) or LightGCN, to assess their performance in the same context. 3. How are the multiple learnable propagation paths in the meta-multigraph learning module designed and selected? Have you tried different numbers or types of propagation paths and how well have they worked?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

\

Authorsrebuttal2024-08-13

Thank you for your quick feedback and for contributing to our improved score. We greatly appreciate your efforts. We apologize that some issues still need improvement, and we would like to resolve them as follows. ## Response to Weakness 1 To investigate the computational efficiency of the proposed approach, we have compared it with RCD on ASSISTments and Math datasets in terms of model inference time and training time. **Table III** in ***global.pdf*** presents the overall comparison, where the time of the proposed DisenGCD under different hyperparameters *P* is also reported. As can be seen, DisenGCD's inference time is better than RCD's. This indicates that the proposed DisenGCD is more efficient than RCD, further signifying that it is promising to extend DisenGCD to dynamic CD. It can be seen from **Table III:Lower**: although a larger *P* will make DisenGCD take more time to train the model, the proposed DisenGCD achieves the best performance on both two datasets when *P*=5 and its runtime does not increase too much, which is much better than RCD. In summary, the above comparison shows the model efficiency superiority of the proposed DisenGCD. ## Response to Weakness 2 We have to argue that the diagnostic function in the proposed DisenGCD is interpretable. Firstly, $\mathbf{h}\_{si} = F_{si}(\overline{\mathbf{S}_i}+ \overline{\mathbf{C}_k})$ denotes the combination of the learned student representation and the learned concept representations, which is similar to the combination in RCD, i.e., $Concat(\mathbf{S}\_i, \mathbf{C}_k)$. As a result, $\mathbf{h}\_{si}$ can be seen as the student's mastery of each knowledge concept. Secondly $\mathbf{h}\_{ej} = F\_{ej}(\overline{\mathbf{E}_j}+\overline{\mathbf{C}_k})$ denotes the combination of the learned exercise representation and the learned concept representations, which is also similar to the combination in RCD for exercises, i.e., $Concat(\mathbf{E}\_j, \mathbf{C}_k)$. Therefore, $\mathbf{h}\_{ej}$ can represent the exercise difficulty of each concept. Thirdly, $\mathbf{h}\_{simi} = \sigma(F\_{simi}(\mathbf{h}\_{si}\cdot\mathbf{h}\_{ej} ))$ is mainly used to measure the similarity between $\mathbf{h}\_{si}$ and $\mathbf{h}\_{ej}$ via a dot-product followed by an FC layer and a Sigmoid function. Therefore, a higher value in each bit of $\mathbf{h}\_{simi}$ represents a higher mastery on each concept, further indicating a higher probability of answering the related exercises. Finally, $\hat{r\_{ij}} = (\sum Q_k\cdot \mathbf{h}_{simi})/\sum Q_k$ has a similar idea to NCD to compute the overall mastery averaged over all concepts contained in exercise $e_j$. In summary, the proposed model is well-interpretable, as agreed by **Reviewer 2m1M**. ## Response to Weakness 3 We have to admit that data privacy and security are indeed a crucial problem in intelligent education. However, existing cognitive diagnosis models did not explore this. To the best of our knowledge, we think the following two potential techniques can be applied to CD to solve data privacy and security issues. 1. The first is to apply the differential privacy technique. Differential privacy [R1] protects individual data by adding noise, ensuring personal information can’t be easily inferred from data analysis. Differential privacy is widely used in fields like computer vision [R2], recommender systems [R3,R4], and many others [R5]. Among these, the recommendation task is most related to cognitive diagnosis (CD), and thus the differential privacy can be easily extended to CD by summarizing the experiences of these approaches. A successful example of extending techniques in the recommendation to CD is the MF [R6] model. 2. The second is to apply the Federated Learning technique. It trains a model across multiple devices without sharing data, improving privacy and efficiency. It has succeeded in many domains, including recommender systems [R7]. According to its successful experiences in the recommendation, federated learning can also be extended to CD. Moreover, a few studies [R8] have applied federated learning to knowledge tracing (KT), making good progress. KT is the most related task to CD in intelligent education; thus, their experiences can be brought to CD to solve data privacy and security problems. To sum up, the proposed DisenGCD and existing CDMs could be combined with differential privacy and federated learning to solve the data privacy and security problem in intelligent education. We will discuss this problem in revised paper. ``` [R1] Dwork, Differential privacy. [R2] Mixed differential privacy in computer vision. [R3] A differential privacy framework for matrix factorization recommender systems. [R4] Applying differential privacy to matrix factorization. [R5] Differential privacy: A survey of results. [R6] Matrix factorization techniques for recommender systems. [R7] Federated recommendation systems. [R8] Federated deep knowledge tracing. ```

Reviewer 9V3d5/10 · confidence 3/52024-07-12

Summary

This paper introduces DisenGCD, a new framework for cognitive diagnosis in educational contexts. The authors make several contributions: 1) They propose a disentangled graph learning approach, separating the typically unified graph into three distinct graphs: student-exercise-concept interactions, exercise-concept relations, and concept dependencies. 2) They develop a meta multigraph module for learning student representations, which innovatively allows access to lower-order exercise representations. 3) They employ GAT-based modules to learn exercise and concept representations on the disentangled graphs. 4) They design a new diagnostic function to effectively combine the learned representations from the different graphs. The authors evaluate DisenGCD on multiple datasets, demonstrating performance improvements and enhanced robustness compared to state-of-the-art methods in cognitive diagnosis.

Strengths

The paper presents several notable strengths: 1) The proposed disentangled graph learning approach is a novel contribution to the field of cognitive diagnosis. By separating different types of information (student-exercise-concept interactions, exercise-concept relations, and concept dependencies) into distinct graphs, the authors have created a more robust model that appears less susceptible to noise in the data. This is a significant advancement over existing unified graph approaches. 2) The meta multigraph module introduces an innovative mechanism for student representation learning. By allowing access to lower-order exercise representations, this module potentially captures more nuanced relationships in the data, which could be particularly valuable in educational contexts where student knowledge evolves over time. 3) The empirical evaluation is comprehensive and well-executed. The authors conduct experiments on multiple datasets (ASSISTments, Math, and SLP), demonstrating consistent improvements in performance and robustness over state-of-the-art methods. This thorough evaluation strengthens the paper's claims and increases confidence in the generalizability of the proposed method. 4) The paper includes detailed ablation studies and analyses that effectively validate the different components of DisenGCD. These studies provide valuable insights into the contribution of each component and help justify the design choices made by the authors.

Weaknesses

1) While the disentangled graph approach is innovative, it may lose some valuable cross-entity interactions. The separation of student-exercise interactions from exercise-concept relations could potentially limit the model's ability to capture complex, multi-hop relationships that might exist in a unified graph structure. 2) The meta multigraph module, although showing improvements, adds considerable complexity to the model. It's not clear if this added complexity is always justified, particularly for simpler datasets or scenarios with limited noise. A more thorough analysis of the trade-off between model complexity and performance gains would strengthen the paper. 3) The paper lacks a comprehensive discussion on how DisenGCD handles cold-start problems, especially for new students or exercises that don't have established interactions in the graph structure. This is a common challenge in educational settings and needs to be discussed.

Questions

1) How does DisenGCD adapt to dynamic changes in the knowledge structure, such as the introduction of new concepts or the discovery of new relationships between existing concepts? Does the disentangled structure present any challenges in updating the model under these scenarios? 2) In Section 4.1, you describe the meta multigraph as containing P hyper-nodes, and in your experiments you set P to 5. How did you determine this value? Have you explored the impact of different values of P on the model's performance and computational efficiency? 3) While your experiments demonstrate improved robustness to interaction noise, how does DisenGCD perform when noise is introduced in the concept dependency graph or exercise-concept relation graph? This could simulate errors in curriculum design or expert knowledge. 4) Have you investigated whether the disentangled approach leads to any loss of performance in scenarios where complex, cross-entity relationships are crucial for accurate diagnosis? Are there cases where a unified graph approach might outperform DisenGCD?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

N/A

Reviewer 2m1M5/10 · confidence 1/52024-07-13

Summary

The paper presents DisenGCD, a novel cognitive diagnosis model designed to enhance the robustness and accuracy of student, exercise, and concept representations by leveraging a meta multigraph-assisted disentangled graph learning framework. DisenGCD constructs and disentangles three specific graphs for interactions, relations, and dependencies, thus mitigating the negative impact of noise in students’ interactions. The framework incorporates a meta multigraph learning module and GAT to improve the learning of latent representations, demonstrating superior performance and robustness compared to state-of-the-art CDMs on multiple datasets.

Strengths

1. The use of a meta multigraph-assisted disentangled graph learning framework is a novel approach that significantly enhances the robustness of cognitive diagnosis models. 2. DisenGCD shows better performance in terms of AUC, accuracy, and RMSE compared to other state-of-the-art CDMs. 3. The model effectively handles noise in student interactions, ensuring that the learning process remains accurate and reliable. 4. The experiments cover multiple datasets and different data splits, providing a thorough validation of the model’s effectiveness. 5. The diagnostic function of DisenGCD maintains high interpretability, comparable to traditional models like NCD, IRT, and MIRT.

Weaknesses

1. The implementation of the meta multigraph learning module and the disentangled graph learning framework could be complex and resource-intensive. 2. The paper does not report error bars or other statistical significance measures, which are important for validating experimental results. 3. The comparison does not include some recent models like SCD, which could provide a more comprehensive evaluation. 4. While some limitations are discussed, a more detailed analysis of potential weaknesses and future improvements could enhance the paper’s transparency and reliability.

Questions

1. Can the model be adapted to handle real-time data for dynamic cognitive diagnosis? 2. How does the choice of hyperparameters affect the model’s performance and robustness? 3. Can the disentangled graph learning framework be integrated with other neural network architectures for further performance enhancement?

Rating

5

Confidence

1

Soundness

3

Presentation

3

Contribution

3

Limitations

NA

Reviewer 53z42024-08-10

Thanks for the detailed response from the authors. After reading the rebuttal, I decide to raise my score to 5.

Reviewer 9V3d2024-08-12

In general, I am satisfied with the answers. I also think the paper studies an interesting domain where GNNs are applied. I will keep my score, but the paper seems to be above the acceptance bar.

Authorsrebuttal2024-08-13

Many thanks for taking the time to carefully read the rebuttal. We appreciate your recognition of the value of our work! We are pleased to address your concerns and will revise the paper according to your suggestions.

Reviewer 7tDb2024-08-13

The author has addressed some of my concerns, and I am inclined to raise my score. However, some issues still need further improvement in the final version.

Reviewer 2m1M2024-08-14

Thanks for the rebuttal. I have read through it and appreciate the response. I will keep my score unchanged.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC