Intruding with Words: Towards Understanding Graph Injection Attacks at the Text Level

Summary

The paper studies Graph Injection Attacks on text-attributed graphs. The study presents three attack designs: Vanilla Text-level GIA (VTGIA), Inversion-based Text-level GIA (ITGIA), and Word-frequency-based Text-level GIA (WTGIA). The key contributions include demonstrating the effectiveness of text level perturbation on Graph Injection Attack and the performance of using LLMs as a defender. WTGIA shows significant attack potential while maintaining understandability. It is a timely study on the vulnerabilities of text attributes in graphs.

Strengths

1. Good investigation on the role of text manipulation in graph injection attacks. The text attributes in graphs are getting increasingly important. Demonstrating their vulnerabilities to GIA and the role of LLM can play as a defender are important to applications using text-attributed graphs. 2. The proposed WTGIA method achieves good attack performance with understandable text perturbation.

Weaknesses

1. The embedding-text-embedding process is not very clear to me. Additional explanation will be helpful. It is unclear how different embedding models may affect the performance of this process. 2. The term interpretability in this paper has different semantics to the commonly used one in explainable AI. It is confusing and creates unnecessary obstacles in reading. It is better replace with understandability.

Questions

1. What is the computing complexity of WTGIA? How long does it take to generate an attack in average in your experiment environment? 2. What is the impact of different text embedding models on the attack performance?

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

Limitations on transferability is discussed in the paper.

Summary

This paper investigates an interesting topic. The authors first discuss the practicality of Graph Injection Attacks (GIA), arguing that previous approaches, which merely inject harmful embeddings, are unrealistic. It introduces three text-level GIA methods: ITGIA, VTGIA, and WTGIA, and conducts a thorough examination of their attacking performance and interpretability. The findings reveal that WTGIA excels in both attacking effectiveness and interpretability. Additionally, the paper explores the defensive capabilities of Large Language Models (LLMs) and discovers that they can effectively evade the impact of injected nodes. This underscores the importance of enhancing the performance of text-level attackers.

Strengths

1. This paper first addresses the impracticality of embedding-based GIAs and explores text-level GIAs, introducing three novel text-level GIA methods. Among these, WTGIA effectively balances attack performance and interpretability. 2. The authors have conducted comprehensive experiments that thoroughly address the research questions posed in the study. 3. The theorem discusses the conditions under which destructive texts can be identified, providing a clear theoretical foundation for the work. 4. This paper offers valuable insights into the new challenges associated with text-level GIAs, identifying emerging directions in this field.

Weaknesses

1. The authors should provide references for the cited papers on TDGIA, ATDGIA, MetaGIA, and AGIA. 2. The authors should explain why the attack performance initially decreases and then increases as the sparsity budget increases, as shown in Figure 5. 3. The authors should clarify how the LLM optimizes the graph structure after generating the text for the injected nodes.

Questions

1. For VTGIA, how does the LLM optimize the graph structure after generating the text for the injected nodes? 2. The attacking method of WTGIA is interesting. Could this idea inspire other areas of GNN? 3. In Section 4, could you clarify the embedding-text-embedding process? Specifically, in Section 3, under Analysis of Performance and Interpretability, it seems that after inversion, text is generated. What embedding results after this inversion? 4. Why do you introduce FGSM to generate binary embeddings? Is it not possible to directly transform the embeddings into binary embeddings? 5. In Figure 5, why does the attack performance initially decrease and then increase as the sparsity budget increases?

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

Some parts of Section 4.1 are hard to understand, I admit I didn't check the proof related to the theorem in the Appendix. I think the authors could add figures or improve their presentations to better illustrate WTGIA.

Summary

The paper studies GIAs, particularly focusing on text-attributed graphs (TAGs). It introduces a method for GIAs by injecting textual content directly into the graph, as opposed to the traditional method of embedding-level attacks. The authors propose three new attack designs: Vanilla Text-level GIA (VTGIA), Inversion-based Text-level GIA (ITGIA), and Word-frequency-based Text-level GIA (WTGIA). They demonstrate that text interpretability is necessary for the effectiveness of these attacks. They also show that defenses can be enhanced using customized text embedding methods or large language model (LLM)–based predictors.

Strengths

I found this to be a very original paper as it shifts from embedding-level GIAs to text-level GIAs. Three new attacks are created and are tested theoretically and empirically. Most importantly, the GIAs created ensure that the attacked nodes have semantically useful information, making the attack more realistic, using LLMs.

Weaknesses

The paper does not explore the robustness of the proposed GIAs against more sophisticated defense mechanisms beyond customized text embedding methods and LLM-based predictors. Although the authors mention that LLM-based defenses enhance protection against text-level GIAs, the experiments primarily focus on a few specific methods and datasets. These are limited in scope. The trade-offs between performance, unnoticeability, and interpretability, as discussed in Sections 4.1 and 4.2, could be examined more thoroughly. While the paper shows that increasing the sparsity budget enhances attack performance but reduces interpretability, further experiments across a wider variety of datasets could validate these findings more convincingly, especially with additional baselines. Including more traditional GIA methods or other recent advancements in graph attack techniques would also be useful. The discussion on the practical implementation of these attacks in real-world applications could be expanded. The paper mainly provides experimental insights but lacks detailed exploration of the real-world scenarios you stake your paper around, and practical challenges like the scalability of the proposed methods to larger and more complex graphs, the computational costs associated with implementing these attacks, and potential detection mechanisms in real-world systems are not thoroughly addressed. Providing more actionable insights and guidelines for practitioners could significantly enhance the practical relevance of the research. Lastly, while the authors discuss the potential for enhancing defenses using customized text embedding methods or LLM-based predictors, they do not explore the possibility of integrating multiple defense strategies. Combining different defense mechanisms could potentially create more robust protection schemes against text-level GIAs. Typos: line 296 and line 211

Questions

1. Can the authors provide more details on how the sparsity budget impacts the interpretability and effectiveness of the text-level GIAs across different datasets? 2. How would this work apply to dynamic graphs where nodes and edges evolve over time? 3. Are there any challenges in scaling the proposed GIAs to larger graphs or more complex datasets?

Rating

8

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

N/A

Summary

This paper explores the vulnerability of GNNs to Graph injection Attacks (GIAs), which involve injecting malicious nodes into a graph. This paper explores GIAs at the text level, presenting three types of GIA designs that inject textual content into the graphs. The significance of text interpretability in attack effectiveness and the defense against text-level GIAs are also explored.

Strengths

1. The idea is straightforward and easy to follow. 2. The study of text-level graph injection attacks is interesting and practical.

Weaknesses

1. In line 126–131, the paper claims that traditional methods focus mainly on the embedding level. However, some existing works (e.g., [1]) focus on the raw feature level. The authors fail to discuss why these works cannot be applied to this task. 2. Limited technical contribution. In this paper, three kinds of text-level GIA are proposed. However, I think all of them are simply combining both existing GIA techniques and language models, which are kind of incremental contributions. 3. I think the proposed text-level GIA is similar to the traditional GIA, especially when the nodes' texts are encoded into embeddings by LMs. Therefore, I think some existing defense methods against GIA can be also applied to this task. The authors need to discuss that. 4. The experiments only focus on small-scale datasets. More large-scale datasets need to be included. 5. Only vanilla GCN is used in the experiments, which may induce to biases to the conclusions. More advanced GNNs need to be included. [1] Understanding and Improving Graph Injection Attack by Promoting Unnoticeability. ICLR 2022.

Questions

Refer to weaknesses.

Rating

4

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

A section to discuss the limitations is missing.

Response to Authors

Thanks for the detailed response. I am pleased to note that you have addressed all the concerns I raised. I appreciate your efforts in clarifying these points and I have raised the scores.