Robust Offline Active Learning on Graphs

We consider the problem of active learning on graphs, which has crucial applications in many real-world networks where labeling node responses is expensive. In this paper, we propose an offline active learning method that selects nodes to query by explicitly incorporating information from both the network structure and node covariates. Building on graph signal recovery theories and the random spectral sparsification technique, the proposed method adopts a two-stage biased sampling strategy that takes both informativeness and representativeness into consideration for node querying. Informativeness refers to the complexity of graph signals that are learnable from the responses of queried nodes, while representativeness refers to the capacity of queried nodes to control generalization errors given noisy node-level information. We establish a theoretical relationship between generalization error and the number of nodes selected by the proposed method. Our theoretical results demonstrate the trade-off between informativeness and representativeness in active learning. Extensive numerical experiments show that the proposed method is competitive with existing graph-based active learning methods, especially when node covariates and responses contain noises. Additionally, the proposed method is applicable to both regression and classification tasks on graphs.

Paper

References (39)

Scroll for more · 27 remaining

Similar papers

Peer review

Reviewer hH2h5/10 · confidence 3/52024-07-12

Summary

This paper proposes an offline active learning method that selects nodes to query by explicitly incorporating information from both the network structure and node covariates. This paper establishs a theoretical relationship between generalization error and the number of nodes selected by the proposed method.

Strengths

1. The theoretical analysis is sufficient. 2. Offline graph active learning is important. 3. The proposed method is easy to implement.

Weaknesses

I appreciate the author's theoretical analysis, but the experimental section is clearly insufficient. Recent work on graph active learning[1] has been conducted on larger-scale datasets like Arxiv and Products. Conducting experiments solely on datasets such as Cora is not adequate. [1] Partition-based active learning for graph neural networks. TMLR 2023.

Questions

Please provide detailed experimental results for additional datasets.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors adequately addressed the limitations and potential negative societal impact of their work.

Reviewer 7FJA6/10 · confidence 2/52024-07-12

Summary

The paper proposes a strategy for collecting labeled data for a semi-supervised learning algorithm focused specifically on learning on graphs. The paper provides a theoretical analysis of the proposed method, capturing both the quality of the samples that are selected for labeling as well as the the prediction error of the overall learning procedure. Experimental results show that the method the applicability of the method to real-world datasets.

Strengths

- The method proposed in the paper, as well as the problem setting are explained clearly. - The paper provides theoretical guarantees for the method that indicate its superiority compared to random sampling (Theorem 2) and characterize the error rate that can be achieved with this active SSL strategy (Theorem 3).

Weaknesses

- The empirical analysis does not very convincingly suggest that the proposed method is better than the baselines considered. Moreover, it would be helpful if the figures showed confidence intervals or error bars. - The experimental results only compare with a few heuristics for data collection. It would be informative to consider other AL works proposed in the graph learning literature as baselines. - It is not very clear how the paper is positioned in the literature. It would help to have a related work section that can indicate prior works on AL and active SSL on graphs. - The clarity of the section 3 could potentially be improved, perhaps by reducing the amount of symbols to the ones that are strictly necessary and providing more clearly marked (e.g. with paragraph titles) intuitive descriptions of the steps that need to be taken and the obstacles that need to be overcome. Minor remarks: - lines 19-21: 3 different learning paradigms are mentioned in the first two sentences (active, semi-supervised and transductive learning). It would help if it was clearer early on how they are relevant for the problem that motivates this work. - line 89: undefined symbol $\mathcal{B}$

Questions

- What is the computational efficiency of the proposed method? - How tight is the upper bound in Theorem 3? Would it be possible to compare it to numbers from simulations on some simple synthetic settings? - How does the method compare to online AL methods for graphs (e.g. [36, 37, 30] etc) or other offline AL methods for graphs?

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

While the paper discusses some of the limitations of the proposed approach, it would be good to have a more detailed section on the computation cost of running the method as well as show the impact of various hyperparameters of the method on performance (e.g. m, various properties of the network and the generating process for (X, Y) etc).

Reviewer emFb5/10 · confidence 4/52024-07-13

Summary

The paper addresses the challenge of active learning on graphs where labeling node responses is costly. The authors propose an offline active learning method that selects nodes by incorporating both network structure and node covariates. The method leverages graph signal recovery theories and random spectral sparsification, employing a two-stage biased sampling strategy to balance informativeness and representativeness.

Strengths

- The paper introduces a novel offline active learning approach that integrates network structure and node covariates. - The proposed method is validated through extensive experiments on both synthetic and real-world datasets, showcasing its robustness and effectiveness.

Weaknesses

- How does the proposed method perform on networks with varying levels of homophily and heterophily? Can it adapt to different types of network structures? - Can the method be extended to online active learning scenarios, where nodes are queried in a sequential manner rather than in batches? - How does the performance of the proposed method compare with state-of-the-art graph neural network-based active learning methods [1] in a broader range of network topologies and noise conditions? - Is there a significant computational overhead associated with the two-stage biased sampling strategy, and how does it impact the scalability of the method for large-scale networks? [1] Focus on Informative Graphs! Semi-supervised Active Learning for Graph-level Classification. Pattern Recognition 2024

Questions

See above.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

NA

Reviewer emFb2024-08-14

Thanks for the response and I will maintain my score.

Thanks for the response and I will maintain my score.

Authorsrebuttal2024-08-14

Thanks for the comment

We thank the reviewer very much for the feedback. Please do let us know if there is any effect we can futher make to address your concerns. Thank you!

Reviewer 4PW96/10 · confidence 2/52024-07-13

Summary

The work proposes an offline/batch active learning method for querying labels for nodes of a graph. The setting assumes access to noisy responses for the subset of nodes queried by the active learner. On the theoretical side, gains are shown over random selection, and bounds are shown on the generalization error of the proposed method. On the empirical side, the proposed algorithms is evaluated against random selection and other relevant baselines, and ablation studies are provided.

Strengths

- Label efficient learning in structured settings is a fundamental learning problem with numerous applications. - The proposed approach for graph active learning has both theoretical and empirical verified advantages over random selection. - The meta-approach of balancing informativeness and representativeness can be useful in other contexts. - Authors provide empirical evaluation which show superiority of proposed approach, and ablation studies which investigate the role of algorithmic components.

Weaknesses

- Missing connections to related theoretical literature on active learning on graphs. E.g. [1, 2] [1] Dasarathy, G., Nowak, R., & Zhu, X. (2015, June). S2: An efficient graph based active learning algorithm with application to nonparametric classification. In Conference on Learning Theory (pp. 503-522). PMLR. [2] Zhang, J., Katz-Samuels, J., & Nowak, R. (2022, June). Galaxy: Graph-based active learning at the extreme. In International Conference on Machine Learning (pp. 26223-26238). PMLR. - Presentation can be made clearer, see suggestions below.

Questions

- What is the significance of assumptions (1) and (2)? (lines 93-94) Specifically, is there a way to evaluate how strong assumption (1) is in practice? Also what is meant by "$\bf f$ is influenced by node covariates $\bf X$"? - Can you shed more light on the trade-off between informativeness and representativeness? Is there a standard way to quantify these? - "Robust" in the title is a bit confusing as it may be read as adversarial robustness, while you only seem to consider random noise. Consider using an alternative like "noise-tolerant". Furthermore, it would be good to state the type of noise (independent, bounded variance) considered early on, in Sections 1 and 2 (e.g. lines 58, 85). - Is there any approximation or other guarantee known for the quality of the proposed greedy approach (line 119) for selecting $\mathcal{S}$ to maximize the threshold frequency? - Presentation of the greedy algorithm can be improved say using an algobox. - Line 127: prescence or abscence - Could you elaborate on the size of $m$ needed in Theorem 2? Can this be empirically estimated? - What is the running time complexity for the proposed algorithm? - Is it possible to empirically verify the tightness/looseness of the theoretical bounds? - Repeated references [2] and [3]

Rating

6

Confidence

2

Soundness

3

Presentation

2

Contribution

3

Limitations

Authors should elaborate further on the limitations, e.g. assumptions needed for the theoretical results.

Reviewer 4PW92024-08-09

Thank you for the detailed response. I do not have further questions, and retain my current score.

Reviewer hH2h2024-08-10

Thank you for the detailed response. I raised my score to 5.

Authorsrebuttal2024-08-10

Thank you for raising the score!

We appreciate your recognition of the updated results and the improved score. If there are any other comments or questions, we would be pleased to discuss and clarify further!

Authorsrebuttal2024-08-12

Follow-up on our response to your feedback

We are very grateful for the time and effort you have devoted to reviewing our work, and we deeply appreciate your insightful, valuable, and encouraging comments. In response to your questions, we conducted additional experiments to compare the proposed method with SoTA offline graph-based active learning methods (RIM, GPT, SPA, FeatProp), and online methods (AGE, IGP). The numerical comparisons are conducted on networks with various topologies and response noise (Figure 4), different levels of homophily (Table 1), and larger scales (Table 2). In these experiments, the performance of the proposed method is either best or close to the best one. In addition, we follow your suggestion to add error bars when presenting the results. We discuss the optimality of the generalization bound in Theorem 3 and illustrate the tightness via simulation in Figure 1. We also provide a detailed discussion of the computational complexity of the proposed method, which is at $\mathcal{O}(pn^2)$ where $n$ and $p$ are the number of nodes and node features, respectively. We sincerely hope our response adequately addresses your concerns and will definitely incorporate these changes in the revised version. We look forward to your feedback with great anticipation.

Reviewer 7FJA2024-08-13

Response to rebuttal

I would like to thank the author's for their detailed answers and for the additional experiments. After carefully reading the responses I have decided to maintain my score.

Authorsrebuttal2024-08-13

Thank you for the feedback!

We appreciate your valuable feedback, which has greatly improved our paper. If you have any further comments or questions, we would be more than happy to discuss and provide clarification.

Area Chair uxNW2024-08-12

technical questions

Dear Authors and Reviewers, Based on the current ratings, the paper is at borderline. Since the discussion with authors will end soon, I hope the authors or reviewers could clarify some of my questions so that I can make informed decision. My questions will be mostly theoretical, given the subject of the work. 1. The paper claims using active learning techniques to save labels. What is the obtained label complexity and how does it improve upon passive learning? 2. The paper claims robustness. How is the noise model related to applications and what is the theoretical noise tolerance? 3. In Theorem 2, it is shown that E(Delta) - E(Delta_R) is lower bounded by some quantity. Can you showcase how the lower bound scales under simple assumptions, say the random graph G(n, p) where there are n nodes and each pair is connected with probability p independently? 4. When does the generalization error in Theorem 3 vanish with respect to the sample size, at what rate? Again, can you show concrete results under the G(n, p) model? Thanks, Area Chair

Authorsrebuttal2024-08-13

Response to AC's comments

We appreciate AC's insightful theoretical questions! The point-by-point responses are as follows: **Q1** Denote the eigenvalue distribution $F(\omega)= \frac{1}{n}\sum_{i=1}^n\mathbf{1}(\lambda_i \leq \omega) $ and the eigenvector space $\mathbf{U_d} = \text{Span}(U_1,\cdots,U_d)$, where ${(\lambda_i)}_{i=1}^{n}$, and ${(U_i)}$ are eigenvalues and $d$-leading eigenvectors of the normalized graph laplacian. Assuming the frequency of $\mathbf{f}$ is lower than $\omega_0$, we have $\mathbf{f}\in \mathbf{U}_d$ with $d = \lceil nF(\omega_0) \rceil$. Wlog, we assume $r_d = d$ in Theorem 3, i.e, $\mathbf{f}$ has component on each $U_i,i=1,\cdots,d$. With MSE $\frac{1}{n}\mathbf{E}\| \hat{\mathbf{f}} - \mathbf{f}\|_2^2$ fixed, the label complexity of our method is $O(d)$, while that of passive learning is $O(\tilde{d}\log \tilde{d})$ [1] with $\tilde{d} = \lceil nF\big(\frac{n}{\mathcal{B}}F^{-1}(\frac{d}{n})\big)\rceil$ where $\mathcal{B}<n$ is the number of query. Notice that $\tilde{d}>d$ due to the monotonicity of $F(\cdot)$ and $\tilde{d} \rightarrow d$ as $\mathcal{B}$ increases. Our method improves in two aspects: (1) the information selection identifies $\mathbf{f}$ via $O(d)$ queries, and passive learning uses $O(\tilde{d})$ queries, though this improvement decreases as $\mathcal{B}$ increases. (2) Our method improves by a log factor compared to passive learning via actively controlling the condition number of the covariate matrix, therefore the estimation variance. **Q2** In the real world, data labels aren’t always accurate/reliable. In this paper, we introduce label noise via a probabilistic model $Y_i = \mathbf{f}(i) + \epsilon_i$, where $Y_i$ and $\mathbf{f}(i)$ are the queried and true label on the $i$th node, and $(\epsilon_i)_{i=1}^n$ are independent errors satisfying $\mathbf{E}(\epsilon_i) = 0$ and $\mathbf{Var}(\epsilon_i) = \sigma^2$. Our noise model generalizes to different label types in applications. For continuous labels $Y_i$, one can set $\epsilon_i \sim \text{Normal}(0,\sigma^2)$. For binary label with $\mathbf{f}(i) = -1 \ \text{or} 1$, the typical noise model is $ Y_i = \mathbf{f}(i)$ with prob. $p>0.5$ and $ Y_i = - \mathbf{f}(i)$ with $1-p$. The binary noise model can be reparameterized via our noise model as $Y_i = \text{sign}(Y^*_i)$ where $Y^*_i = \mathbf{f}(i) + \epsilon_i$. The binary $\epsilon_i = 2\mathbf{f}(i)(1-p)$ with prob. $p$ and $\epsilon_i = -2\mathbf{f}(i)p$ with $1- p$, and $\mathbf{E}(\epsilon_i) = 0$ and $\mathbf{Var}(\epsilon_i) = 4p(1-p)$. Similarly, our noise model also accommodates multi-class labels via one-vs-other reparameterization. While most literature consider noise level $\sigma^2$ as constant, our method theoretically guarantees recovery as long as noise level $\sigma^2 = O(\mathcal{B}^\eta)$ with $\eta<1$. This result is useful when human labelling errors increase with workload. Additionally, our node selection strategy is invulnerable to label noise due to the offline nature, and our graph function space alleviates high-frequency noise in node features $\mathbf{X}$ due to network smoothing. **Q3** For a random graph $G(n,p)$, one has both lower and upper bounds on the node degree with high probability. Specifically, with probability greater than $1 - \exp\big(-np(c-1)^2\big)$, the maximum node degree $d_{max} \leq cnp$ with constant $c>1$. Given $np \rightarrow \infty$, the difference in information gain between the proposed method and random sampling can be simplified as $$ \mathbf{E}(\Delta) - \mathbf{E}(\Delta_R) > O(\frac{1}{ c^3 p^2 n^2}) -\frac{1}{n},$$ with probability larger than $1 - \exp\big(-np(c-1)^2\big)$. Therefore, the proposed method achieves larger information gain under the network sparsity regime $p = O(n^{-1/2}$), covering a broad range of real networks. **Q4** Following the same definition as in Q1, and assuming that the frequency of $\mathbf{f}$ is lower than $\omega$, the generalization error in Theorem 3 is $$ \frac{1}{n}\mathbf{E}\| \hat{\mathbf{f}} - \mathbf{f}\|_2^2 \leq O\Big(\frac{\text{rank}(\mathbf{U}_d^T\mathbf{X})}{\mathcal{B}}\Big),$$ where $d = nF(\omega)$, $\mathcal{B}$ is the number of queries, and $\mathbf{X}$ is node covariate matrix. With a fixed complexity of $\mathbf{f}$, the MSE decays at a rate of $\frac{1}{\mathcal{B}}$. The network topology affects the convergence rate through both the size of eigenvector space ($d$) and the alignment between two column spaces ($\mathbf{U}_d^T\mathbf{X}$). For illustration, we can obtain a looser bound as $O\Big(\frac{nF(\omega)}{\mathcal{B}}\Big)$. For random graph, the eigenvalue distribution of graph laplacian $F(\omega;n,p)$ can be analytically represented as the convolution of the standard normal distribution and the Wigner’s semi-circular distribution [2]. [1] Randomized algorithms for matrices and data. Found. and Trend. in Mach. Learn. (2011)\ [2] Spectral distributions of adjacency and Laplacian matrices of random graphs. Ann. of App. Prob. (2010)

Authorsrebuttal2024-08-12

Follow-up on our response to your feedback

We sincerely appreciate the time and effort you've put into reviewing our work, and we are truly thankful for your insightful, valuable, and encouraging feedback. In response to your constructive suggestions on numerical experiments, we have conducted additional experiments on real-world networks exhibiting different levels of homophily and heterophily (Table 1), as well as on synthetic networks with different topologies, including community structure, scale-free properties, and small-world properties (Figure 4). As a further supplement to Figure 4, we perform the proposed method with SoTA offline active learning methods on Erdős–Rényi random graph since many social and biological networks can be modeled via this model. The prediction MSE from different methods on **Erdős–Rényi graph** (n=100, p=0.25) with varying response noise are illustrated in the following table: | Method | $\sigma^2$=0.5 | $\sigma^2$=0.6 | $\sigma^2$=0.7 | $\sigma^2$=0.8 | $\sigma^2$=0.9 | $\sigma^2$=1.0 | |------------|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------| | D-Optimal | 0.48 &plusmn; 0.03 | 0.69 &plusmn; 0.04 | 0.93&plusmn;0.06 | 1.22 &plusmn; 0.08 | 1.54&plusmn; 0.1 | 1.9 &plusmn; 0.12 | | SPA | 0.61 &plusmn; 0.05 | 0.88 &plusmn; 0.08 | 1.2 &plusmn; 0.1 |1.57 &plusmn;0.13 |1.99 &plusmn; 0.17 | 2.46 &plusmn; 0.21 | | RIM | 0.49 &plusmn; 0.02 | 0.7 &plusmn; 0.02 |0.95 &plusmn; 0.03 |1.24 &plusmn;0.04 |1.57&plusmn; 0.06| 1.94 &plusmn; 0.07 | | GPT |0.58 &plusmn; 0.04 | 0.83 &plusmn; 0.06 |1.13 &plusmn; 0.08 |1.48 &plusmn;0.1 |1.87&plusmn; 0.14| 2.31 &plusmn; 0.17 | | Proposed | **0.45**&plusmn; 0.01|**0.65**&plusmn; 0.02 |**0.89**&plusmn; 0.03|**1.16**&plusmn; 0.03|**1.47**&plusmn; 0.04|**1.81**&plusmn; 0.05| Consistent with the results on other network topologies in figure 4, our method still outperforms other competing methods on Erdős–Rényi graph. Additionally, we discussed the computational overhead of the proposed method in details, and empirically demonstrated the scalability of our sampling algorithm on large-scale networks (Table 3). We sincerely hope our response adequately addresses your concerns and aids in the evaluation of our work. We look forward to further discussions with you.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC