Developing generalizable models that can effectively learn from limited data and with minimal reliance on human supervision is a significant objective within the machine learning community, particularly in the era of deep neural networks. Therefore, to achieve data-efficient learning, researchers typically explore approaches that can leverage more related or unlabeled data without necessitating additional manual labeling efforts, such as Semi-Supervised Learning (SSL), Transfer Learning (TL), and Data Augmentation (DA). SSL leverages unlabeled data in the training process, while TL enables the transfer of expertise from related data distributions. DA broadens the dataset by synthesizing new data from existing examples. However, the significance of additional knowledge contained within labels has been largely overlooked in research. In this paper, we propose a novel perspective on data efficiency that involves exploiting the semantic information contained in the labels of the available data. Specifically, we introduce a Language Semantic Graph (LSG) which is constructed from labels manifest as natural language descriptions. Upon this graph, an auxiliary graph neural network is trained to extract high-level semantic relations and then used to guide the training of the primary model, enabling more adequate utilization of label knowledge. Across image, video, and audio modalities, we utilize the LSG method in both TL and SSL scenarios and illustrate its versatility in significantly enhancing performance compared to other data-efficient learning approaches. Additionally, our in-depth analysis shows that the LSG method also expedites the training process.
Paper
Similar papers
Peer review
Summary
This paper proposed a general framework for exploiting semantic information within labels in classification tasks to improve the performance of deep nueral networks. The framework is both task-agnostic and model-agnostic, making it applicable to a wide range of classification tasks and modalities. The framework first concatenates label concepts/descriptions with prompting templates, and feed these templates into a frozen language models to generate embeddings. These embeddings are subsequently used to construct a semantic graph, which is trained using a graph convolutional network (GCN). When applying the semantic graph for guiding the training process, two regularization objective is added based on the semantic grpah in order to enhance the model with label semantics. The goal of the whole framework is similar to the prompt learning in NLP, while its capability of applying to tasks in other modality improve its novelty and contributions. In experiments, the authors show that their proposed framework successfully outperforms many other baselines based on transfer learning, semi-supervised learning and data augmentation.
Strengths
1. The paper itself is clearly written and presented, enabling readers to understand the proposed framework easily. 2. The proposed framework for incorporating semantic information of labels for tasks in various modalities is novel and widely applicable. 3. From the experiment section, the performance of the proposed framework significantly outperforms several baselines based on transfer learning, semi-supervised learning, and data augmentation.
Weaknesses
The proposed framework requires that the target task be a classification task and that each class has explicit and rich semantics. If a class does not have an explicit semantic meaning, the LSG may not work. This requirement limits the applicability of the LSG framework to many classification tasks.
Questions
My questions are mainly about the generalization ability of the proposed framework: 1. Can you provide more examples of prompt templates used for generating embeddings? 2. What is the performace of data on some large-scale datasets (i.e. when the data source is rich)? In Co-tuning's paper, they also conducted experiments COCO-70, which is considered a large-scale dataset with 1k samples per class. 3. What is the ituition behind adding sample-sample interaction in the augmented graph (i.e. the matrix M). Does it bring any benefit to LSG? 4. I noticed that all the tasks used in the experiments have at least 50 categories. I am curious if the LSG framework can be applied to binary classification tasks or tasks with a small number of categories (<10). In my opinion, it may be difficult to apply LSG to binary classification because the descriptions of the labels are generally two sides of one statement. For tasks with fewer than 10 categories, it may be challenging to train a GCN. It would be great if the authors could have some disscussion about these tasks.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors did not discuss limitations of the work. I do not see any potential negative societal impact.
Summary
This paper employ a language semantic graph to cature the relationship among different class, with the hope to alleviate the requirement of extensive training data and, in particular, human supervision. Generally, this paper first build the language semantic graph with the pre-trained language models. After that, the paper introduces two additional losses $L_align$ and $L_r$ that exploit the discarded label semantic information for complement.
Strengths
I have carefully read this papar some times, and I'm interested in injecting knowledge to the model for effective learning. The strengths can be describued as follow: 1. The idea is meaningful and interesting that employing prior knowledge for effective learning, and the proposed method is novel, at least I've not read some related paper. 2. The experiment results are effective compred with baseline models.
Weaknesses
1. The author not provide the code, and I'm very confued on the key section.
Questions
1. The final loss function in eq.(7) don't include the loss L_node, and what is the influce of L_node? 2. The language semantic graph is built with the embedding of label, so why you not using the embedding to capture the smantic relathionship among labels rather than using a discrete semantic graph? 3. I'm confused on m the size of the prompt set, what is the prompt set? 4.The section 3.2 are very confused for me, why the two proposed loss can help data effective learning. The author doesn't give a detail description for eq(5) and eq(6). And I'm very confused on why need the proposed two align loss, and the expective result of the proposed align loss.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
1 poor
Contribution
2 fair
Limitations
The developed method are depend on pre-trained language model, and sometimes the labels in a task maybe not have an effective embedding with the pretrained language models.
Summary
The paper addresses the importance of labels’ semantic meanings when training models. First, the framework constructs an LSG graph. Node features are text embeddings generated by language models, and the similarity matrix constructs edges. After that, a GCN is trained to aggregate node features of the LSG with the node classification loss. The GCN will be used for data-efficient learning afterward (regularization loss and alignment loss).
Strengths
* The motivation and proposed method is sound. * The details are well illustrated and are easy to understand. * Experiments cover a variety of tasks and well support the claims. Ablation studies also show the effectiveness of each component.
Weaknesses
* Since the LSG is the critical component of the proposed framework, the quality analysis of LSG is preferred. For example, a visualization of LSG showing how well it grasps the semantics of labels. * The notion should be clearer. The F, C and $\mathcal{C}$, $\mathcal{F}$ are confusing when first reading. I would suggest using subscripts to distinguish them.
Questions
* How to make the inference? Is GCN used in the inference or only the primary model is used? * I am curious about the stability of the training. The augmented graph contains a batch of inputs, possibly introducing interference between samples after GCN. Will this affect the training quality/stability? For example, will the training be affected much if we change the batch size or random seed? * Is the primary model feature encoder the same as the model used to generate node embeddings? I.e., are “LM” and “F” in the figure 1 the same?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Authors do not include limitations. Some limitations I can think of: 1. The current structure may have much more training burden than vanilla fine-tuning. 2. The method is only applicable for classification methods. 3. Authors could also try experiments on text tasks.
Summary
The paper introduces the Language Semantic Graph (LSG), a novel approach to data-efficient learning that leverages semantic information from labels. The LSG is used to train an auxiliary graph neural network, which then guides the primary model's training, enhancing the utilization of label knowledge. This method is applicable across various modalities, including image, video, and audio, and has shown significant performance enhancement in both Transfer Learning and Semi-Supervised Learning scenarios. Experiments were conducted on seven standard datasets covering images, videos, and audios, using several deep neural networks with different architectures and pretraining datasets. The results show that LSG significantly outperforms other methods, especially when labeled data is scarce. It also demonstrates promising potential in semi-supervised settings, achieving the best performance across all labeling rates and datasets. When applied to self-supervised pretrained models, LSG shows consistent gains. It also improves model performance on both in-distribution and out-of-distribution samples, indicating that label semantic relations help the model learn more robust features. In video and audio experiments, LSG consistently improves the fine-tuning accuracy across all tasks with limited labeled samples. It outperforms other methods, boosting accuracy significantly. For audio experiments, LSG achieves an average of 5.56% accuracy enhancement from the baseline, demonstrating its wide applicability across various modalities. "LSG consists of two parts: an auxiliary graph neural network that extracts knowledge from the semantic graph and two novel optimization objectives that transfer the knowledge to primary models." The authors demonstrate that LSG is applicable on image, video and audio models and brings significant performance gains to the model under Transfer Learning and Semi-Supervised Learning scenarios.
Strengths
* Innovative Approach: The paper introduces a novel method, the Language Semantic Graph (LSG), which leverages semantic information from labels to guide the training of machine learning models. This approach offers a new perspective on data-efficient learning that has not been extensively explored in previous research. * Versatility Across Modalities: The LSG method is applicable across various modalities, including image, video, and audio. This wide applicability demonstrates the robustness and flexibility of the proposed method. * Superior Performance: The LSG method shows significant enhancement in performance compared to other data-efficient learning approaches in both Transfer Learning and Semi-Supervised Learning scenarios. This is a strong indication of the effectiveness of the proposed method. * Robustness to Data Scarcity: The LSG method performs particularly well when labeled data is scarce, which is a common challenge in machine learning. This makes it a valuable tool for scenarios where data collection and labeling are costly or impractical. * Improved Out-of-Distribution Performance: The LSG method improves model performance on both in-distribution and out-of-distribution samples. This suggests that the label semantic relations help the model learn features that are more robust to distribution shift, enhancing the model's generalizability.
Weaknesses
The effectiveness of the LSG method relies heavily on the quality and semantic richness of the labels. In scenarios where labels are sparse, ambiguous, or not well-defined, the performance of the LSG method could be compromised.
Questions
*The LSG method relies heavily on the quality and semantic richness of the labels. How does the quality of the labels impact the performance of the LSG method? Could the LSG method be adapted to work effectively with less informative or ambiguous labels, and if so, how? * The paper primarily focuses on classification tasks. Could the LSG method be adapted or extended to other tasks, and if so, what modifications would be necessary?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
* The LSG method relies heavily on the quality and semantic richness of the labels. If the labels are not well-defined, sparse, or ambiguous, the performance of the LSG method could be compromised. This dependence on high-quality labels could limit the applicability of the method in certain scenarios.
Summary
The typical supervised training ignores the semantic information in the labels. This paper proposes to use the label semantic information during fine-tuning. 1) A label semantic graph is constructed by calculating the sentence embedding similarity of the label descriptions; 2) a GNN is trained on the semantic graph with a node classification objective; 3) the GNN is frozen and guides the training of a visual classifier with two additional loss terms. Specifically, at training time, for each data batch, we can create a graph with the images and labels as nodes; the GNN can be used to encode the graph and the data representations are treated as the initial node features; one loss terms classify the final node features while one loss term minimises the distance between the data representations and the final node features after GNN. The experiments are extensive, covering image, audio, and video classification. The approach can be used in a semi-supervised setting as well, where the pseudo-label data are only used in the additional loss terms and do not bias the classification head. The performance improvement is significant in many cases.
Strengths
1. The high-level motivation makes sense. It is desirable to utilise the semantic information within the labels during training. 2. The performance improvement is significant in many settings and the evaluation is comprehensive.
Weaknesses
1. While the idea of using label semantic information is good, the presented solution seems ad-hoc and not well motivated. Conceptually, the new information we introduce is by constructing the semantic graph A with a pre-trained text encoder; after this, it seems rather unclear what motivates the two-step approach with a GNN. In addition, I find the training objective of the GNN unclear (see Question). The role of multiple language prompts is also unclear. Is it because using multiple prompts builds a more informative graph? 2. One straightforward way to utilise the label semantic information is to directly use a language encoder to encode the labels and then fine-tune the language encoder along with the data encoder (this is different from the Language Head baseline where the language embeddings are fixed). According to ELEVATAR, this greatly outperforms vanilla fine-tuning. I wonder if the authors have considered this baseline, especially in the setting of Table 2: can we use CLIP image encoder + CLIP language encoder? Basically, if our pre-trained model already considers the semantic information of the labels (e.g., CLIP), does the approach still bring much improvements? ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models
Questions
For training the GNN, isn’t the node classification objective trivial? The objective is basically recovering the concept mentioned in the text prompt, which seems like an easy to solve task; the node feature at the first layer (K^0) should be enough to perfectly solve the task. This is different from the attribute classification task in typical GNN training, where there is not enough information to determine the node label without the adjacency matrix.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
No evident negative societal impact.
Thanks for the update, the authors addressed my concerns well. So I update the score from 6 to 7
Thanks for the update, the authors addressed my concerns well. So I update the score from 6 to 7
About the Anonymous Code Link
Esteemd Area Chair, It seems that the reviewer cannot get our anonymous code link which is sent previously. I wonder if the code link can be post here where the comment is visible to all the reviewers? Authors.
Acknowledgement of Code Link
Thanks for the message. -I responded to that reviewer with the code link (that message should be visible to you) -I also made a general post to the reviewers with the link you sent (that message should not be visible to you). So you don't need to do anything from your end. Thanks!
We have posted a new official comment and asked for permission to post code link
According to the official notification, we cannot post our code link here. (In notification) "All the texts you post (rebuttal, discussion and PDF) should not contain any links to external pages. If you were asked by the reviewers to provide code, please send an anonymized link to the AC in a separate comment." To address this issue, we have posted a new official comment visible to all reviewers and the AC at the top, and asked for AC's permission to post our code link there. We believe you will be informed once we get the permission.
Thanks for your further response, and can you describe the detail of Prototype align. and Contrastive align. Further, I think providing experiment results of these two methods on all the other datasets is help to further improve the quality of papers.
Code link is now available and results of the two variants on all datasets
Thanks for your valuable feedback that helps to improve our work, and we would like to assure you that all our results can be reproduced by the code we provided, if that is what you concerned. Here we make further clarification of the two ablation variants we used: - prototype alignment: each class prototype is calculated by the average of the label embeddings from this class. $c_k=\frac{1}{m}\sum_i{t_k^{(i)}}$, and the prototype alignment loss for each image feature $h$ with label $y$ is the cross-entropy loss $l_{pa} = -\log \frac{e^{h\cdot c_y}}{\sum_k e^{h\cdot c_k}}$ after applying $l_2$ normalization on both features and prototypes. - contrastive alignment: label embeddings are divided into $K$ subsets according to their embedded label, i.e., $[t_k^{(1)},...,t_k^{(m)}]$ for label $k$. For each image feature $h$ with label $y$, the contrastive alignment loss is a supervised contrastive loss [1] regarding embeddings in subset $y$ as positive pairs and other embeddings as negative pairs. [1] Supervised Contrastive Learning. We show the full results of these two variants on *Aircraft*, *StanfordCars*, *CUB200* for fine-tuning and *OfficeHome* for domain generalization. |Method|Air-15%|Air-30%|Air-50%|Air-100%| Car-15% | Car-30% | Car-50% | Car-100% | CUB-15% | CUB-30% | CUB-50% | CUB-100% | |--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:| |Prototype Alignment|51.1|68.8|76.5|84.6| 52.4 | 73.0 | 82.6 | 89.5 | 56.0 | 70.1 | 76.8 | 81.9 | |Contrastive Alignment| 54.2 | 70.7 | 78.3 | 85.1 | 54.3 | 73.8 | 83.2 | 90.1 | 56.7 | 69.9 | 77.1 | 82.1 | |**LSG**| **55.6** | **72.0** |**79.5**|**86.7**| **55.4** | **75.5** | **83.8** | **90.7** | **57.7** | **70.6** | **77.2** | **82.2** | | Source domain | Ar | Ar | Ar | Ar | Cl | Cl | Cl | Cl | Pr | Pr | Pr | Pr |Rw|Rw|Rw|Rw| Avg. ID| Avg. OOD| | --------------------- | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | | Target domain | **Ar** | Cl | Pr | Rw | **Cl** | Ar | Pr | Rw | **Pr** | Ar | Cl | Rw |**Rw**|Ar|Cl|Pr| Avg. ID| Avg. OOD| | Prototype Alignment | 85.0 | 56.9 | 71.3 | 78.1 | 86.2 | 69.5 | 72.9 | 75.9 | 94.3 | 61.9 | 50.4 | 80.6 |90.7|71.8|55.9|81.0|89.0|68.8| | Contrastive Alignment | 85.6 | 57.1 | 71.6 | 78.5 | **86.3** | 69.9 | 73.0 | 76.7 | 95.0 | 62.5 | 52.1 | 80.7 |90.8|72.2|56.2|81.4|89.4|69.3| | **LSG** | **85.8** | **57.7** | **74.0** | **79.9** | **86.3** | **71.7** | **75.4** | **77.8** | **95.1** | **65.8** | **54.7** | **82.1** |**91.2**|**73.8**|**58.0**|**82.3**|**89.6**|**71.1**| We observe that LSG consistently outperforms these two variants. These results will be added to the appendix of our paper.
Thanks for your detailed response. In my opinion, the aim of the proposed graph method is to capture the semantic relationship between labels and discriminative information from labels. I have raised my score from 3 to 4.
Code link
The authors sent me the anonymized code link. You can find it here: https://transfer.sh/Woy9yv4SYN/lsg_code.zip
Method Contribution Still Unclear
Dear authors, Thank you for the detailed rebuttal. For Q2, while I agree that L_align makes sense, my question was mainly targeted at the first-stage training. As you mentioned, when training only with L_node in the first stage, the training objective appears very simple: leveraging the initial label embeddings (K^0) is pretty enough to do node classification; I fail to see why "train GNN to obtain better label embeddings that are both semantic discriminative and structure-preserving". If the node classification can be perfectly performed with only node features at K^0 without information from the adjacency matrix, then I think the network is not encouraged to preserve/utilize the graph structure? I.e., the GCN may simply reduce to a network that acts like it directly predict the class from K^0. Is there some kind of loss/accuracy we could report for the first-stage training? I would image that the model can achieve 100% accuracy easily. In other words, I do not see the benefit of using a GCN and L_align in the second stage (compared to the language head baseline), as I suspect that the GCN may reduce to a very simple node-feature-readout-network since its first-stage objective is easy.
Experiments that proves our GCN is more than a readout network and is necessary
Thanks for your insightful feedback. First of all, we would like to demonstrate from three aspects that our GCN classifies nodes **according to both the initial node features and the graph topology**. - We conduct a new ablation study as follows. - The GCN is firstly trained using standard procedure in our first stage. Then we change the adjacency matrix $A$ of the original graph to an identity matrix $I$ (in other words, we erase all the topology information and leave the node features unchange). We use the trained GCN to predict the nodes on the new graph and denote this variant as *GCN w/ $I$*. **We observe significant accuracy drops compared to the prediction accurcies on the original graph** (*GCN w/ $A$*). This observation shows that the trained GCN actually depends on the adjacency matrix $A$, rather than solely using the initial node features. - For a better comparison, we train a linear classifier on the node features, which can be viewed as the node-feature-readout-network you mentioned. The results in table below show that the readout network performs better than our GCN when only node features are provided, yet underperforms the GCN when the topology information is included. Therefore, we conclude that the GCN trained by $\mathcal L_{node}$ is more than a readout network. - | Variant| *Aircraft* | *StanfordCars* | *CUB200* | | --| :--: | :--: | :--: | | GCN w/ $A$ | 100 | 99.7 | 100 | | GCN w/ $I$ | 89.7 | 90.9 | 93.5 | |linear probe | 96.4 | 98.8 | 100 | - We refer back to the results in Fig. 2(d), which are reported here below (the GCN accuracy is tested on a validation graph based on another node feature set). Please note that our designed graph includes a few edges connecting nodes from different classes, where its amount is controlled by cross-label edge ratio. Therefore, different ratio corresponds to different graph topology on top of a same node feature set. We observe that the accuracy of GCN is influenced by the edge ratio, namely the topology information. Moreover, GCN trained from different topology structure has different impacts on the primary model accuracy in the 2nd stage, all proving that GCN learns the structure information. - | Cross-label Edge Ratio|0.1%|0.2%|0.3%|2.0%| | -- | :--: | :--: | :--: | :--: | | GCN acc. (validation graph) | 99.8 | 99.6 | 99.1 | 75.5 | | primary model acc.| 70.7 | 71.2 | 71.3 | 68.7 | - The T-SNE visualization also supports our claim. In the original node feature space, there exist a few "prompt clusters" which is hard to discriminate by class. Specifically, each sentence we sent into PLM consists of a prompt template and a label. In some cases, the prompt template dominents in the text embedding and overshaows the class discriminative information. However, the GCN refined embedding space no longer has such prompt cluster, **demonstraing the effect of graph edges that connect these nodes to others sharing same label**. Table below compares the Calinski-Harabasz Index between the two embedding space, where higher score means better clustering, and shows GCN learns discriminative info. - | Embedding| *OfficeHome* | *Aircraft* | *StanfordCars* | *CUB200* | |--| :--:|:--: | :--: | :--: | | Initial Label Embedding |85.5|86.5|137.2|157.8 | | GCN refined Embedding | **1395.2** |**1364.1** | **1481.7** | **916.3** | Next, we show the loss and accuracy curve of GCN on Aircraft below. GCN finally reaches 100% accuracy on training graph, yet requires a period time of training. |Iter |1| 10| 50 |100 | 200 | 500| 1000 | 2000 | 5000 | |--| --|--| --| --|--|--|--|--|--| | loss |4.61| 4.55 | 4.08 | 3.95 | 3.86 | 3.78 | 3.72 | 3.66 | 3.63 | | acc. |0.0| 2.0 | 60.2 | 85.7 | 94.1 | 97.0 | 99.0 | 100 | 100 | Finally, we show results of prototype alignment on more datasets. According to your description, we think this prototype alignment is what you mean by "language head baseline", since that the prototypes calculated by averaging node features is a readout network and it **supervises the projected image feature** $h$, replacing our GCN. In contrast, by language head we refer to directly replacing the random initialized classifier by language initialization, therefore is not suitable to ablate GCN. We see from results that a **readout network (prototype alignment) is less beneficial than our GCN**. Moreover, **$\mathcal L_r$ cannot be applied without GCN in the 2nd stage**. Therefore, we argue that $\mathcal L_{node}$ trained GCN is necessary in the 2nd stage. (OH stands for OfficeHome dataset and the detailed results can be found in response to reviewer 3r7Y.) |Method| Car-15% | Car-30% | Car-50% | Car-100% | CUB-15% | CUB-30% | CUB-50% | CUB-100% | OH OOD Avg. | |--|--|--|--|--|--|--|--|--|--| |Prototype Alignment| 52.4 | 73.0 | 82.6 | 89.5 | 56.0 | 70.1 | 76.8 | 81.9 | 68.9 | |LSG| **55.4** | **75.5** | **83.8** | **90.7** | **57.7** | **70.6** | **77.2** | **82.2** | **71.1** |
Looking Forward to Seeing Your New Feedback.
Dear Reviewer ZDzt, We are wondering if our newest response resolves your main concern about what the GCN has learned. Specifically, we have provided in our new response strong evidence showing that our GCN indeed requires edge information to make correct predictions. We have also provided comprehensive experiment results on all vision datasets to verify that the trained GCN is more beneficial than a node-feature-readout-network. We are more than delightful to discuss any further questions you have about this key issue. Thanks again for your valuable time and feedback. Sincerely, Authors
Dear Authors, Thank you for your detailed response and additional results. Indeed, the experiments support that LSG is not "simply" a node-feature-readout network, but I am concerned that it is somehow close to it. The Prototype Alignment achieves quite close performance to LSG (from my observation); thus I am concerned about the benefit of LSG. In addition, the GCN first-stage training converges in 5000 steps and as the authors suggest, "the prompt template dominents in the text embedding and overshadows the class discriminative information"; thus I am worried that the GCN training might be just removing such noise and recovering the information loss, which makes the contribution a bit thin. Thus, I would like to keep my score but I appreciate the rebuttal of the authors.
Emphasis on two key points
We thank you for your valuable time and feedback. We would like to emphasize two points regarding the key issue. Stage one is in preparation for the second stage and is designed to be lightweight. The simple yet effective loss $\mathcal L_{node}$ is enough to train a GCN beneficial to the second stage. Please note that our **methodology innovation** of guiding the model learning with semantic graph is largely embodied in the second training stage, which we call ''the primary training stage'' in our paper. LSG is significant in performance and widely applicable to tasks of different modalities. To our best knowledge, the 'Prototype Alignment (PA)' variant that appeared in the ablation study during rebuttal has not been explored either, and thus both PA and LSG validate our **conceptual innovation** of learning label semantic knowledge for better representation. And it is shown that LSG is a more sophisticated method than PA.
Thanks for the detailed response! I will raise my confidence score. One remaining question (which also mentioned by some other reviewers) is whether GCN is necessary. Since the training time in stage 1 is so short, the training objective may be quite easy. I am unsure whether GCN uses the embedding as a "shortcut" to do classifications and ignore edges.
Experiments that proves our GCN does not ignore edges and is necessary
Thanks for your valuable feedback. To answer your question, similar to our response to reviewer ZDzt, we would like to demonstrate from three aspects that our GCN classifies nodes **according to both the initial node features and the graph topology** (it does not ignore edges). - We conduct a new ablation study as follows. - The GCN is firstly trained using standard procedure in our first stage. Then we change the adjacency matrix $A$ of the original graph to an identity matrix $I$ (in other words, we erase all the topology information and leave the node features unchange). We use the trained GCN to predict the nodes on the new graph and denote this variant as *GCN w/ $I$*. **We observe significant accuracy drops compared to the prediction accurcies on the original graph** (*GCN w/ $A$*). This observation shows that the trained GCN actually depends on the adjacency matrix $A$, rather than solely using the initial node features. - For a better comparison, we train a linear classifier on the node features, which can be equivalent to a GCN "using the shortcut" as you mentioned. The results in table below show that the linear classifier performs better than our GCN when only node features are provided, yet underperforms the GCN when the topology information is included. Therefore, we conclude that the GCN trained by $\mathcal L_{node}$ does not simply using the shortcut. - | Variant| *Aircraft* | *StanfordCars* | *CUB200* | | --| :--: | :--: | :--: | | GCN w/ $A$ | 100 | 99.7 | 100 | | GCN w/ $I$ | 89.7 | 90.9 | 93.5 | |linear probe | 96.4 | 98.8 | 100 | - We refer back to the results in Fig. 2(d), which are reported here below (the GCN accuracy is tested on a validation graph based on another node feature set). Please note that our designed graph includes a few edges connecting nodes from different classes, where its amount is controlled by cross-label edge ratio. Therefore, different ratio corresponds to different graph topology on top of a same node feature set. We observe that the accuracy of GCN is influenced by the edge ratio, namely the topology information. Moreover, GCN trained from different topology structure has different impacts on the primary model accuracy in the 2nd stage, all proving that GCN pays attention to the edge information. - |Cross-label Edge Ratio|0.1%|0.2%|0.3%|2.0%| | -- | :--: | :--: | :--: | :--: | | GCN acc. (validation graph) | 99.8 | 99.6 | 99.1 | 75.5 | | primary model acc.| 70.7 | 71.2 | 71.3 | 68.7 | - The T-SNE visualization also supports our claim. In the original node feature space, there exist a few "prompt clusters" which is hard to discriminate by class. Specifically, each sentence we sent into PLM consists of a prompt template and a label. In some cases, the prompt template dominents in the text embedding and overshaows the class discriminative information. However, the GCN refined embedding space no longer has such prompt cluster, **demonstraing the effect of graph edges that connect these nodes to others sharing same label**. Next, we show that the trained GCN is necessary for the 2nd stage via showing full results of the prototype alignment variant. The prototype alignment refers to aligning image features with prototypes of original label embeddings, therefore it only leverages the embedding information and ignores the graph topology. We see from results that **prototype alignment is less beneficial than our GCN on all datasets**. |Method| Car-15% | Car-30% | Car-50% | Car-100% | CUB-15% | CUB-30% | CUB-50% | CUB-100% | |--|--|--|--|--|--|--|--|--| |Prototype Alignment| 52.4 | 73.0| 82.6| 89.5 | 56.0| 70.1| 76.8| 81.9 | |LSG| **55.4** | **75.5** | **83.8** | **90.7** | **57.7** | **70.6** | **77.2** | **82.2** | | Source domain|Ar|Ar|Ar|Ar|Cl|Cl|Cl|Cl|Pr|Pr|Pr|Pr|Rw|Rw|Rw|Rw| Avg. ID| Avg. OOD| |--|:--:|:--:| :--: | :--: | :--: | :--: | :---: | :--: | :--: | :--: | :---: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | | Target domain | **Ar**|Cl| Pr| Rw | **Cl** | Ar | Pr | Rw | **Pr** | Ar | Cl | Rw |**Rw**|Ar|Cl|Pr| Avg. ID| Avg. OOD| | Prototype Alignment | 85.0 | 56.9 | 71.3 | 78.1 | 86.2 | 69.5 | 72.9 | 75.9 | 94.3 | 61.9 | 50.4 | 80.6 |90.7|71.8|55.9|81.0|89.0|68.8| | Contrastive Alignment | 85.6 | 57.1 | 71.6 | 78.5 | **86.3** | 69.9 | 73.0 | 76.7 | 95.0 | 62.5 | 52.1 | 80.7 |90.8|72.2|56.2|81.4|89.4|69.3| | **LSG**| **85.8** | **57.7** | **74.0** | **79.9** | **86.3** | **71.7** | **75.4** | **77.8** | **95.1** | **65.8** | **54.7** | **82.1** |**91.2**|**73.8**|**58.0**|**82.3**|**89.6**|**71.1**| Finally, **$\mathcal L_r$ cannot be applied without GCN in the 2nd stage**, whereas its effectiveness is demonstrated in the ablation study in table 6. In conclusion, we argue that $\mathcal L_{node}$ trained GCN is necessary for our proposed method.
Thanks for your response. The additional results make me more confident about the paper. I will keep my score unchanged since I already lean to accept.
Thank you for the response.
Thank you for the detailed rebuttal, It makes me feel clearer toward this paper. After reviewing all other reviewers opinions and the corresponding response from the authors, I decide to remain my current score.
Thanks for the rebuttal (here and below for each reviewer). We will take it into account during the discussion period.
Decision
Accept (poster)