Deep supervised models possess significant capability to assimilate extensive training data, thereby presenting an opportunity to enhance model performance through training on multiple datasets. However, conflicts arising from different label spaces among datasets may adversely affect model performance. In this paper, we propose a novel approach to automatically construct a unified label space across multiple datasets using graph neural networks. This enables semantic segmentation models to be trained simultaneously on multiple datasets, resulting in performance improvements. Unlike existing methods, our approach facilitates seamless training without the need for additional manual reannotation or taxonomy reconciliation. This significantly enhances the efficiency and effectiveness of multi-dataset segmentation model training. The results demonstrate that our method significantly outperforms other multi-dataset training methods when trained on seven datasets simultaneously, and achieves state-of-the-art performance on the WildDash 2 benchmark.
Paper
Similar papers
Peer review
Summary
The paper proposes a multi-dataset training approach that works on top of a unified output taxonomy mapped onto individual dataset-specific taxonomies. First, a multi-head model is trained. Then, a unified taxonomy is automatically constructed by merging semantically identical classes, determined based on segmentation performance on individual classes when a connection is made. After that, GNNs are used to further refine the mapping from the individual dataset-specific classes to the unified taxonomy. The input nodes of the GNN are dataset-specific labels and unified classes. Dataset-specific labels are first enriched with a textual description provided by ChatGPT, which is then encoded with LLaMA-2 and combined with a trainable dataset embedding to create node input features. The addition of the trainable dataset embedding enables the model to learn the different meanings of classes with the same name in different datasets. Finally, the training process alternates between fine-tuning the segmentation network and refining the mappings with the GNN.
Strengths
1. Well-written and clear. 2. Achieved state-of-the-art results on the WildDash 2 benchmark, demonstrating improvements over previous work. 3. Introduces a novel approach for label unification.
Weaknesses
1. The paper primarily focuses on individual benchmark performance rather than the quality of the recovered taxonomy. In practical scenarios, it's crucial to reason within a unified label space. Mistakes such as equating MPL:tunnel to ADE:fireplace could mean the difference between advancing or halting in a real-world scenario. 2. The model achieving state-of-the-art performance on WildDash is trained exclusively on driving datasets, unlike other methods submitted that were trained on the full RVC collection (including ADE20K and COCO), potentially impacting performance in road driving contexts. 3. There is significant variation in training design choices in Table 2, making it challenging to isolate the impact of each individual choice (e.g., backbone, selection of dataset-specific classes, etc.). 4. The initialization of the adjacency matrix appears to be a crucial step for model performance, yet it is not discussed in the main paper.
Questions
1. What is the size of the created taxonomy in different tables (e.g. in Table 5)? Furthermore, Table 5 could also include the performance after initializing the adjacency matrix, without additional GNN training. 2. How does the quality of the created taxonomy compare to the manual taxonomy provided by [5]? Are all relationships recovered? For example, IDD:tunnel is not connected to MPL:tunnel in Figure 5. 3. How are unseen datasets mapped to the unified taxonomy in Table 3? Is it done manually or automatically? 4. Is there any advantage to using the formulation in Eq. 6 as opposed to NLL+ [4]? 5. How much does the trained taxonomy differ from the initialized taxonomy? Do all unified classes "survive" after training, or do some become obsolete? 6. What exactly is the zero-shot model on WildDash 2 in Table 4? 7. How long does it take to initialize the adjacency matrix? What exactly is the difference compared to [52]? Are there similarities to [5], such as pairwise merging?
Rating
4
Confidence
4
Soundness
2
Presentation
3
Contribution
3
Limitations
1. Focus on individual benchmark performance, rather than the process and the quality of the produced taxonomy.
We are pleased to inform you that we have obtained the complete experimental results regarding the initialization, answering W4 & Q1. Our experiments were conducted on three datasets (Cityscapes, Sunrgbd and CamVid) using two 32GB V100 GPUs. After completing 50,000 multi-SegHead training iterations, we performed 50,000 segmentation network training iterations for each group using the same multi-head model parameters. For the experiments involving GNN, an additional 30,000 GNN training iterations were included (without updating the segmentation network parameters). The results indicate that our GNN training shows a performance improvement compared to the results obtained using Algorithm 2 for initializing the adjacency matrix without GNN training. Algorithm 2 provides a good starting point for GNN training and contributes to the overall performance enhancement. | Method|\|L\||CS|SUN|CV|Mean| |--|--|--|--|--|--| |Randomized adjacency matrix with GNN training|54| 78.0|43.1|82.4|67.8| |Intialized adjacency matrix without GNN training|56|78.0| 42.0|82.6|67.5| |Ours (Intialized adjacency matrix with GNN training)| 54|78.4|43.3|82.6|68.1|
Dear Reviewer, Thank you for your valuable feedback. We agree that evaluating a universal taxonomy instead of merely benchmark performance of downstream tasks is a crucial aspect of this research, particularly in terms of relation discovery quality. However, we currently face a challenge due to the lack of a dedicated benchmark specifically designed for assessing taxonomy quality. While the manually curated taxonomy provided by [5] serves as a helpful reference, it also incorporates expert knowledge that might not be well aligned with the visual features present in the dataset images. Furthermore, there is currently an absence of well-established metrics specifically for this type of evaluation. Given these current limitations, this work has primarily focused on evaluating model performance across different datasets. Moving forward, we are committed to exploring ways to better evaluate relation discovery quality and to address the concerns you have raised. We appreciate your insights and believe they are instrumental in guiding our future research endeavors. Regarding the WD2 performance, our zero-shot model achieved SOTA results compared to other zero-shot models on the leaderboard, which we believe highlights the contributions of our approach. It’s important to note that the Vistas dataset lacks annotations for the pickup, van, and autorickshaw categories present in WD2. Additionally, we did not utilize the relabeled data provided by WD2 for these categories in the MVD, Cityscapes, and IDD datasets during training. Therefore, given the dataset bias, it is expected that the zero-shot model would perform lower on WD2 compared to models specifically trained on the WD2 dataset. Thanks!
Summary
This paper introduces a method using GNN to automatically construct a unified label space across multiple datasets, addressing the issue of label space conflicts in multi-dataset semantic segmentation. The method eliminates the need for manual re-annotation or iterative training, significantly enhancing the efficiency and effectiveness of model training. Experiments show that this method has certain effectiveness.
Strengths
1. The motivation behind this approach is clear, and automatically constructing a unified label space across multiple datasets in multi-dataset semantic segmentation is meaningful. 2. The authors use numerous illustrations and provide a detailed description of the experimental parameters, making it easy to follow and reproduce.
Weaknesses
1. This paper proposes a new method, but in my view, it is merely a collection of tricks. For example, $d_i$ is introduced in Equation (1), but the authors neither explain its role through ablation experiments nor from a methodological perspective. It is recommended that the authors provide a more detailed explanation of the motivation for each component of the method. 2. Although the motivation for this work is multi-dataset semantic segmentation, the experimental comparison methods do not include the latest methods from the multi-dataset semantic segmentation community, such as [1,2], etc. 3. This paper only uses HRNet-W48 as the backbone. The authors are encouraged to further explore the scalability of the proposed method on transformer-based networks. 4. From Table 5, it appears that the improvement in results due to label description is not significant. However, using GPT to complete this step inevitably incurs a substantial consumption of time and resources, which raises questions about whether it is worth it. Additionally, the authors should discuss the impact of the proposed method on time and space costs. [1] Gu X, Cui Y, Huang J, et al. Dataseg: Taming a universal multi-dataset multi-task segmentation model. NeurIPS 2023. [2] Wang L, Li D, Liu H, et al. Cross-dataset collaborative learning for semantic segmentation in autonomous driving. AAAI 2022.
Questions
Please see Weaknesses.
Rating
4
Confidence
3
Soundness
3
Presentation
3
Contribution
2
Limitations
Limitations are discussed in the Conclusion.
Dear Reviewer, Thank you for your thoughtful feedback and for acknowledging our efforts in the rebuttal. To the best of our knowledge, the SOTA method we refer to is the one presented in IJCV 2024 [5], where our approach demonstrated significant performance improvements in both the 7ds and WD2 ( 51.3 vs. 56.4 on 7ds and 46.9 vs. 50.2 on WD2). Could you please clarify if the SOTA methods you are referring to are those mentioned in your initial review? If so, we have already emphasized that these methods are not directly comparable within our *dataset-agnostic* setting, as they only provide *dataset-aware* prediction and are unable to provide predictions in a unified label space. If you are referring to other methods, we would appreciate it if you could provide some examples for our reference. We would also like to emphasize that we did not intend to avoid comparisons. Rather, it was challenging to produce the comparison results within the limited time frame. We plan to include additional experimental data in the revised paper, such as results on WD2 using the full RVC collection for training and on 7ds with the initialized adjacency matrix without GNN training. As for your suggestion to include Transformer-based methods, we greatly appreciate this valuable input. We will certainly consider exploring and discussing these approaches in a future version of the paper. Thanks! We look forward to your any further feedback.
What I mean by SOTA methods is that they are transformer-based approaches. The proposed method brings considerable improvements with HRNet or SNp as the backbone. However, my main concern is that HRNet and SNp are simple backbones proposed years ago. Therefore, I remain skeptical whether the improvements are still reasonable when the backbone is replaced by more complex and effective ones such as a Transformer. The authors suggested replacing HRNet with other SOTA backbones such as Transfomrer, to validate the generalization and effectiveness of their proposed method.
Dear Reviewer, Thank you for highlighting this important consideration. We fully agree that exploring Transformer-based methods could further validate the generalization and effectiveness of our proposed approach. However, we have currently chosen to use CNN models for the following two reasons, with plans to explore Transformer-based models in future work: * For consistency and fairness in comparison with other multi-dataset semantic segmentation training methods, we have utilized CNN models, as these are also employed by the methods we compared against. The consistency of using CNNs allowed us to establish a solid baseline and ensure that our comparisons were on equal footing. * As you mentioned, Transformer-based methods have demonstrated significant improvements over traditional CNN backbones like HRNet and SNp. However, we would like to highlight that, even with a CNN backbone, our current method has achieved state-of-the-art results on WD2 benchmark, surpassing other approaches on the leaderboard, including those that employ Transformer-based methods [26][44]. We anticipate that incorporating a Transformer backbone could likely yield even greater improvements. We are committed to further enhancing our approach by exploring Transformer-based models in future research, and we appreciate your valuable feedback on this matter.
I acknowledge the authors' efforts. The authors expect that using Transformer models will lead to greater improvements. However, based on my experience, some methods that are quite effective with CNNs see only limited improvements or even a decline in performance when switched to Transformers. Therefore, I still believe that papers submitted to top machine learning conferences should validate the effectiveness of their methods with Transformers. That said, I am open to discussions with other reviewers and AC. If everyone believes this concern is insignificant, I am inclined to accept the paper. Accordingly, I will lower my confidence rating to 3.
Summary
This paper introduces a method to automatically match and unify different label spaces for semantic segmentation. This allows to train a single model on multiple, differently annotated datasets. The authors can show that this can yield benefits in overall model performance, also compared to other multi-label approaches. The method is the currently best result on the public WildDash2 benchmark.
Strengths
- This method is the current SOTA on the public learderboard of WildDash2, which means the proposed method holds against a very rigorous testing setup. - With the current knowledge that data scale is one of the key ingredients in well-generalizing models, progress on multi-dataset-training has high potential for impact. For example, the current leader of all ScanNet 3D segmentation leaderboards is also a method based on multi-dataset-training. - The presentation is very clear and structured for the most parts of the paper.
Weaknesses
- It is currently unclear to me based on what information the graph connectivity is learned. How the method is described, the name of the annotated label is the only information that is put into the nodes, and all matching is based on that plus a hallucinated description from GPT. However, the authors mention in line 199-200 that the method would leverage visual similarity between annotations. It is not clear to me how this is facilitated. - All results are based on an architecture from 2019. It is lucky that WildDash2 has no stronger competitors in the leaderboard, but it remains unclear whether the method would yield the same improvements for a more competitive base architecture (e.g., mIoU of 77% on Cityscapes is a very bad baseline performance in Table 2. The achieved “improved” 80.7% are on-par with DeepLabv3, which was SOTA in 2018). - The method underperforms on more general indoor/outdoor datasets. Both on ADE and COCO, MSeg is better (even when predicting less classes, and in contrast to the bolded numbers, which are somehow ignoring MSeg) - I don’t think it is well discussed how hallucinating label descriptions through ChatGPT can introduce mistakes into the methodology. I don’t know about all of the datasets, but for example Cityscapes provides their own descriptions of the labels, which are the descriptions that also label workers use to label the data (see https://www.cityscapes-dataset.com/dataset-overview/#labeling-policy ). LLMs that are simply prompted with the label name can easily make up wrong descriptions that are not calibrated to the label policy of the annotation.
Questions
see above. The points that can hopefully easily be clarified by the authors are the question how visual information is used in the matching, what to do about GPT outputting wrong definitions, and why the method underperforms on more general datasets with larger label spaces.
Rating
7
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
- The broader impact does not at all discuss potential societal impact of the work. While it is true that it reduces the re-labeling effort, the method still requires segmentation labels, which are the much higher cost compared to re-labeling existing segmentation labels. What should be rather discussed are the societal impact of deploying a method with an automatically generated label space, and what the implications of this are to safety and certification when trying to deploy this method e.g. to automated driving. - The discussed limitations are OK. It would be interesting of the authors could further comment on ways how to potentially get around the scaling problem of current requiring all datasets loaded on one node, and every dataset beeing sampled for every batch.
Dear Reviewer, Thank you for your thoughtful feedback. We acknowledge the limitation of our current approach, where label descriptions generated by GPT are used to construct node features purely based on textual information. In future work, we plan to address this limitation by incorporating official label descriptions and introducing mask image features corresponding to the label categories to enhance the node features. Regarding your question about MSeg, we would like to clarify that MSeg performs label merging or omission for certain datasets (e.g., ADE, COCO), while for others (e.g., CS, SUN), it uses the same evaluation categories as we do. For these datasets where a different evaluation protocol was used, while direct comparisons are not possible, this does explain the higher mIoU values observed in their results. However, we believe that on datasets where the same evaluation protocol is used, a fair comparison can be made. Thank you again for your detailed feedback and for maintaining your positive view of our work.
Summary
The paper presents an approach to automatically train a semantic segmentation network using multiple datasets with individually different class label policies. A unified label space emerges during training automatically without direct manual label mapping definitions by using a graph neural network (GNN) which is guided by textual descriptions of each label in unison with a multi-head segmentation head. The training alternates between GNN and segmentation network training to jointly optimize the label mapping for multiple dataset as well as the actual image segmentation task itself. Multiple experiments evaluate the approach to unify 7 mixed datasets (CS, MPL, SUN, BDD, IDD, ADE, COCO) as well as 5 road-scene datasets (CS, IDD, BDD, MPL, WD2). Performance is compared by both training single heads and using the Multi-Seg. Head. as well as benchmarking on WD2. Additional ablation experiments showcases the value of textual label descriptions for the unification process. In summary this approach allows for a joint training with multiple mixed segmentation datasets without additional manual intervention.
Strengths
*) Generation of segmentation label space mappings to automatically create a unified label space. *) Great overall performance for each tested dataset *) Record benchmark results for Wilddash2 *) Many experiments to showcase the approach on multiple driving-scene datasets as well as some more generic datasets. *) Supplemental contains the full training code to recreate the experiments
Weaknesses
*) Some details are omitted (see Questions) making it hard to judge the extent/validity of some claims (e.g. robustness/generalization). *) Multi-head training and evaluations favor the accumulation of per-dataset bias. The cross-validations are a good start but there is no evaluation of "new"/out-of-distribution labels from one dataset within the other. This amplifies dataset-specific training (each head is optimized to work for one specific dataset); thus training to "solve" datasets rather than the task at hand. L281 (tunnel == fireplace) is a great observation but also shows the weakness of the approach. *) "Splitting" of source labels is not supported; the multi-heads are trained in isolation so if a label in one dataset requires splitting in another dataset, the respective heads are isolated (one has the split, the other doesn't). During inference of an unseen image this can result in only a single merged representation instead of the more detailed split. This effect is underrepresented during evaluation as dataset bias further separates the heads to work in favor of the overlapping test data. *) Both WD1 and WD2 are used. WD2 supersedes WD1 in every way /(it also includes all WD1 frames with the extended label policy) and crucially also uses a unified label policy combining IDD, MPL, CS & WD1 (80 label cat.; the benchmark uses the reduced WD2_eval 25 cat policy). So contrary to L223, WD2 does contain the labels "lane marking", "crosswalk" and "manhole". WD2 has a different publication associated with it which is unmentioned: CVPR22 "Unifying Panoptic Segmentation for Autonomous Driving". The paper/experiments would be easier to understand with only WD2. *) For some figures/descriptions in the paper it is unclear which dataset combinations was used during training. Introduction of fixed names (e.g. 5ds vs. 7ds for 5/7 datasets; or explicitly mention "domain-general"/"domain-specific" wording) could help.
Questions
*) The creation of textual descriptions using ChatGPT is not described at all; which version of ChatGPT was used? How were per-dataset label spaces described as textual input? Were images/masks/masked images used with a multi-modal version of ChatGPT (e.g. GPT4V; GPT4o; ...). Using masks together with images as inputs for ChatGPT is non-trivial. L110 contains no details; wording in Figure 2 suggests a multi-modal model as "image of" is part of the dataset label text or simply the "blind" pure-text completion of prompts in the style "An image of <label> from the dataset <dataset>....". Please clarify; the text label descriptions are an important aspect of the paper's novelty! *) Where is the final resulting unified label mapping? Figure 5 shows a small part but the full joined label set would give a good indication of generalization vs. specialization. *) Supplem. B. (target number of unified labels) is a crucial component as it directly steers the separation/merging of similar concepts (e.g. seperate "Cityscapes_road"/"MPL_road" labels vs. a unified "road" label). How is the hyperparameter Lambda|L| chosen and what number of labels did the experiments end up with? *) What pre-training was used for the initialization of the multi-heads? *) WD2 negative examples are a defined subset within the benchmarking dataset. However, the official rules of the benchmark state, that both regular/"positive" and negative frames/subsets have to be treated equally. The comparability within the benchmark is only held upright, as long as this rule is observed. L246/L247 can be interpreted otherwise: "We map non-evaluated classes to a void label"; is this applied to both positive and negative frames alike?
Rating
5
Confidence
5
Soundness
2
Presentation
3
Contribution
3
Limitations
*) Limitations relating to dataset bias could be mentioned.
Summary
This paper presents a novel approach for automated label unification across multiple datasets in semantic segmentation using Graph Neural Networks (GNNs). The proposed method aims to create a unified label space that allows semantic segmentation models to be trained on multiple datasets simultaneously without the need for manual re-annotation or taxonomy reconciliation. The results demonstrate significant performance improvements over existing methods, achieving state-of-the-art performance on the WildDash 2 benchmark.
Strengths
+ Novel Approach: The use of GNNs to automatically construct a unified label space is innovative and addresses the challenge of conflicting label spaces in multi-dataset training. + Performance: The method shows impressive performance improvements, achieving state-of-the-art results on the WildDash 2 benchmark and outperforming other multi-dataset training methods. + Efficiency: The approach eliminates the need for manual reannotation and iterative training procedures, significantly enhancing the efficiency of multi-dataset segmentation model training. + Robustness: The method demonstrates good generalization to unseen datasets, indicating its robustness and applicability across various scenarios.
Weaknesses
This indeed seems a interesting problem. I have several questions: 1. How do you select the unified label size N? Is it adaptive to different dataset labels, or preset hyper-parameters? 2. Can you show more results of the bipartite graph generated the GNN? Like what labels is hard to link by text features, but linked by the GNN. 3. Is the generated graph by GNN influenced by the number of datasets in training or the models? For example, if I use different models for the training, will the graph be the same? If I use two dataset in multi-dataset training, will the nodes of these two dataset labels have the same edges as in when I use three datasets? 4. Can I apply the generated graph to other models' training without the need for doing GNN training again? If not, how much training cost will GNN bring?
Questions
Please see the weaknesses.
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
The paper is interesting and the experiments seem to prove the effectiveness. But I need more evidence to prove the robustness of the GNN results when using different datasets and models.
Dear Authors, thanks for the great work and the clarifications/ additional efforts you put into the rebuttal! In general I'm satisfied with the update and would currently vote for Accept. The main weakness/flaw that remains is the hallucinated description of the labels generated by GTP3.5 solely from essentially extending a text "<label> for dataset <name>" without incorporating the images/masks themselves. However, it is interesting/provocative that this improves the automatic label unification. The results/generalizations are promising and the approach is novel enough. Thanks!
Dear Reviewer, thank you very much for your positive feedback. We acknowledge your concern regarding the hallucinated descriptions of the labels generated by GPT-3.5, particularly the potential shortcomings of relying on extended text without incorporating the images or masks themselves. We agree that integrating the corresponding image masks into the node features could enhance the accuracy of label generation and help mitigate the issues related to erroneous outputs from the model. Indeed, this is an important consideration, and we plan to explore this direction in our future research. Thanks!
read rebuttal
Dear authors, thank you for your rebuttal, which I have read. Ultimately my view on this paper does not change. There seems to be a limitation of hallucinating definitions for labels and matching them purely on textual information, where visual information for matching is only propagated during training, meaning unseen datasets are aligned purely on GPT-generated definitions. I don't fully understand the comment about MSeg. Are the authors suggesting mIoU values are compared between methods that are measured over different sets of labels? That would be strange. In any case, I can accept that MSeg does not follow a standard protocol for these datasets. Overall I lean towards keeping my original score of accept, pending the discussion with other reviewers.
I acknowledge the authors' efforts in the rebuttal and have read it. The paper itself is interesting and adequate from a technical point of view. However, my main concern is the lack of comparisons and discussion with previous SOTA in the semantic segmentation community, which hinders the evaluation of this paper. I believe that high time complexity should not be a reason to avoid comparisons. On the contrary, it could introduce new challenges for practical applications. I suggest the authors include Transformer-based comparison methods and optimize the time complexity. Therefore, I will maintain my score.
Feedback on the rebuttal
I would like to thank the authors for the thorough feedback and additional experiments. Still, even after reading it, I am keeping my original score. The new results show that even though the proposed method improves the results on individual benchmarks it does so at the expense of relation discovery quality (Q2, W4, and corresponding answers by the authors). This is connected to my main concern (W1), and that is that individual benchmark performance is not a good proxy for evaluating the task of label unification. Furthermore, the WD2 performance is named as one of the strengths of this approach, but at this moment it is not clear if the improvements are due to the methodological contributions of this paper or the choice of training data. I am inclined to believe that it is due to omission of COCO and ADE20K classes in the final taxonomy. This is also suggested by the significantly worse performance of the zero-shot model which was trained on the Vistas dataset which should be enough for a good result on WD benchmark. With regards to W3, previous work comes to similar conclusions, so that is not enough of a contribution.
Thanks for the response. The rebuttal is clear and solves my concern. I would like to raise my rating to 6.
Decision
Accept (poster)