Navigating Data Heterogeneity in Federated Learning A Semi-Supervised Federated Object Detection

Summary

To solve the challenges with limited high-quality labels and non-IID client data in federated learning, the authors present a pioneering SSFOD framework, designed for scenarios where labeled data reside only at the server while clients possess unlabeled data. Meanwhile, they propose the FedSTO, which consists of selective training, orthogonal enhancement, and personalized EMA-driven semi-efficient teacher. Finally, FedSTO achieves 0.082 and 0.035 higher mAP@0.5 when compared to partially supervised and SSFL baselines respectively.

Strengths

1. Good motivation. On the one hand, selective training can address the primary challenge of establishing a robust backbone for object detectors in FL. Specifically, it fosters more consistent representations by sharing the same non-backbone part. On the other hand, orthogonal enhancement reduces the bias towards specific weather conditions and the heterogeneity of object categories, leading to improved performance. 2. Many experiments have also verified the effectiveness of the proposed FedSTO.

Weaknesses

1. The authors do not detailed introduce Personalized Pseudo Labeling for Unlabeled Clients, which is not easy to understand. 2. I don't understand what Fig.3 wants to express, why there will be Fully supervised in Unlabeled overcast, rainy, and snowy. 3. In Fig.1, I don't know how the orthogonal enhancement works on the neck and head. Besides, I can not understand how to generate pseudo labels and assign them to the client models.

Questions

1. What do T1 and T2 in Algorithm 1 represent?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

Yes, the authors have described limitations.

Summary

This paper introduces a novel framework called Semi-Supervised Federated Object Detection (SSFOD) to tackle the problem of object detection in a federated learning setting. In this framework, the server possesses labeled data, while the clients hold unlabeled data from different distributions. The proposed approach consists of two stages: selective training and orthogonal enhancement. In the selective training stage, the focus is on updating only the backbone parameters on the clients to establish a robust backbone for the object detector. This selective approach helps improve the model's generalization capabilities across different distributions. The orthogonal enhancement stage follows, where all parameters are fine-tuned with orthogonal regularization. This regularization promotes representation divergence and robustness, further enhancing the model's performance. The paper also introduces a personalized pseudo label assigner based on a local exponential moving average (EMA) model. This assigner generates high-quality pseudo labels for object detection tasks, facilitating the training process in the semi-supervised setting. To evaluate the proposed SSFOD framework, this paper conducts experiments on three datasets: BDD100K, Cityscapes, and SODA10M. The results demonstrate that the proposed method achieves state-of-the-art performance when compared to existing approaches in both semi-supervised and federated learning domains.

Strengths

1, This paper is well-motivated. In practical applications, not all data on clients are labeled, and how to leverage unlabeled data is important for FL. 2, The writing is clear and easy to follow. 3, The paper performs extensive experiments on three diverse datasets, encompassing varying scales, complexities, and domains. Moreover, the proposed method is compared against multiple baselines as well as state-of-the-art techniques. The experimental results consistently demonstrate improvements across different metrics, object categories, weather conditions, and data distributions. This comprehensive evaluation reinforces the effectiveness and robustness of the proposed method, highlighting its superiority over existing approaches in various scenarios.

Weaknesses

1, This paper lacks discussion on the communication efficiency and scalability aspects of the proposed method, which are important considerations for its practical implementation. Specifically, it does not address the communication overhead associated with uploading only the backbone parameters or utilizing local exponential moving average (EMA) models. Furthermore, the paper does not investigate the performance of the method as the number of clients or the size of unlabeled data increases. 2, It could be better to compare or relate the proposed method with existing works on semi-supervised or self-supervised FL [1,2,3,4]. What are the advantages of the proposed method over these existing methods? [1] Zhuang, Weiming, Yonggang Wen, and Shuai Zhang. "Divergence-aware federated self-supervised learning." arXiv preprint arXiv:2204.04385 (2022). [2] Zhang, Fengda, et al. "Federated unsupervised representation learning." arXiv preprint arXiv:2010.08982 (2020). [3] Wu, Yawen, et al. "Federated contrastive learning for volumetric medical image segmentation." International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2021. [4] Dong, Nanqing, and Irina Voiculescu. "Federated contrastive learning for decentralized unlabeled medical images." International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2021.

Questions

Please refer to the Weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Please refer to the Weakness.

Summary

This work focuses on a practical application of federated learning, federated semi-supervised learning for object detection. It assumes that the server has labeled data and the clients only have unlabeled data. The proposed method is two-fold: selective training and orthogonal enhancement.

Strengths

- This paper integrates multiple existing techniques for federated semi-supervised learning. It is technically sound. - The paper is generally well-written, clear, and easy to follow. - Experiments on three object detection datasets demonstrate the effectiveness of the proposed method.

Weaknesses

- The novelty is somewhat limited. The novelty lies in the integration of existing methods and making them work on the target use case, while the key algorithms are more or less adopted from existing works. - The evaluation scale can be extended to a larger number of clients. The majority of the experiments are run with 3 clients and only one experiment is run with 20 clients. The autonomous driving use case is more like cross-device FL. It’s important to evaluate on more clients with client sampling. - Some existing works with related techniques are not discussed: [1] and [2] fixed the head and only trains the backbone like the selective training. [3] and [4] also uses EMA in local training. - Since the first stage is to learn representation, a comparison could be done with federated self-supervised learning methods [3][4] for learning visual representations. - The organization of section 3 is not very straightforward. For example, 'Personalized Pseudo Labeling for Unlabeled Clients' is more suitable in Section 4 instead of problem statement. [1] Fedbabu: Towards enhanced representation for federated image classification, ICLR’22. [2] Spherefed: Hyperspherical federated learning. ECCV’22 [3] Collaborative unsupervised visual representation learning from decentralized data. ICCV’21 [4] Divergence-aware federated self-supervised learning. ICLR’22.

Questions

- Line 3 of the Algorithm conducts client sampling, while it seems that the experiments do not conduct client sampling. - The server and client are assumed to be from the dataset. What would be the impact if the dataset in the server and clients are not from the same dataset? It would be more practical as we are unable to assume that we can collect the data in a server similar to the clients.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are not discussed in the paper. It could contains some of the aspects such as the scale of the experiments.

Summary

This paper explores Semi-Supervised Federated Object Detection (SSFOD), a pioneering framework for distributed data sources with limited high-quality labels and non-IID client data, particularly in applications like autonomous driving. The authors present a two-stage strategy, FedSTO, encompassing Selective Training followed by Orthogonally Enhanced full-parameter training, to address data shift while representing the first implementation of SSFOD for clients with 0% labeled non-IID data. The proposed approach includes selective refinement of the detector backbone to avert overfitting, orthogonality regularization to enhance representation divergence, and local EMA-driven pseudo label assignment to produce high-quality pseudo labels.

Strengths

The paper provides helpful figures, explores the valuable direction of semi-supervised Federated Object Detection (SSFOD), and presents an effective two-stage strategy called FedSTO for addressing data shift in a distributed data source. The proposed approach achieves state-of-the-art results in multiple datasets. Additionally, the paper provides a clear problem statement that is easy to understand.

Weaknesses

However, there are a few areas for improvement before the paper can be considered for publication. First, the references list is incomplete as some essential references are missing, such as "Federated learning with label distribution skew via logits calibration."

Questions

see weaknesses

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors did not discuss limitation and ethical issues in the main body.

Summary

This paper presents a Semi-Supervised Federated Object Detection (SSFOD) framework featuring a two-stage training strategy, FedSTO, designed to address the challenges of heterogeneous unlabeled data in federated learning. The proposed framework employs selective training and orthogonality regularization with personalized EMA-driven pseudo-labeling to facilitate robust and diverse feature learning, enhancing object detection performance across multiple weather conditions and data distributions. The empirical results provide evidence of the merits of FedSTO over existing federated and semi-supervised learning methodologies. Notably, despite non-IID clients having no labels, FedSTO achieves performance comparable to fully supervised centralized models.

Strengths

• The paper introduces a solution for SSFL for object detection with a one-stage detector in non-IID weather conditions. • The paper outlines a new approach to SSFOD, combining existing methods and adapting them to the federated learning setting. • The paper is well-structured, has clear headings and subheadings, and is easy to follow.

Weaknesses

1. The inclusion of a federated setting with fully labeled data on at least one dataset would have provided a valuable comparison to the proposed approach. Also, while the comparison with the partially supervised baseline helps establish the lower bound of server pretraining, it is not a fair comparison due to the significant difference in training data volume. 2. Although the models are evaluated on their respective datasets, it would have been beneficial to assess their performance on a global test set as well. Context-specific evaluations are necessary, but examining the models' generalization can help avoid the issue of overly specialized clients, which is a concern in personalization. 3. An interesting analysis would have been to investigate how the performance gap of the proposed method changes with increasing amounts of data. The reported performance of the Fully Supervised Centralized Yolo-v5 Large in the paper is notably lower compared to other papers on full-scale datasets (e.g., YoloV5s achieves 77.2 in https://arxiv.org/pdf/2108.11250v7.pdf). This raises the question of whether the low performance and the closure in performance with the centralized setting is due to the limited training data. It would be valuable to compare the proposed approach against these scenarios with more data available or if possible employing a full server pretraining. 4. Regarding Cityscape, since the dataset does not provide precise weather information for each annotation, the data is distributed uniformly at random, and it's not clear how it is non-IID. Also, the non-IID aspect mentioned in the paper is limited to addressing the skew in feature distribution induced by weather variations. The authors could have utilized foggy-Cityscape and KITTI datasets o obtain a more realistic non-IID setting. 5. The proposed SSFOD problem setting is similar to unsupervised domain adaptative object detectors and test-time domain adaptation. Even, the ingredients of the proposed solution such as EMA are also known in the semi-supervised OD and domain adaptation literature. To me, the only difference is that the model parameters are updated on local clients instead of central weight update. Please clarify the differences and advantages of the proposed solution compared to similar domain-adapted object detection methods. 6. Many key components such as FPT with orthogonal regularization are adapted from existing literature such as [16]. Although I agree that the proposed problem setting is unique and novel, it is also important to clarify the novel technical contributions of the proposed solution. 7. What is the rationale behind using Yolov5? It will be beneficial to show that the proposed solution is generalizable on other families of object detectors such as Faster-R CNN and recent transformer-based object detectors such as DETR/Deformable DETR. 8. mAP@0.5 is not an ideal evaluation metric to qualify localization capabilities. Please report mAP@0.75 and COCO-style mAP for a better understanding of the precise localization capabilities of the model. 9. EMA, pseudo-label assignment on unlabelled data needs to be better explained. What are the training loss functions used for backbone weight update during the selective training step? 10. It is mentioned that the proposed solution uses augmentations such as Mosaic, left-right flip, large-scale jittering. But, it is not clear how are the corresponding ground truth box positions determined in the case of unlabeled images. 11. As the authors mention that SSFOD introduces additional computational overhead, it is desired to quantify the computational resources employed. 12. The paper contains some errors, such as a missing dot in Line 139 and the use of "IID" in Table 5.

Questions

Please see the weaknesses mentioned above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper does not have an explicit section for "limitations and negative social impact", but that is not a major concern for the research topic studied in the paper.

Thank you for the rebuttal and additional experiments.

Thanks for the well-organized rebuttal from the authors. The feedback solves most of my concerns regarding the paper, so I am tending to increase my rating to Weak Accept. But, I also would like to hear from other reviewers regarding their thoughts on the rebuttal and if they have some open issues.

Thank you for the rebuttal

Thank you for the additional details and explanations, which resolved some of my concerns. Nevertheless, the overall quality is still borderline, and my rating keeps unchanged.