One-shot Federated Learning via Synthetic Distiller-Distillate Communication

Summary

The paper introduces FedSD2C, a new one-shot Federated Learning (FL) framework designed to address challenges in existing methods. Previous approaches have used data-free knowledge distillation to improve one-shot FL, but these methods struggle with data heterogeneity and scalability issues. FedSD2C aims to solve these issues by: 1.Introducing a distiller to synthesize informative distillates directly from local data, reducing information loss. 2.Sharing synthetic distillates instead of inconsistent local models to address data heterogeneity. The authors claim that empirical results show FedSD2C outperforms other one-shot FL methods, especially with more complex and real datasets.

Strengths

This paper focuses on one-shot Federated Learning (FL), which is an intriguing topic. The paper presents a comprehensive set of experiments, both on model performance and privacy.

Weaknesses

Thanks for the authors' efforts in presenting this paper. After carefully reading it, I have the following questions and comments: 1. On line 52, the values 4.21 and 2.06 are not clearly explained. I suggest adding a figure or table to provide more specific details about these numbers. 2. While I can imagine that transmitting synthetic data could improve the global model's performance more easily than transmitting the model itself, this inevitably leads to privacy concerns. Although the authors provide some data reconstruction experiments, the results are not entirely convincing: a) To my knowledge, dataset distillation/coreset selection does not inherently protect privacy [1]. I suggest the authors include additional experiments on Membership Inference Attacks (MIA) (based on results from data-free KD [2], I suspect coresets provide even less privacy protection, since data-free KD doesn't use any 'real' training data ). b) According to the privacy onion concept [3], memorization is relative, then in your method the selected images may be at higher risk of privacy leakage compared to those not selected. 3. Compared to other methods, the proposed method communication overhead seems larger. It requires to transmit both the auto-encoder and a large amount of synthetic data. 4. The method relies heavily on Stable Diffusion's auto-encoder, which seems to utilize very strong prior information. How well would your method perform on datasets where diffusion models are not as proficient, such as medical datasets? 5. To my knowledge, another one-shot FL work[4] that relies on Stable Diffusion achieved results of about 75.0 on ImageNette, which appears significantly better than your 55.90. [1] No Free Lunch in "Privacy for Free: How does Dataset Condensation Help Privacy" [2] Evaluations of Machine Learning Privacy Defenses are Misleading [3] The Privacy Onion Effect: Memorization is Relative [4] Federated Generative Learning with Foundation Models

Questions

see above

Rating

4

Confidence

5

Soundness

3

Presentation

2

Contribution

2

Limitations

see above

Summary

This paper proposes a one-shot FL method (FedSD2C), utilizing V-information to select local core set data and server-pretrained autoencoder and Fourier-domain perturbation to ensure privacy preservation for local "distillate" sharing. In comparison to existing works such as DENSE and Co-Boosting, FedSD2C can reduce information loss in one-shot FL and improve up to 2.7x global model performance.

Strengths

The paper tackles an important setting in FL and the one-shot performance outperforms listed related works.

Weaknesses

- The assumption that the server holds an autoencoder is pretty strong, as the autoencoder must be trained in the clients' data domain to ensure it works. - The paper only showed that Fourier-based perturbation can 'visually' protect privacy by using PSNR as a metric. Although the paper mentioned MIA, it did not evaluate existing privacy attacks. - The clarity of paper writing can be improved. Essential details are missing to understand the contributions. See more in the Questions section.

Questions

- Can you more rigorously demonstrate the privacy-preservation statement? For example, differential privacy is a widely accepted concept. Could you please show the provable privacy-preservation guarantee like DP? - It is unclear how Figure 1 is generated. The authors should detail the datasets, model, and necessary information. Otherwise, comparing the figures is meaningless. - What is the computational cost for the core-set selection step? - It is unclear which parts of the algorithm contribute to handling the heterogeneous data. It seems that coreset selection could contribute well if intentionally sampling (almost) balanced data. - What is the number of clients for Table 1?

Rating

4

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

Limitation was discussed

Summary

The paper presents FedSD2C, a novel one-shot federated learning (FL) framework that aims to improve communication efficiency, privacy preservation, and model performance. The approach addresses issues with data heterogeneity and information loss by synthesizing informative distillates from local data and sharing these instead of local models. Empirical results show that FedSD2C significantly outperforms existing one-shot FL methods.

Strengths

1. One-shot FL is a potential direction that can significantly minimize communication costs in FL. 2. This approach does not rely on sharing private data, which protects the data privacy. 3. Using V-information and fourier transform perturbation is interesting. 4. Experimental results show significant improvements.

Weaknesses

1. A critical question on the optimal observer of approximating the V-information is that, the local model is trained locally, which may not indicate good V-information of the global datasets, i.e., all local dataets. 2. The privacy is not strictly guaranteed. The perturbation with 3, 4, 5 cannot ensure data privacy. 3. Figure 2 shows the reconstruction is very similar with the original images. 4. There is no theoretical analysis on privacy and generalization of the proposed method. 5. The pre-trained autoencoder plays a key role in the framework. But the experimental study doesn't investigate the impact of the pre-trained autoencoder on the performance.

Questions

1. What are the key hyper-parameters used in the core-set selection algorithm? 2. According to Algorithm 2, the core-set includes patches with different scales. How does the encoder handle patches with different scales? 3. There is still a huge performance gap between FedSD2C and Central in Table 1. What are the main factors that result in the performance drop?

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors have pointed out the computational overhead on local devices as the limitation.

Summary

This paper proposes a one-shot federated learning approach designed to enhance privacy protection, communication efficiency, and model performance. Firstly, the authors introduce a Core-Set selection method based on V-information to extract the most informative data from the original dataset. The amplitude spectrum of the images in the Core-Set is then perturbed using a Fourier transform, and these perturbed images are input into an Autoencoder to obtain their representations. Finally, these representations are transmitted to the server, which decodes the image information from the representations for training.

Strengths

The one-shot learning method proposed in this paper enhances privacy protection, transmission efficiency, and model performance compared to previous methods. The approach appears innovative, employing theoretically grounded techniques such as the Core-Set selection method, and using an Autoencoder and decoder for data transmission, which improves both transmission efficiency and model performance.

Weaknesses

1. In the Core-Set Selection stage, the authors do not explicitly define the patches used in Level 1: identifying the most informative image segments. I have the following questions: Are the patches the same size across different datasets? How many patches are extracted from each image? Does the size or number of patches affect the results? If so, how can one determine the optimal patch size? 2. In the perturbation stage, the authors do not provide sufficient reasoning for using the Fourier transform. Why must the Fourier transform be used instead of other transformations? For example, does using a wavelet transform and merging its low-frequency components achieve similar effects? Additionally, the authors mention that the merged image can be random noise. Is this too idealistic? Does using particularly weak random noise also achieve similar effects? 3. During the transmission phase using an Autoencoder, does the performance of the model depend on the pre-training of the Autoencoder? If a pre-trained model is not used, does this method still work effectively? 4. It would be beneficial for the authors to create a framework diagram of the entire method to enhance readability. 5. The paper contains many capitalization and punctuation errors, such as in line 50 and line 229. Additionally, please carefully review the citations in the article as they are quite disorganized, including issues with formatting, capitalization, and more. Please ensure consistency throughout.

Questions

1. The paper utilizes several large datasets, but is the method effective for low-resolution datasets such as CIFAR-10? 2. The authors should provide the model performance using only the Core-Set, where clients send all their original Core-Set data to the server for centralized training. 3. Please provide additional visual metrics beyond PSNR to make the results more convincing, such as SSIM. 4. The authors should conduct ablation experiments to demonstrate the performance of each part of the proposed method. For example, evaluating the model performance without the Core-Set data selection step. 5. Since only representations are ultimately transmitted, and attackers cannot recover images from representations, I wonder if the image perturbation stage is still necessary?

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

Yes, have discussed the limitations

Paper discussion

Hi all, Thanks again for your reviews! If you haven't done so already, please respond to the rebuttals before the end of the author-reviewer discussion period. If you don't have any further questions for the authors then a simple "thank you for the rebuttal" would suffice. All the best, Area Chair

response

Thanks for authors' response. As for the experiment on privacy attacks, since your model's performance is too low (far below 90%), it will also lead to biased MIA results. Therefore, I don't recommend claiming in the paper that your method can protect privacy (without any differential privacy guarantee). For another one-shot FL work[1], I think they also tested the method on medical datasets. After reading the rebuttal, I am happy with increasing the score to a 5 or 6 (if there is no overstatement in the final version). [1] Zhang J, et al. Federated Generative Learning with Foundation Models. 2023.

Thanks for the response

Dear authors, Thanks for your response. You have addressed most of my concerns, so I will raise the score to 5. However, one remaining concern is that you need to compare the Fourier transform and wavelet for different scenarios. For certain scenarios, wavelet may outperform the Fourier transform.

Response to rebuttal

Thanks for the clarification and responses. I will rasie my rating to 5.