Library

Cooperative Hardware-Prompt Learning for Snapshot Compressive Imaging

Existing reconstruction models in snapshot compressive imaging systems (SCI) are trained with a single well-calibrated hardware instance, making their performance vulnerable to hardware shifts and limited in adapting to multiple hardware configurations. To facilitate cross-hardware learning, previous efforts attempt to directly collect multi-hardware data and perform centralized training, which is impractical due to severe user data privacy concerns and hardware heterogeneity across different platforms/institutions. In this study, we explicitly consider data privacy and heterogeneity in cooperatively optimizing SCI systems by proposing a Federated Hardware-Prompt learning (FedHP) framework. Rather than mitigating the client drift by rectifying the gradients, which only takes effect on the learning manifold but fails to solve the heterogeneity rooted in the input data space, FedHP learns a hardware-conditioned prompter to align inconsistent data distribution across clients, serving as an indicator of the data inconsistency among different hardware (e.g., coded apertures). Extensive experimental results demonstrate that the proposed FedHP coordinates the pre-trained model to multiple hardware configurations, outperforming prevalent FL frameworks for 0.35dB under challenging heterogeneous settings. Moreover, a Snapshot Spectral Heterogeneous Dataset has been built upon multiple practical SCI systems. Data and code are aveilable at https://github.com/Jiamian-Wang/FedHP-Snapshot-Compressive-Imaging

cs.CVEngineeringComputer Science
View graphSource

Paper

Similar papers

FedHC: A Scalable Federated Learning Framework for Heterogeneous and Resource-Constrained ClientsMin Zhang, Fuxun Yu, Yongbo Yu et al.2023 · arXiv (Cornell University) · 3 citations In Library
Efficient Client Sampling with Compression in Heterogeneous Federated LearningOuiame Marnissi, Hajar El Hammouti, El Houcine Bergou2024 · Conference on Computer Communications Workshops · 0 citations In Library
Federated Learning of Generative Image Priors for MRI ReconstructionGokberk Elmas, Salman Ul Hassan Dar, Yılmaz Korkmaz et al.2022 · IEEE Transactions on Medical Imaging In Library
Federated Learning of Generative Image Priors for MRI ReconstructionGokberk Elmas, Salman UH Dar, Yılmaz Korkmaz et al.2022 · arXiv (Cornell University) · 122 citations In Library
Hardware-Sensitive Fairness in Heterogeneous Federated LearningZahidur Talukder, Bingqian Lu, Shaolei Ren et al.2024 · ACM Transactions on Modeling and Performance Evaluation of Computing Systems · 2 citations In Library
Addressing data heterogeneity in distributed medical imaging with heterosync learningHang-tong Hu, Ming-De Li, Xinxin Lin et al.2025 · Nature Communications · 5 citations In Library
FedCompass: Efficient Cross-Silo Federated Learning on Heterogeneous Client Devices using a Computing Power Aware SchedulerZilinghan Li, Pranshu Chaturvedi, Shilan He et al.2023 · arXiv (Cornell University) · 24 citations In Library
CA-HFP: Curvature-Aware Heterogeneous Federated Pruning with Model ReconstructionGang Hu, Yinglei Teng, Pengfei Wu et al.2026 · 0 citations In Library
FedHGPrompt: Privacy-Preserving Federated Prompt Learning for Few-Shot Heterogeneous Graph LearningXijun Wu, Jianjun Shi, Xinming Zhang2026 · Entropy · 0 citations In Library
Hypernetwork-based Personalized Federated Learning for Multi-Institutional CT ImagingZiyuan Yang, Wenjun Xia, Zexin Lu et al.2022 · IEEE Transactions on Neural Networks and Learning Systems · 54 citations In Library

Peer review

Accept (poster)NeurIPS 2024 · via OpenReview
Reviewer aeby5/10 · confidence 3/52024-07-10

Summary

The authors present a Federated Hardware-Prompt learning (FedHP) framework to address the fact that compressive snapshot spectral imaging devices may not be easily tuneable against changes in the coded aperture, and that in fact the said access to coded apertures may not be possible due to privacy reasons. The authors solve this by hardware prompt learning, which essentially learns from observing diverse coded aperture samples of all clients, regularizing the input data space and achieving the goal of coping with heterogeneity sourcing from hardware. The results show on a specific dataset improvement across all 10 samples in terms of spectral reconstruction quality. The comparison is primarily in the sense of federated learning approaches. Typo: figure 3 caption -> colorblue shouldn’t be there

Strengths

The presentation is somewhat accessible to a generally knowledgeable non-expert in federated learning, in that the purposes are clear.

Weaknesses

The biggest weakness is arguably that the paper covers a somewhat very niche topic, which is the application of a federated learning scheme to compressive snapshot spectral imaging. To some extent one would expect the technique to abstract away from the specific case of CASSI, as the solution does not particularly pertain to CASSI. In addition, due to limited data available in this setup and to very limited size datasets, it is difficult to ascertain the significance of the findings.

Questions

Can the authors extend this to any other compressive imaging scheme? Or perhaps disentangle the improvements in terms of FedHP, from those specific to the application? This would also broaden the data available for validating the experiments.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

The addressed setup assumes that the problem the authors propose to tackle is meaningfully posed, I.e., that federated learning in the chosen formulation is practically meaningful. The reviewer is not sure whether this is a practically relevant problem considering that CASSI systems are arguably scientific instrumentation/experimental devices whose calibration is likely done per case anyways. In addition the topic would appear to be more meaningful for publications that cover CASSI systems such as IEEE TGARRS or the like. It is hard for this reviewer to disentangle the margin of novelty of this paper in terms of federated learning approach vs. impact on the target application.

Reviewer aeby2024-08-13

The reviewer is willing to acknowledge the amount of work done by the authors to prove their point, and has decided to update the recommendation. Please update, however, your references to mention that significant future work will be done in the direction of lensless cameras, and CT reconstruction, i.e., in the presence of hardware-defined forward models. This must also include thoroughly all hyperspectral and multispectral compressive imaging strategies that include, among others, random masks applied in a number of ways, including those by random convolution, if the authors see them as feasible. In absence of other forward models in the current paper, it is important that the authors update the manuscript accordingly with a comprehensive set of references on what forward models could be studied in future work, that should be supported by experimental evidence. Thanks again for addressing my concerns.

Authorsrebuttal2024-08-13

Response to Reviewer aeby

We thank the reviewer for acknowledging the response and for updating the recommendation. In line with the reviewer’s suggestions, we will revise the manuscript to outline potential future work in areas such as lensless cameras, LiDAR, HDR cameras, and CT reconstruction, especially concerning hardware-defined forward models. We will also include a comprehensive set of references on related hyperspectral and multispectral imaging strategies, incorporating forward models that could be explored in the future work. We much appreciate the reviewer's insightful suggestions, which have significantly contributed to the enhancement of the paper's quality and scope!

Reviewer 1X5M5/10 · confidence 4/52024-07-12

Summary

The paper addresses the challenges faced in snapshot compressive imaging (SCI) systems due to hardware shifts and the need for adaptability across multiple hardware configurations. By introducing a hardware-prompt network and leveraging federated learning, the framework enhances the adaptability and performance of SCI models across different hardware configurations.

Strengths

1. The manuscript is well-organized with a clear and logical structure that enhances the readability of the content. 2. The paper provides a detailed background on SCI and FL. The planned release of the Snapshot Spectral Heterogeneous Dataset (SSHD) will significantly aid future research. 3. Using different coded apertures for different clients closely mirrors real-world scenarios, adding significant practical relevance to the study.

Weaknesses

1. The literature review on federated learning (FL) heterogeneity in the Introduction section lacks comprehensiveness. There are numerous recent papers addressing heterogeneity in FL that are not cited here. Additionally, the references included are somewhat outdated. Including more current and diverse references would strengthen the review and provide a more accurate context for the study. 2. the manuscript explains that the coded apertures for each client follow a specific distribution Pc, it does not provide further details about the exact nature or type of this distribution. 3. There are many ways to partition data to construct heterogeneous scenarios, such as practical and pathological methods. The approach of equally splitting the training dataset according to the number of clients is not very convincing. The authors should try different partitioning methods. 4. It is unclear which datasets were used to obtain the experimental results in Tables 1 and 2. The authors did not specify this, which creates confusion in the experimental analysis.

Questions

1. What is the rationale for using adaptors, and what is their function? 2. What network models are used in the comparison methods? It is necessary to clearly state the fairness of the validated methods. 3. The explanation for Figure 3 is not detailed enough. For example, what is "Patch"?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

1. In the "Discussion of the client number" section, the number of clients increases very little, and the metrics slightly decline. However, the authors conclude that the performance is stable with the change in the number of clients. The small variation in the number of clients is unconvincing. A larger difference in the number of clients should be set to demonstrate this more effectively. 2. The authors mention the "presence of data privacy" in the contributions, but there is no further discussion or experimental comparison regarding data privacy in the subsequent sections. This makes it difficult to validate their contribution to data privacy protection.

Reviewer 1X5M2024-08-12

The authors have addressed most of my concerns; however, two issues remain unresolved, so I will keep my rating unchanged. 1) Regarding the discussion of the client number, Table 4(a) in the paper shows that when C=5, FedHP outperforms FedAvg by 0.27 dB. However, in the rebuttal (Section A3.8), this performance gap is reported as 0.21 dB. The authors have not provided an explanation for this discrepancy. 2) The authors mentioned in the rebuttal that they would add a detailed description of Privacy Protection, but this has not been presented.

Authorsrebuttal2024-08-12

Response to Reviewer 1X5M

We appreciate the reviewer's constructive comments and the recognition of our rebuttal. The presentation of *0.21dB improvement for C=5* in `A3.8` was a typographical error and should be consistent with the manuscript, which correctly states a *0.27 dB improvement for C=5*. Our intention was to highlight the advantage of the proposed FedHP and FedAvg as presented in Table 4 (a) for *C=5*. Additionally, we provide a detailed description of the privacy protection inherent in our approach as follows. FedHP inherently addresses privacy from different perspectives. (1) **Hardware decentralization**: In the FedHP framework, real hardware configurations (e.g., real masks) remain confidential to the local clients. This design makes it difficult to reverse-engineer the pattern or values of the real mask without direct sharing. (2) **Raw data decentralization**: FedHP maintains a private hyperspectral dataset for each client. The hyperspectral images are processed locally (e.g., encoding or data augmentation) and never leaves the client, thereby minimizing the risk of exposure. (3) **Training process decentralization**: FedHP only collects the local updates from the prompt network, which are then shared with the central server. The local updates are anonymized and aggregated without accessing underlying data, preventing any tracing back to the data source and thus protecting confidentiality. In Table 3, we quantitatively compared the performance of the proposed “FedHP” and “FedHP w/o FL” under privacy-constrained environments. FedHP demonstrates a $0.6$ dB average improvement (e.g., $31.35$ *v.s.* $30.75$), showcasing its robust model performance and offering a significant privacy advantage that aligns with regulations restricting data sharing. We will add the above discussions into the manuscript. We sincerely expect our response can help solve the reviewer’s concern and expect a future discussion with the reviewer!

Reviewer Jxiy6/10 · confidence 4/52024-07-13

Summary

Most existing reconstruction models in snapshot compressive imaging systems are trained using a single hardware configuration, making them highly susceptible to hardware variations. Previous approaches attempted to address this issue by centralizing data from multiple hardware configurations for training, but this proved difficult due to hardware heterogeneity across different platforms and privacy concerns. This paper proposes a Federated Hardware-Prompt Learning (FedHP) framework, which aligns data distributions across different hardware configurations by correcting the data distribution at the source, thereby enabling the trained model to adapt to multiple hardware configurations. The performance on existing datasets shows an improvement compared to previous popular training frameworks. Additionally, the authors have released their own created dataset and code.

Strengths

1.Previous work focused on the data itself, directly correcting various types of data through network models. In contrast, the authors of this paper focus on the root cause of the differences—hardware. They address the issue from the perspective of learning the differences in hardware. 2.The method proposed by the authors has achieved excellent performance compared to existing mainstream methods, and the average performance has also improved.

Weaknesses

1.The number of clients used in the experiments is still relatively small. Although a simple comparison of the impact of different numbers of clients was made, there is not much difference in performance compared to other methods when the number of clients is larger. 2.Although good results were reported on simulated data, more results on real data should be included to evaluate the effectiveness of the proosed method.

Questions

Why does the prompter lead to such a significant improvement, while the effect of the adaptor is not as pronounced? Please provide an in-depth analysis.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The generalization to different hardware systems is crucial for deep learning based methods. The current form of this manuscript only reported on a small scale real dataset captured by several systems. A larger dataset captured by more systems is necessary to evaluate the method.

Reviewer BdgG6/10 · confidence 4/52024-07-13

Summary

The paper introduces FedHP, a reconstruction method for snapshot compressive imaging systems, which addresses the challenge of cross-hardware learning by proposing a federated learning approach. The key contribution lies in using a hardware-conditioned prompter to align data distributions across different hardware configurations, thereby enhancing the adaptability of pre-trained models without compromising data privacy.

Strengths

1. The writing of the paper is good, making it easy to read and follow with clear arguments. 2. The problem defined in the paper is novel with a clear motivation, providing good inspiration for solving the issue of inconsistent device configurations in snapshot compressive imaging. 3. The proposed method is clear and the conclusions are relatively convincing. Overall, it is an interesting work.

Weaknesses

1. There are some typos in the writing. For example, the caption of Figure 3 and the bold parts in the second row of Table 1 and the eighth row of Table 2 are confusing. 2. The proposed FedHP method is relatively straightforward and lacks deeper insights. Moreover, it does not show a significant performance improvement compared to FedAvg. 3. The experiments are not comprehensive enough. Given that this work aims to address the snapshot compressive imaging (SCI) problem, I suggest adding experiments to test the applicability of other SCI systems, such as Coded Aperture Compressive Temporal Imaging (CACTI). 4. There is a lack of sufficient real-world experiments. It would be beneficial to set up multiple independent SCI systems to test the algorithm's performance. Including reconstruction results obtained from these real-world systems is recommended.

Questions

1. All the experiments in this paper are based on the SD-CASSI model. Can the same FedHP model be simultaneously applicable to both DD-CASSI and SD-CASSI architectures, which have significantly different designs? 2. Although the proposed method outperforms other algorithms in terms of performance metrics, there are still many artifacts in the reconstructed images. While I understand that this is maybe due to the precision issues of the CASSI system, it is crucial for evaluating the practical usability of the algorithm. Additionally, I am not sure whether the spectral accuracy of the reconstructed images is also optimal in statistical terms, which is vital for spectral imaging systems. 3. Furthermore, if possible, I hope the authors can also address the concerns I raised in the Weaknesses section.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Reviewer aeby2024-08-12

The authors have addressed all of my questions with their view on why FedHP is a substantial improvement for CASSI. This reviewer would like to thank them for the time spent in providing further experiments on the CACTI case, as well as the time spent to prepare the replies. Focusing on the new Table T1, it appears that FedAvg and FedHP on CACTI perform very closely in PSNR and SSIM, notably in PSNR one could argue that the two distributions overlap. Have the authors performed an analysis of the residuals so that they would be able to ascertain whether the resulting models are statistically distinguishable from their residuals? I.e., not looking only at the mean and standard deviation but at whether the resulting distribution of residuals from FedHP is similar to that of FedAvg. It appears that the results are in a close tie on the current dataset and, not having immediate clarity of how sensitive the dataset is, it is difficult to ascertain whether the proposed technique is sufficiently of impact. The authors report they believe that their technique could be generalized to "several potential related applications that might benefit from the proposed method, such as Lensless camera ,LiDAR, HDR camera, or CT-Reconstruction". This reviewer fully agrees that additional evidence on other modalities, potentially with larger datasets to ascertain the margin between the baseline and FedHP, could increase very significantly the strength and quality of the present paper submission. Due to this, the reviewer would lean to leave my previous rating unaltered.

Authorsrebuttal2024-08-13

Response to Reviewer aeby

We appreciate the reviewer's constructive comments and the recognition of our rebuttal. To address the concern regarding the performance comparison between FedHP and FedAvg, we conducted a statistical analysis using a paired t-test to compare the PSNR and SSIM values from FedHP and FedAvg. Specifically, we define the hypotheses as follows: (1) Null hypothesis ($H_0$): there is no significant difference in the PSNR and SSIM values between FedAvg and proposed FedHP. (2) Alternative hypothesis ($H_a$): there is a significant difference in the PSNR and SSIM values between FedAvg and proposed FedHP. We calculated the differences based on the averaged PSNR and SSIM values for each scene from both FedAvg and FedHP, resulting in ten differences values for PSNR ($d_{PSNR}$) and SSIM ($d_{SSIM}$). We performed the paired t-test using $t = \frac{\bar{d}}{s\_d/\sqrt{n}}$, where $\bar{d}$ denotes the mean of the difference values for either PSNR ($d_{PSNR}$) or SSIM ($d_{SSIM}$), $s\_d$ is the standard deviations, and $n$ is the number of the paired observations (e.g., $10$). We calculated the p-value upon the t-distribution for a two-tailed test using the formula p-value$= 2 \times P(T>|t|)$, where $P(T>|t|)$ denotes the probability that a t-distributed random variable with $n-1$ degrees of freedom exceeds the absolute value of the observed t-statistic. For PSNR, we observe $t=2.50$ and p-value$=0.034$. Since the p-value is less than the typical significance level of $0.05$. Therefore, we reject the null hypothesis ($H_0$) and conclude that there is a statistically significant difference between the PSNR values of FedAvg and FedHP. For SSIM, we observe $t=7.39$ and p-value$=0.00004$. The p-value of is significantly less than $0.05$, indicating a very strong statistically significant difference between the SSIM values of FedAvg and FedHP. The test results in PSNR and SSIM confirms that the performance gap between FedHP and FedAvg is statistically significant. We thank the reviewer for the valuable feedback, which has enhanced the quality of our submission. We will add the above discussions into the final version and look forward to any further suggestions from the reviewer. We sincerely appreciate the reviewer’s time and effort.

Reviewer aeby2024-08-13

This reviewer appreciates very much the extra trials provided by the authors and believes this paper is borderline, leaning on the previous rating. The reviewer stresses how the newly brought evidence is valuable in making a point on the results with the studied dataset and architectures. However to increase the quality of the submission and to broaden its scope, which at present is very narrow, the reviewer invites the authors to investigate the "several potential related applications that might benefit from the proposed method, such as Lensless camera, LiDAR, HDR camera, or CT-Reconstruction". Showing that the technique applies regardless of the forward model, or on a range of forward models could allow the authors to increase the value proposition of this paper to a "Cooperative Hardware-Prompt Learning for Linear Sensing Models", broadening the results with more datasets such as those in Compressive CT-Reconstruction or MRI, for example. The rating on this reviewer's side is exactly borderline (leaning to borderline reject).

Authorsrebuttal2024-08-13

Response to Reviewer aeby

We sincerely appreciate the reviewer’s constructive comments and the time in reviewing the submission. Due to the limited rebuttal time and the significant workload involved in deploying new hardware and collecting additional data, we will investigate the potential applications such as Lensless cameras, LiDAR, HDR cameras, and CT-Reconstruction in future work. The proposed FedHP can address the challenge of hardware heterogeneity in snapshot compressive imaging by integrating hardware prompt learning within a federated learning framework, highlighting the unique intersection of computational imaging and privacy-preserving federated learning. We would appreciate it if the reviewer can further assess the contribution of this work from the above perspective. We thank the reviewer again for the valuable comments and effort in reviewing this work!

Reviewer Jxiy2024-08-13

Thanks for the explanations. Concerns like real-system evaluation cannot be addressed in such a short period. I keep my initial rating.

Authorsrebuttal2024-08-13

Response to Reviewer Jxiy

We appreciate the reviewer's recognition of our response and support for our work!

Reviewer BdgG2024-08-13

The author has addressed most of my concerns. I will keep my score unchanged.

Authorsrebuttal2024-08-13

Response to Reviewer BdgG

We appreciate the reviewer's approval for our response and recognition of our work!

Reviewer 1X5M2024-08-14

Thanks to the authors for addressing my concerns.

Authorsrebuttal2024-08-14

Response to Reviewer 1X5M

We appreciate the reviewer's valuable comments. We thank the reviewer's recognition of our rebuttal!

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC