DCT-CryptoNets: Scaling Private Inference in the Frequency Domain

Dear EoRi, We are grateful for your insightful feedback and careful consideration of our work. We hope that the following answers all of the questions that you had regarding this work: --- ### Response to Weaknesses/Questions > Q1/Q2: … why is the model trained locally? … If the client could train the model by themselves, why do they do inference locally? - DCT-CryptoNets employs a practical approach to private inference by performing model training locally on plaintext data. This strategy is motivated by the significant computational overhead associated with homomorphic training, which remains a challenging research area. Given that FHE-based inference on a single image can take several minutes, training a large model directly in the encrypted domain would be prohibitively time-consuming, potentially requiring weeks or even months. - However, this local training approach aligns well with the paradigm of inference model as a service (MLaaS), where a company or organization leverages its own secure infrastructure to train a model. The trained model is then encrypted and deployed in a potentially less trusted environment, such as the cloud, to provide private inference services to users. This separation of training and inference not only addresses the current limitations of homomorphic training but also enhances the practicality of deploying privacy-preserving models in real-world settings, ensuring both model confidentiality and user data privacy. > Q3: If they want to deploy the encrypted model on the cloud for other clients as a service. How is the key management solved? - We appreciate you raising this important consideration. In many private inference applications utilizing FHE, both a *secret* key and *evaluation* key are generated. The *evaluation* key can be seen as a specific type of public key which can be used to run a trained fully homomorphic encrypted neural network (FHENN). This approach ensures that only the user possessing their own *secret* key can decrypt their results: - Steps: 1. A trained FHENN is deployed to the cloud server. 2. User requests the FHENN cryptographic parameters. Once received they generated both the *secret* and *evaluation* keys locally. Each client would therefore have their own secret key. 3. User sends the *evaluation* key along with their encrypted inputs to the server. 4. Server runs private inference with the users’ *evaluation* key and encrypted inputs. Then sends encrypted results back to the user. 5. User then decrypts the results with their *secret* key. - We have included more detail in a pre-existing appendix section (Section A.2) which elaborates on these questions.

Dear SG3z, Thank you for your thoughtful review. We particularly appreciate your understanding of the key strengths of our approach. We would like to point out a potential typo and clarify that our method employs TFHE. Besides this, we hope that the following answers the questions that you had regarding our work: --- ### Response to Questions > Q1: Are there any special scaling or normalization techniques that need to be considered when considering the YCrCb color space components of an image? - Indeed, normalization is necessary when working with DCT coefficients, just as it is for other image representations. We follow the standard practice of normalizing the tensors using the mean and variance calculated from the full training dataset. However, since pre-computed mean and variance values for DCT coefficients on benchmark image datasets are not readily available, these statistics need to be calculated beforehand. - This normalization was not explicitly mentioned in the “Methodology” text but can be seen in the last step of the “DCT Encoding” section in Figure 2. This is also most evident in the provided anonymous code repository. > Q2: Can GPUs be used to any advantage in this approach? - GPUs can in fact be leveraged to accelerate DCT-CryptoNets. Recent advancements in FHE libraries, such as the introduction of GPU support in Concrete-ML v1.7.0 (released Sept. 2024), are already demonstrating the potential for latency improvements. While initial benchmarks show a 1-2x speedup, we anticipate even greater gains as these libraries mature and incorporate further optimizations. This aligns with existing research showcasing substantial speedups (up to 20x or more) for TFHE on GPUs [1, 2]. - Furthermore, initiatives like the DARPA DPrive program, focused on developing dedicated homomorphic hardware accelerators, hold the promise of even more dramatic latency improvements (up to 1,000x). We believe that hardware acceleration, both through GPUs and specialized hardware, will be crucial for the widespread adoption of privacy-enhancing technologies like fully homomorphic encrypted neural networks (FHENNs). > Q3: Are there any privacy preservation guarantees or "leakage" guarantees that could be developed around this approach. (I realize this may be challenging mathematically.) - Thank you for bringing this important point up. While FHE offers robust protection against many traditional attacks, understanding the potential vulnerabilities and attack surfaces in the context of DCT-CryptoNets is crucial for ensuring the system's overall security and privacy guarantees. Many of these considerations are specifically focusing on the security considerations unique to FHE-based private inference. - *Server-Side Attacks:* The server only receives encrypted DCT coefficients, protecting against a wide range of server-side attacks, including those from an honest-but-curious adversary. However, in the case of a malicious server, additional measures like verifiable computation (VC) may be necessary to ensure correct function evaluation [3, 4]. Integrating VC with FHE is an active area of research. - *Communication Channel Security:* Encrypted communication between the client and server safeguards against eavesdropping. However, secure key transfer mechanisms are crucial to prevent unauthorized access to the decryption key. - *Client-Side Vulnerabilities:* Data privacy relies heavily on the security of the client device, as this is where the data exists in plaintext. Robust client-side security measures, such as secure key storage and protection against malware, are essential. While FHE itself is secure against various attacks such as chosen/known plaintext attacks and chosen ciphertext attacks [5], these protections are contingent on the client's ability to safeguard their own secret key. - We have added a new appendix section (Section A.1) which details all of the above threat models and mitigation strategies. --- ### References [1] Morshed et al., CPU and GPU accelerated fully homomorphic encryption, IEEE HOST, 2020. [2] Wang et al., HE-Booster: An efficient polynomial arithmetic acceleration on GPUs for fully homomorphic encryption. IEEE Transactions on Parallel and Distributed Systems, 2023. [3] Viand et al., Verifiable fully homomorphic encryption, Arxiv, 2023. [4] Atapoor et al., Verifiable FHE via lattice-based SNARKs, IACR Communications in Cryptology, 2024. [5] Chris Peikert. A decade of lattice cryptography, Found. Trends Theor. Comput. Sci., 2016

Dear qW2q, We sincerely appreciate your feedback and the time you invested in reviewing our submission. Your constructive feedback provides both encouragement and practical guidance for strengthening this work. We are eager to respond to your questions and comments: --- ### Response to Weaknesses/Questions > W1.1: The lack of sufficient algorithmic contributions and research insights makes it less suitable for the ML conference… Operating in the frequency domain for private inference benefits is not a novel concept (see [1,2]) … Thus, a more fitting venue for this work might be a cryptography-focused conference. - We appreciate your feedback regarding the contributions and insights of our work. While individual components have definitely been explored, our work aims to bridge the gap towards practical privacy-preserving learning on large-scale networks and images. While fundamentally a cryptographic solution, our core focus and impact lie in advancing privacy in ML, making this contribution well-suited for this venue. - We acknowledge prior work on frequency-domain techniques for private inference, such as those utilizing the homomorphic discrete Fourier transform (HDFT) for the BFV [2] and CKKS schemes [4]. These methods, while introducing specialized convolutional blocks for frequency-domain processing within the homomorphic circuit, often exhibit limitations in accuracy and scalability. Such as Falcon [2] achieving 76.5% on CIFAR-10, ENSEI [1] achieving 82.0% on CIFAR-10 and Kim et al. [4] scaling to a Plain-18 but only encrypting the last 8 layers when operating on ImageNet. - In contrast, DCT-CryptoNets learns features from the DCT representation of images directly, enabling the network to learn from the rate of change in intensities. It does so without specialized convolution operators, enabling seamless integration with existing CNN architectures, achieves much higher accuracy than existing HDFT methods, and shows scalability to ImageNet, a feat unmatched by other frequency-domain private inference methods. - To address concerns about novelty and relevance to the ML domain, we've clarified the distinctions between DCT-CryptoNets and existing frequency-domain private inference techniques in Sections 2.2 and 2.5 (highlighted in red). We believe these revisions better demonstrate the contributions and applicability of our work within an ML context. > W1.2: … Also, the usage of quantization-aware training for lower-frequency components is simply an engineering tweak. … - We appreciate your perspective. However, we believe that quantization-aware training (QAT) is not merely an engineering tweak but a crucial component for optimizing TFHE-based FHENNs. TFHE computations fundamentally rely on Boolean and integer logic gates, making QAT essential for aligning network parameters with this representation to achieve high accuracy. > W2: The practicality of HE-only private inference remains questionable, especially when compared to hybrid protocol-based approaches … Thus, the primary motivation for pursuing HE-only inference appears to be the benefit of non-interactive private inference. - We agree that currently there are performance gaps between fully homomorphic encryption (FHE) and hybrid private inference methods. While hybrid approaches may offer better efficiency today, non-interactive FHE offers superior security, particularly in zero-trust environments or under threats like malicious servers. This enhanced security stems from FHE's ability to maintain constant encryption throughout the computation, unlike interactive hybrid methods that involve potentially risky decryption of intermediate values. - The current state of FHENNs is indeed largely confined to smaller images and neural networks. Our work is a stepping stone that aims to bridge the efficiency gap in FHE methods, facilitating their wider adoption. We anticipate that the combined progress in ML algorithms and systems, akin to this work, as well as improvements in homomorphic hardware acceleration will ultimately establish FHE as a compelling and practical solution for privacy-preserving ML inference applications. --- ### References [1] Song et al., ENSEI: Efficient secure inference via frequency-domain homomorphic convolution for privacy-preserving visual recognition, CVPR 2020. [2] Li et al., Falcon: A fourier transform based approach for fast and secure convolutional neural network predictions, CVPR 2020. [3] Huang et al., Cheetah: Lean and fast secure Two-Party deep neural network inference, USENIX Security 2022. [4] Kim et al., Optimized privacy-preserving CNN inference with fully homomorphic encryption. IEEE Transactions on Information Forensics and Security, 2023.