Summary
The authors have proposed a co-design method for making the Homomorphic encryption (HE) evaluation faster in private inference. Specifically, they used a new HE encoding mechanism to leverage the properties of the black-circulant matrix to reduce the HE rotations and ciphertext-plaintext multiplications. The experiments are performed on various CNN models and ViT models with multiple datasets.
Strengths
1. Authors have tailored the SOTA HE algorithm, BSGS implemented in BOLT, to leverage the block-circulant matrix properties to further speed up the HE evaluation in private inference. They have implemented various crypto algorithms, which require a reasonable effort in C++.
2. Thorogh experimentation, including ViT models on ImageNet, TinyImagNet, and CIFAR-10/100 datasets.
3. The paper is well-structured and thoroughly written, making it easy to follow even for those who are not experts in this field.
Weaknesses
$\bullet$ **Limited Novelty** The idea of employing black-circulant matrices for faster convolution, both in plaintext [1] and ciphertext [2], is not new. The authors have just tailored them for the BGSG algorithm, by making the block-size variable.
Moreover, the efficiency of the proposed encoding method, CirEncode, has been validated with a relatively smaller hidden size. For example, ViT with hidden dimensions 192 and 256. In contrast, the BERT-based model, used in BOLT, has a hidden dimension of 768. The efficacy of the encoding mechanism should be validated against 768 and/or 1024 hidden dimensions.
$\bullet$ **Compariosn against HybReNets** Authors have shown comparison with ResNet18 baseline in Figure 9. However, recently, there have been more efficient baselines for private inference have been devised such as HybReNets (e.g., HRN-5x5x3x and HRN-2x5x3x) [3]. These privacy-friendly baseline networks have a relatively higher number of channels in deeper layers, thus, the efficacy of the encoding mechanism should be validated with the networks with a higher number of channels.
$\bullet$ **Writing**
The year is missing in citations 27 and 38.
[1] Ding et al., CirCNN: Accelerating and Compressing Deep Neural Networks Using Block-Circulant Weight Matrices, MICRO 2017.
[2] Lou et al., Falcon: Fast Spectral Inference on Encrypted Data, NeurIPS 2020.
[3] Jha et al., DeepReShape: Redesigning Neural Networks for Efficient Private Inference, TMLR 2024.
Questions
1. For ViT experiments on CIFAR, did you use pre-trained ImageNet mode?
2. Why MobileNetV2 is used for benchmarking? If FLOPs efficiency is the reason, it should also be evaluated on RegNet-x [4] and ConvNeXt [5] models, and if required, by substituting complex non-linearities with privacy-friendly counterparts (LN with BN and GELU with ReLU).
3. Which Crypto-primitive is used for evaluating the nonlinear computations?
4. What is the overhead of finding the optimal block size using second-order information? That is, how much time is required to find the optimal block size for each layer? How does this timing overhead increase with the network size, particularly in wider networks (e.g., in HybReNets [3])
[4] Radosavovic et al., Designing Network Design Spaces, CVPR 2020.
[5] Woo et al., ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders, CVPR 2023.
*I'm open to improving the score if the key concerns are addressed in the rebuttal*
Limitations
Addressed in the paper.