CoPriv: Network/Protocol Co-Optimization for Communication-Efficient Private Inference

Summary

This paper presents a framework that simultaneously optimizes the 2PC inference protocol and the neural network architecture to achieve a significant reduction in communication. The framework outperforms state-of-the-art (SOTA) approaches by achieving communication reduction.

Strengths

1. The paper is well-written and easy to understand 2. The paper highlights the current scenario where nonlinearity no longer dominates the communication overhead.

Weaknesses

1. It would be beneficial to include more ablation experiments to demonstrate the individual contributions of the ReLU pruning and re-parameterization approaches. 2. Providing the percentage or number of multiplication reduction resulting from the Winograd algorithm would provide additional insights into the overall communication reduction. 3. Please ensure that the legends are appropriately positioned within the figures, specifically those that cover certain data points. 4. Empirically comparing the proposed ReLU pruning method with prior methods would provide a deeper understanding of its effectiveness. 5. The most recent work on the ReLU reduction is as follows, the author could consider including it in the introduction: "S. Kundu, et al., Making Models Shallow Again: Jointly Learning to Reduce Non-Linearity and Depth for Latency-Efficient Private Inference, CVPRW 2023."

Questions

Please refer to the weaknesses section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Please refer to the weaknesses section.

Summary

The paper presents optimizations to secure two-party computation of convolutional network inference. There are optimizations for both linear and non-linear layers, resulting in an overall single-digit factor improvement.

Strengths

The optimizations look interesting and are underlined well with benchmarks. I particularly appreciate the trade-off visualization in Figure 8.

Weaknesses

Table 1 doesn't make sense to me. I don't think there is merit in pointing out that prior work hasn't optimized certain aspects because it might be that the work is efficient without extra effort. If anything, the table should contain numerical speed-ups. Using the acronym ASS for arithmetic secret sharing might put off readers as it's identical to a vulgar word.

Questions

n/a

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

n/a

Summary

The paper introduces CoPriv, a framework that optimizes the 2-party computation (2PC) inference protocol and the deep neural network (DNN) architecture to reduce communication overhead. CoPriv features a new 2PC protocol for convolution based on Winograd transformation and develops DNN-aware optimization to reduce inference communication.

Strengths

The authors highlight a significant point that pruning ReLU may no longer be the most effective method for reducing computational and communication costs in private inference. This is due to the increasing prominence of linear and truncation operations. This insight is of considerable importance to the community.

Weaknesses

1. Figure 1 could benefit from more specific details. The left figure should include specific numbers for each portion and the amount of communication cost reduction achieved by each technique. It's also unclear which private inference method is used in the left figure. 2. The paper does not clearly explain why the DNN-aware adaptive convolution protocol can reduce communication costs. As a major contribution, it would be beneficial if the authors could provide a detailed explanation of how the selection of protocol initializer impacts communication costs and the criteria for selecting the initializer. 3. The novelty of the proposed ReLU pruning method is not clear unless the authors can explain how L_{comm} affects the training results. It would be valuable to compare the proposed ReLU pruning method with DeepReduce/SNL/SENet in terms of which ReLUs remain in the network and final accuracy. 4. It would be beneficial to differentiate the proposed re-parameterization method from the following work: "Making Models Shallow Again: Jointly Learning to Reduce Non-Linearity and Depth for Latency-Efficient Private Inference" (https://openaccess.thecvf.com/content/CVPR2023W/ECV/papers/Kundu_Making_Models_Shallow_Again_Jointly_Learning_To_Reduce_Non-Linearity_and_CVPRW_2023_paper.pdf). 5. From Figure 9, it appears that the authors do not apply ReLU pruning and re-parameterization to each block. The criteria for deciding which blocks are suitable for ReLU pruning and re-parameterization are not clear. 6. In Table 4, the accuracy remains the same for MobieNetV2 with and without pruning+re-parameterization. Since pruning and re-parameterization typically lead to accuracy degradation, could the authors provide an explanation for these results? 7. Given that Cheetah outperforms CrypTFlow2, it would be more valuable and informative to compare the proposed protocol optimization method with Cheetah rather than CrypTFlow2. This comparison could provide a more accurate assessment of the proposed method's performance relative to the current state-of-the-art.

Questions

Please refer to the weakness section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

While the authors acknowledge the limitation of Winograd convolution, stating that it can only be applied to 3x3 depth-wise convolution, they do not discuss the limitations of their entire work.

Summary

This paper presents CoPriv that jointly optimizes 2PC protocols and DNN architectures. It argues that SOTA 2PC protocols mainly focus on minimizing ReLU-based metric, which no longer contributes to the majority of communication. It proposes a new protocol for convolution with Winograd transformation and proposes a series of other DNN-aware optimizations. It shows communication reduction compared to SOTA protocols, CrypTFlow2, and other network optimization methods.

Strengths

(1) The observations in the motivation section are informative, well written and supported. (2) The Winograd based transformation with tile aggregation is well motivated and well illustrated (Figure 4). (3) The adaptive convolutional protocol is novel, insightful and effective. (4) The ablation study which adds optimizations step by step is conclusive and well-conducted. (5) The proposed methods consistently achieve communication reduction compared to SOTA.

Weaknesses

The implication of end-to-end speedup is not well studied.

Questions

The paper shows a consistent communication reduction, which the reviewer appreciated. However, it would be better if the author can provide some analysis on the end-to-end speedup using common setups.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper only analyzes the communication volume, where the effect on the communication time, inference speedup has not been well discussed. It can be the case that with higher bandwidth, this communication volume reduction will not be significant. However, the reviewer appreciates the communication volume reduction alone, and would not consider this as a major limitation.

Thanks!

Thank you to the authors for the clarifications. My concerns are now addressed, and I have accordingly raised my score.