Summary
This work targets better accuracy vs. efficiency trade-offs for multiplication-free tiny neural networks. Specifically, it uses multiplication-based Conv in training to augment Shift-based Conv and Add-based Conv optimization for higher accuracy in the Shift/Add-based networks. The experiments on image classification tasks show the proposed ShiftAddAug framework can have higher accuracy (e.g., +4.05% on CIFAR-100) while reducing energy consumption (e.g., 68.9% reduction) as compared to multiplication-based networks.
Strengths
1. Motivation: Current tiny DNNs are primarily designed using multiplication-based operators, often overlooking the more energy-efficient shift and add operators. Exploring shift/add-based tiny DNNs is, therefore, a worthwhile endeavor.
2. Comprehensive Review of Related Works and Preliminaries: The section on related works thoroughly covers existing multiplication-free networks. Additionally, the preliminaries provide a clear explanation of the shift and add operators utilized in this work.
3. Clear and Understandable Figures: The figures presented are clear, making the entire paper straightforward and easy to follow.
Weaknesses
1. Quality of the Draft: It appears the authors may not have thoroughly proofread their draft before submission. In Table 3, the performance of the proposed ShiftAddAug is denoted as "xx". This is a crucial detail for comprehending the efficacy of the suggested framework.
2. Ambiguity in the Contribution of the Proposed NAS: From the details provided in Sec. 3.4, the introduced NAS, which is highlighted as the third contribution, seems to essentially apply tinyNAS (Lin et al., 2020) over the ShiftAddAug. The search space is presented in Table 2 without clarifying its design rationale. Consequently, the true value-add of the proposed NAS remains ambiguous.
3. Unclear Contribution of the Weight Sharing Strategy: The weight-tearing issue that the proposed weight sharing strategy addresses is previously identified in ShiftAddNAS (You et al., 2022). The weight mapping strategy delineated in Eq. 5 is similar to the approach in ShiftAddNAS, which employed a learnable transformation kernel, T (·), to transition shared weights from a Gaussian to a Laplacian distribution. However, there are no theoretical justifications or empirical findings illustrating why this strategy outperforms the one in ShiftAddNAS.
4. Concerns on the Accuracy of the Efficiency Metric: As mentioned in Sec. 4.1, the efficiency metric chosen for this study is the energy and latency reported by an Eyeriss-like hardware accelerator simulator. However, the referenced studies (Chen et al., 2017; Zhao et al., 2020) are designed for multiplication-based networks. The authors have overlooked elaborating on the specific modifications implemented to adapt the simulator for multiplication-free networks. Given this, it's debatable if the evaluation backdrop is fair for multiplication-based networks. A recommendation for the authors would be to utilize more reproducible metrics, such as the latency from the TVM-based Shift/Add execution in ShiftAddViT (You et al., 2023).
Questions
Besides the previously listed weaknesses, I have the following questions:
1. The experiments exclusively consider the image classification task. How can it be asserted that this is the dominant task for IoT devices?
2. As indicated in Tab. 5, when introducing multiplication into the search space, the proposed ShiftAddAug displays a reduced accuracy compared to the baseline ShiftAddNAS (You et al., 2022). The given justification, which states "our method has given the multiplication-free operators strong capabilities, bridging the gap to the original operator", appears inconsistent. Notably, the ShiftAddNAS baseline actually boasts a similar count of Mult, Shift, and Add parameters. If the assertion were accurate, ShiftAddAug should outperform ShiftAddNAS in terms of accuracy. It seems more plausible that ShiftAddAug adversely affects the efficacy of multiplication-based operators. If this is the case, an exclusive emphasis on shift/add-only networks might be misplaced, as the existing multiplication hardware in IoT devices would remain underutilized.
Rating
3: reject, not good enough
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.