Activation Map Compression through Tensor Decomposition for Deep Learning

Summary

The paper addresses the challenge of on-device training for deep learning models, particularly focusing on the memory bottleneck caused by the storage of activation maps during backpropagation. The authors propose a method to compress activation maps using tensor decomposition techniques, specifically Singular Value Decomposition (SVD) and its tensor variant, High-Order Singular Value Decomposition (HOSVD).

Strengths

1. Using tensor decomposition for activation map compression to address a bottleneck in on-device training. 2. The paper provides a solid theoretical foundation, including error analysis and guarantees for convergence.

Weaknesses

1. Using SVD or tensor decomposition to compress activation maps incurs significant computational overhead. This approach of trading computational complexity for space complexity is questionable, especially considering that the computational capacity of embedded devices is usually also limited. Moreover, this compression may introduce additional information loss, potentially affecting the model's performance. 2. For SGD, recalculating activations instead of storing them is a viable method to save memory. Given that both methods trade computation time for storage space, why use low-rank compression which may potentially affect model performance? 3. With limited computational resources, why not search for a more appropriate model architecture using NAS?

Questions

1. LOMO[1] fused gradient computation and parameter update in one step to minimize the size of gradient tensors. Compared to this approach, what are the advantages of your method? 2. Would using quantization methods instead of compression methods produce better results? [1] Lv, Kai, et al. "Full parameter fine-tuning for large language models with limited resources." arXiv preprint arXiv:2306.09782 (2023).

Rating

3

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

None

Summary

This paper proposes a method to compress activation maps in deep neural networks using tensor decomposition techniques, specifically Singular Value Decomposition (SVD) and Higher Order SVD (HOSVD). The goal is to reduce memory requirements during backpropagation, enabling on-device learning for resource-constrained environments. The authors provide theoretical analysis of their method's impact on memory usage, computational complexity, and error bounds. They demonstrate the effectiveness of their approach through experiments on various tasks, architectures, and datasets.

Strengths

- Novelty: The use of tensor decomposition for compressing activation maps is a novel approach that addresses a significant bottleneck in neural network training. - Efficiency: The method significantly reduces memory usage during backpropagation, which is crucial for deploying deep learning models on resource-constrained edge devices. - Theoretical Support: The paper provides theoretical background for the proposed method, including error analysis and guarantees of minimal information loss.

Weaknesses

- Severe performance degradation relative to memory reduction - Complexity of Implementation: Implementing tensor decomposition techniques like HOSVD can be complex and may require specialized knowledge, potentially limiting its adoption. - Lack of Detailed Rank Selection: The paper does not provide sufficient explanation on how the appropriate rank for decomposition is selected, which is crucial for balancing memory reduction and accuracy. - Inconsistent and contradictory results: SVD frequently outperforms HOSVD, contradicting the authors' hypothesis and lacking adequate explanation. - Limited comparison with other activation map compression techniques.

Questions

- How does the proposed method perform on more diverse and complex neural network architectures, such as transformers? - What are the practical challenges and considerations when deploying this method on real-world edge devices with strict latency and power constraints? - How is the appropriate rank for decomposition selected, and what guidelines can be provided for tuning this parameter effectively?

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

- Specific to Certain Architectures: The experiments focus mainly on specific neural network architectures and tasks. It is unclear how well the method generalizes to other types of models and applications. - Dependence on Hyperparameters: The effectiveness of the compression is highly dependent on the chosen hyperparameters for the decomposition, which may require extensive tuning. - Resource Requirements for Decomposition: While the method reduces memory usage during training, the initial decomposition itself can be computationally intensive and may require powerful hardware. - Practical Usability Concerns: Despite significant memory reduction, the method results in considerable accuracy drops, raising concerns about its practical usability. Vanilla backpropagation, while using more memory, generally maintains higher performance and might be a more realistic approach for actual deployment.

Summary

[Editing to reflect my score increase from 6 to 7 after the author discussion phase.] The authors tackle the problem of memory consumption due to needing to keep realized activation tensors available between their use in the forwards pass and backwards propagation in training neural networks. They propose the use of tensor decomposition, specifically SVD and higher-order SVD, to compress activation maps and show that the compressed activations can be used in backpropagation without first decompressing. An error analysis is performed, and experimental results confirm the hypothesis that significant compression can be applied while maintaining significant explained variance. To validate the method's success on real workloads, the authors apply it to image classification and semantic segmentation tasks, fine-tuning pre-trained models for both tasks, as well as training from scratch for image classification.

Strengths

**Originality** I believe this is a novel application of tensor decomposition, a well-known technique previously applied to weight compression. Adaptively sizing the decomposed tensor to maximize retained information in the activation maps is an advancement over past work. **Quality** The authors have performed a reasonable set of experiments to show the technique works as intended: model quality when activations are subject to tensor decomposition for backpropagation remain competitive for image classification tasks and semantic segmentation tasks. Further, the authors have reported the memory consumption, proving that it is reduced significantly enough to satisfy constraints of edge devices in many cases. I greatly appreciate the time spent in the background, as well as the seemingly offhand note that any error induced by these decompositions will be limited to that layer's weight gradients and does not accumulate deeper into the network. **Clarity** I generally found the manuscript easy to follow: the overall organization was great, and the method itself was explained adequately. **Significance** The intended use case aside, I think even datacenter users may be interested in this technique, as many of today's latest networks are simply huge. Reducing the storage costs for activations between forward and backwards passes could reduce the parallelism needed, and in turn the overhead of communications in distributed training. I'd encourage the authors to consider this angle and apply the technique to, for example, large language models.

Weaknesses

**Originality** Only one piece of past work comes to mind that was omitted: Rhu et al., "Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks" (HPCA 2018) compresses activation maps, using their existing sparsity, to reduce the cost of offloading in order to accelerate training. While novel hardware was designed to keep up with compute throughput, if compression is the only goal, then this could simply be performed in software (like the submission's tensor decomposition). **Quality** A missing piece is the cost of compression: what impact does this have on the training speed? This is related to the claim on line 59 that "gradient computation and parameter updating are considerably more expensive than the forward pass," which isn't obvious to me. At a high level, each of the three calculations (forward propagation, gradient calculation, weight update calculation) are roughly equivalent in cost, and applying the weight update is very simple. Increasing the cost of any one of the phases would seem to be as painful as any of the others. Another claim that stood out, given the stated goal of "reducing the memory required for backpropagation" (line 72) is that the "main challenge limiting the feasibility of on-device learning lies in the computational cost of backpropagation" (line 58). I'd also question the claim that "we can assume that the networks considered are already [weight] compressed" (line 159) - while it's extensively studied, it's not a solved problem. It's fine to not tackle this particular problem, but stating that it's assumed to not be an issue is misleading. Some of the results deserve some extra discussion. Do the authors have a hypothesis about why SVD is superior to HOSVD (line 288)? Why does MobileNetV2's Vanilla BP results get worse when including 4 layers compared to only 2 (Table 2)? I have not seen CIFAR treated as a "downstream task" of ImageNet-1k before; it's typically just a simpler data set used, recently, as a proving ground before testing on larger data sets. Is it common to fine-tune a network trained on ImageNet-1k on CIFAR10/100? Given the significant memory savings of the technique, I'd have liked to see how it performs when fine-tuning all layers, not just a subset. Is there a reason this result wasn't gathered? **Clarity** I've had to make an assumption about the behavior of the method: during forward propagation, I assume the *uncompressed* activations are used, and only compressed when writing to memory to be used later in the backwards pass. (If this is not the case, then errors *will* compound.) Figure 2 shows speedups and space gains, but without indicating what, exactly, is speeding up or seeing a reduction in memory space. I assume, but am not sure, that it is for a single weight gradient calculation (of a size noted in the figure's legend and axes). The wording of Figure 3's caption is confusing: I do not understand the meaning behind "Mbv2 fine-tuned on Cifar10 at initialization." None of the networks used in the experimental results have references attached. For example, MCUNet is prominently used in Figure 4, but I couldn't find either a description of this network or a reference to it in the text. It looks likely that it is in reference [24], but this only has a passing citation in the related work section. There are some typos: - Line 223: "derives" -> "derivatives" - Line 234: the range of explained variance uses two open brackets. - Reference [48] is listed with the incorrect year: 2024 should be 2023. **Significance** There are several ways that the results would be more compelling: - Evaluation on more (difficult) tasks, like language modeling or object detection. - Application to more recent or larger networks. - More closely matching the densely trained or fine-tuned baselines. As this is initial work in this area, the results shouldn't be dismissed, but there's still a non-trivial gap between the tensor-decomposed results and what the networks are capable of learning.

Questions

1. Is there a reason that results for fine-tuning all layers were omitted? 2. Why is SVD superior to HOSVD in some results (line 288)? 3. Why does MobileNetV2's Vanilla BP results get worse when including 4 layers compared to only 2 (Table 2)? 4. What impact does the compression process have on the training speed?

Rating

7

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors have addressed the limitations (in the appendix).

Summary

Tensor low-rank decomposition is used to compress the backpropagation process of full connection and convolution in neural networks. The main process is to decompose input activation tensors into Tucker structures using HOSVD algorithm and truncate subtensors at each mode. Experimental result find an efficient trade-off between the desired explained variance and compression rates.

Strengths

(1) Compute the approximated weight derivatives without reconstructing the activations, through the successive computation of simpler convolutions. (2) The experiments on trade-off between the desired explained variance and compression rates are sufficient. (3) Experimental results obtained on mainstream architectures and tasks show that it has a Pareto advantage over other comparison schemes in terms of the trade-off between accuracy and memory footprint.

Weaknesses

(1) There are a lot of confusing expressions on the formula. (2) In formula 8, How to distinguish $U^{(k_j)}$s when the size retained after truncation on two modes is the same? (3) The symbol $]$ in formula 11 is in the wrong position, and $\frac{\partial A_{i+1}}{\partial W_i}$ should be $\frac{\partial L}{\partial A_{i+1}}$? (4) In Sec3.4 $\frac{\partial L}{\partial A_{i+1}}=\Delta Y$, but in appendix A.2 it expressed as \Delta O.

Questions

(1) In formula 12, how to use the discrete Fourier transform to get $\tilde{I} =\varepsilon I$? What does $uI$in the denominator mean? (2) In formula 16, please give the definitions of $d, h’,w’$. (3) The methods mentioned in the related work are not compared with the proposed methods.

Rating

4

Confidence

5

Soundness

2

Presentation

2

Contribution

2

Limitations

If the contraction between matrix and tensor is replaced by 1*1 convolution, then the proposed method should also be compared with CP decomposition, tensor train decomposition and other related methods.

Start reviewer-author discussions right now

Dear reviewers, Authors submitted rebuttals, which should be visible to you now. Please read the rebuttals carefully and start discussions with the authors now. The reviewer-author discussion period will end on August 13, 2024. Since authors usually need time to prepare for their responses, your quickest response would be very appreciated. In case you had requested additional experiments / analysis and the authors provided in the rebuttal, please pay extra attention to the results. Thank you, Your AC

Comments appreciated, two remaining questions

W1: A reference-by-reference can be acceptable in cases where the later reference supersedes the former. In light of the different target domains (training vs. inference) and your claim that "With the exception of Eliassen et al.’s work which accelerates training runtime, most of these works focus on accelerating inference, in a similar way to traditional model compression," then I think an explicit reference and inclusion is warranted, so thank you for the update. W2.1: What are the units of overhead in Figure 3(c)? W2.2: I understand, now - your "gradient computation and parameter updating" includes all the operations in backprop. Thank you for the clarification. W2.3: Sounds like a good revision, thanks. W3.1: I see - for a given memory budget, HOSVD is superior to SVD. It might make this argument more clear if the SVD and HOSVD results in Table 2 resulted in either the same accuracy or the same memory consumption (or if HOSVD were superior in both metrics). Further, if there's an explanation for something the authors find "Suprising" (line 288), then including the explanation for this phenomenon will help the reader understand the behavior. Figure 4 in the submission looks like it has different data than Figure 1 in the rebuttal PDF. For example, HOSVD (red curve) at 10^2 kB peak memory has an accuracy of ~84% in Figure 4, but only ~66% in Figure 1. Is this a different experiment, or has something else changed? The Gradient Filter curves appear unchanged. W3.2: I see, thank you for the explanation and update. Please note in tables and figures where the data presented is from another source or your own implementation. W3.3: I cannot see in [49] where a model trained with the ImageNet1K is applied to a CIFAR data set; I see different models trained for either data set, but perhaps this detail is omitted since it's so common? Thank you for pointing out the transfer learning application of ImageNet1K->CIFAR10/100 in [2] and [25]. W3.4: Thank you for these additions. W4.1: Thank you for the confirmation, and the proposed revision looks great. W4.2: Thank you for the confirmation. W4.3: I see, so this reflects one input sample from CIFAR10 (on a network pre-trained with ImageNet1K?). Embedded above are two remaining questions: 1) What are the units of overhead in Figure 3(c)? 2) Why does rebuttal Figure 1 differ so much from submission Figure 4?