Summary
The Paper “WATT FOR WHAT: RETHINKING DEEP LEARNING’S ENERGY-PERFORMANCE RELATIONSHIP” addresses the problem of increasing energy consumption and subsequent Carbon footprint of today’s deep learning research.
The authors introduce a novel metric, the GreenQuotientIndex (GQI), which aims to compare machine learning models based on their accuracy and energy efficiency. According to the authors, GQI penalizes models for their excessive electricity usage, with the penalty increasing logarithmically, while also rewarding models for achieving higher accuracy.
The authors then present a number of experiments to compare different deep learning architectures on computer vision tasks by using the proposed metric. They partially train various CNN and Transformer-based existing models on image classification, semantic segmentation and action recognition and track the electricity consumption with CodeCarbon, an open-source python tool.
While the topic is very important the novelty of the paper is very limited. Given that the authors did not conduct full training runs with extensive analysis of the energy consumption and corresponding predictive performance, the contribution of the paper boils down to a metric, that has several flaws (see Weaknesses).
In general I am not a fan of merging energy consumption and model predictive performance into one metric, as it smears things out. If Model A has a GQI 0.1 higher than Model B, it's unclear what this difference means in terms of their performance or efficiency. Furthermore, two models can have the same GQI, but with different values of Accuracy and Energy consumption. Depending on the application, either one would be the better choice (High model performance e.g. in security settings for face recognition vs low energy consumption for embedded devices. It would be much better if both values were to be reported individually, such that researchers can evaluate the full extend and implications of the model themselves. As claimed by current research, it is more important to attribute the same scientific value to improvements in energy efficiency as in predictive performance.
Strengths
- The topic on energy consumption and related carbon footprint of deep learning models is very timely and needs to be addressed urgently
- For Image recognition, the experimental evaluation in terms of different models is quite extensive, including different architectural components.
- The introduction is quite nicely written, though it is more adequate for a comment than a research paper and could do with a few more proofs to its claims.
Weaknesses
The proposed metric has several flaws:
- The hyperparameter ß seems useless, as its sole purpose appears to be scaling to obtain “nice” values (the authors claim the purpose of “better understanding of the values”, but given that the metric has no physical meaning and hyperparameters are chose quite arbitrary, there is no point to that, and absolute values of the metric have no meaning whatsoever. Energy scale is comparable if it is fixed for all investigated cases)
- Using the log of the energy consumption and the power of the accuracy has exactly opposite effects on the resulting metric as intended. The authors chose accuracy^a to reward high accuracies. But their own experiments demonstrate that achieving the last few percent in accuracy gain is the most costly in terms of electricity consumption. And on the other hand, scaling the electricity consumption with the log penalizes smaller, more energy efficient models much harder, than those with excessive energy consumption. For example, the GQI of ViT-H-JFT and ViT-B-scratch is roughly the same, but the energy consumption of the Former is two orders of magnitude larger, while the accuracy difference is only ~10%.
- The hyperparameters an and B are tuned solely to promote certain models (EfficientNet), because for other values, MoblieNet always achieved the highest GQI. This is highly non-scientific.
- The properties of the metric: in Sec 3.1 in the first bullet point, the authors claim the nominator to be in the range [0,inf], and directly following up in the next bullet point state that it is the power function of a number smaller then 1, directly contradicting the first statement. Moreover, the denominator cannot be 0, thereby rendering the statement on its range false as well. Furthermore the claim on the importance of the metric being non-negativeis nonsense. A metric can also be e.g. between -1 and 1 and would make sense. Since the metric has no physical meaning or units, the scale can be completely arbitrary, just like MSE for example.
The experimental evaluation:
- Only computer vision tasks based on images are investigated. In the introduction, the authors clearly discuss that current LLMs are the prime energy consumers. Hence, it would be good to include NLP tasks in the evaluation, especially with respect to the corresponding metrics, such as perplexity, BLUE, or ROGUE.Also, models whose predictive performance is not evaluated on an accuracy between 0 and 1, e.g. regression tasks with MSE/MAE
- There is no proper description of the experiments (see questions)
- While presenting the results, no attempt is made to gain insight into why specific architectures are more efficient than others.
- All experiments disregard data-parallel training (which cannot be assessed using CodeCarbon), but this is an essential part of DL training these days.
Certain assumptions (in the appendix):
- Interpolation: This is dangerous, since the number of epochs for each model can vary gravely. Even if you assume the number of epochs stated in the original papers (which are not always clearly given), this does not factor in things like early stopping. Moreover, the considered number of samples, batch size and other hyperparameters are strongly linked and can gravely affect model convergence.
- The section on hardware effects is pointless. The authors state that the substantial variance in GPU memory are mitigated by using the log of the electricity consumption. But it is well known that GPUs of newer generations (the ones with bigger memory) are also designed under energy efficiency aspects, which is likely to play a much more important role.
Presentation: Though nicely written, the graphical presentation of the paper is really poor:
- The Figures are terrible to read in print-out
- The table references and captions do not match, e.g. Table 1 is referred to as Table 5.2, Table 2 is referred to as Table 5.3, etc.
- Table layouts lack uniformity, even when their content is similar.
- In the description of Fig.2 x- and y-axis are swapped.
- The separation in Training and Test in Tables 1 and 3 is unclear
- All tables and plots match units for electricity
- Figure 1 is nonsensical, it seems like a rather half-baked comparison
Questions
Please describe the experiments more clearly:
- You do not mention the number of epochs you train models on image recognition. In the Appendix you state interpolation from just 1 to 5 epochs?
- If you did not fully train those models, how do you obtain the final accuracy?
- Most experiments, I assume, entail fine-tuning pre-trained models on the mentioned datasets? Though it is very poorly explained.
- Table 1: should 'Train' and 'Test' be over the columns 'Electricity'?
- How do you measure GFLOPs in Table 1 and 3
- It is unclear whether reported GQI is based on training or test electricity consumption