PowerPM: Foundation Model for Power Systems

Summary

The paper introduces the PowerPM: Foundation Model for Power Systems, which is designed to address the challenges of learning a generic representation of electricity time series (ETS) data in power systems. The model incorporates a temporal encoder and a hierarchical encoder to effectively capture the complex hierarchical structure and temporal dependencies present in ETS data. Additionally, PowerPM utilizes a self-supervised pre-training framework that includes masked ETS modeling and dual-view contrastive learning to enhance its ability to capture temporal dependencies and discrepancies across ETS windows. Overall, the contributions of the paper lie in the development of a sophisticated model that can accurately represent and analyze ETS data in power systems, offering valuable insights for various applications in the field.

Strengths

### Strengths Assessment: 1. **Originality:** - The paper demonstrates originality in its approach to addressing the challenges of modeling electricity time series data in power systems. The combination of a temporal encoder and hierarchical encoder, along with the self-supervised pre-training framework, showcases innovative thinking in capturing temporal dependencies and hierarchical correlations simultaneously. - The use of masked ETS modeling and dual-view contrastive learning in the pre-training stage adds a novel dimension to the model's ability to learn universal representations from ETS data. 2. **Quality:** - The quality of the paper is evident in the thorough description of the PowerPM model, its components, and the experimental results obtained. The model's deployment in real-world scenarios and the achievement of state-of-the-art performance on diverse downstream tasks within private and public datasets reflect the high quality of the research. - The incorporation of extensive experiments, ablation studies, and few-shot experiments provides a robust evaluation of the model's effectiveness and generalization ability across various tasks and domains. 3. **Clarity:** - The paper is well-structured and clearly articulates the motivation, methodology, and results of the PowerPM model. The descriptions of the temporal encoder, hierarchical encoder, and the self-supervised pre-training framework are presented in a coherent manner, making it easy for readers to understand the technical aspects of the model. - The inclusion of figures, tables, and detailed explanations aids in clarifying complex concepts such as the model analysis, ablation study, and model scale evaluation. 4. **Significance:** - The significance of the paper lies in its contribution to advancing the modeling of electricity time series data in power systems. By introducing the PowerPM model, which effectively captures temporal dependencies and hierarchical correlations, the research offers a valuable tool for enhancing economic efficiency and promoting low-carbon principles in power systems. - The model's superior performance on diverse downstream tasks, its generalization ability across different datasets, and the economic benefits generated in real-world deployments underscore the practical significance of the research in the field of power systems modeling. Overall, the paper excels in originality, quality, clarity, and significance, making a substantial contribution to the domain of electricity time series data modeling in power systems.

Weaknesses

### Weaknesses Assessment: 1. **Limited Comparison with State-of-the-Art Models:** - While the paper highlights the superior performance of PowerPM over baseline models, there is a lack of comparison with the most recent state-of-the-art models in the field of electricity time series data analysis. Including comparisons with cutting-edge models could provide a more comprehensive evaluation of PowerPM's performance [T1]. 2. **Insufficient Discussion on Model Interpretability:** - The paper could benefit from a more in-depth discussion on the interpretability of the PowerPM model. Providing insights into how the model captures and represents temporal dependencies and hierarchical correlations in ETS data could enhance the understanding of its inner workings and decision-making processes. 3. **Limited Exploration of Hyperparameters and Sensitivity Analysis:** - The paper lacks a detailed exploration of hyperparameters and sensitivity analysis for the PowerPM model. Including a thorough investigation of the impact of hyperparameters on model performance and conducting sensitivity analysis could provide valuable insights into the robustness and stability of the model across different settings. 4. **Scalability and Efficiency Considerations:** - The paper could further address scalability and efficiency considerations of the PowerPM model, especially when applied to large-scale datasets or real-time applications. Discussing the computational requirements, training time, and potential bottlenecks in scaling the model could help in understanding its practical feasibility in industrial settings. 5. **Limited Discussion on Ethical and Societal Implications:** - The paper could expand its discussion to include ethical and societal implications of deploying the PowerPM model in real-world power systems. Addressing issues related to data privacy, fairness, and potential biases in the model's predictions could enhance the overall impact and relevance of the research in a broader context. ### Suggestions for Improvement: 1. **Incorporate Comparison with State-of-the-Art Models:** - Conduct a thorough comparison with the latest state-of-the-art models in electricity time series analysis to provide a more comprehensive evaluation of PowerPM's performance and highlight its competitive advantages. 2. **Enhance Model Interpretability Analysis:** - Include a section dedicated to explaining how the PowerPM model interprets and processes ETS data, shedding light on its decision-making processes and enhancing the transparency of the model. 3. **Conduct Hyperparameter Tuning and Sensitivity Analysis:** - Perform a detailed hyperparameter tuning process and sensitivity analysis to understand the impact of key parameters on model performance and ensure robustness across different scenarios. 4. **Address Scalability and Efficiency Concerns:** - Discuss the scalability and efficiency aspects of the PowerPM model, including computational requirements, training time optimization strategies, and considerations for real-time deployment in power systems. 5. **Expand Discussion on Ethical and Societal Implications:** - Include a section on the ethical and societal implications of deploying the PowerPM model, addressing issues of fairness, bias, and privacy to ensure responsible and ethical use of the model in practical applications.

Questions

### Questions and Suggestions for the Authors: 1. **Clarification on Model Interpretability:** - Can the authors provide more insights into how the PowerPM model interprets and captures temporal dependencies and hierarchical correlations in electricity time series data? Understanding the interpretability of the model can enhance its transparency and trustworthiness in real-world applications. 2. **Explanation on Hyperparameter Selection:** - Could the authors elaborate on the rationale behind the selection of specific hyperparameters for the PowerPM model? Providing details on the hyperparameter tuning process and its impact on model performance would offer valuable insights into the model's robustness. 3. **Discussion on Generalization to Unseen Domains:** - How does the PowerPM model generalize to unseen domains or datasets outside the ones used in the experiments? Understanding the model's ability to adapt to new data distributions and scenarios is crucial for assessing its practical utility in diverse real-world applications. 4. **Scalability and Deployment Considerations:** - What are the scalability considerations for deploying the PowerPM model in large-scale power systems? How does the model handle real-time data processing and what are the potential challenges in scaling it for industrial applications? 5. **Ethical and Fairness Implications:** - Have the authors considered the ethical implications of using the PowerPM model in power systems, particularly in terms of data privacy, fairness, and potential biases? Addressing these concerns can ensure responsible and unbiased deployment of the model in practice. 6. **Future Directions and Extensions:** - Are there any plans to extend the PowerPM model or explore new research directions based on the current findings? Discussing potential future developments or applications of the model can provide insights into the ongoing research trajectory in the field of power systems modeling. 7. **Comparison with Latest Research:** - Have the authors considered comparing the PowerPM model with the most recent state-of-the-art models in electricity time series analysis? Including such comparisons can strengthen the paper's contribution and highlight the advancements made by PowerPM in the field. Addressing these questions and suggestions can enhance the clarity, robustness, and applicability of the PowerPM model, providing valuable insights for both the research community and practical stakeholders in the power systems domain.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

### Assessment of Limitations and Societal Impact Considerations: 1. **Limitations Addressed:** - The authors have discussed the limitations of their work in Appendix F, which is a positive step towards transparency and acknowledging the constraints of the research. This demonstrates a willingness to reflect on the scope and potential weaknesses of the study. 2. **Societal Impact Considerations:** - The paper lacks a comprehensive discussion on the potential negative societal impacts of deploying the PowerPM model in real-world power systems. While some ethical considerations are mentioned, a more thorough exploration of broader societal implications is needed. ### Suggestions for Improvement: 1. **Enhanced Limitations Section:** - The authors could consider expanding the limitations section in the main body of the paper rather than relegating it to an appendix. This would ensure that readers engage with the limitations more prominently and understand the boundaries of the research. 2. **Broader Societal Impact Analysis:** - To address potential negative societal impacts, the authors should conduct a detailed analysis of how the deployment of the PowerPM model could affect various stakeholders, including issues related to fairness, privacy, and bias. Providing mitigation strategies for these impacts would demonstrate a proactive approach to responsible research. 3. **Incorporate Ethical Frameworks:** - Utilize established ethical frameworks or guidelines to evaluate the ethical implications of the PowerPM model in power systems. This could involve considering principles such as fairness, accountability, transparency, and privacy to ensure ethical deployment and minimize negative societal consequences. 4. **Engage with Stakeholders:** - Engage with relevant stakeholders, such as power system operators, regulators, and community representatives, to gather diverse perspectives on the potential societal impacts of implementing the PowerPM model. Incorporating feedback from stakeholders can enrich the discussion on societal implications. 5. **Mitigation Strategies:** - Propose concrete mitigation strategies for addressing any identified negative societal impacts of the PowerPM model. This could include mechanisms for monitoring model performance, ensuring fairness in decision-making processes, and enhancing transparency in model deployment. By addressing these suggestions, the authors can strengthen the ethical foundation of their research, demonstrate a commitment to responsible innovation, and contribute to a more comprehensive understanding of the societal implications of deploying advanced models like PowerPM in practical settings.

Summary

A pre-trained LLM named PowerPM is proposed for modeling Electricity Time Series (ETS) data. PowerPM combines a temporal encoder for capturing temporal patterns and a hierarchical encoder for understanding hierarchical correlations. PowerPM also employs a self-supervised pre-training strategy that incorporates masked ETS modeling and dual-view contrastive learning, enhancing the model's ability to learn from the intricacies of ETS data. PowerPM is specifically designed for power systems, especially for the demand side, making it unique among other general models. This paper claims that deployment of PowerPM in Zhejiang Power Grid had led to significant economic benefits, showing the practical value of the model in real-world settings.

Strengths

This article proposes a model for load time series forecasting on multi-hierarchy demand side and pre-trains it on a large amount of real data. It shows good performance and strong generalizability in experiments. Furthermore, the model is applicable in the real world. For multiple pre-training challenges on multi-hierarchy load time series data, this method integrates corresponding solutions together.

Weaknesses

Since the pre-training dataset is comprehensive, the work lacks an analysis on the effectiveness provided by the dataset. In other words, readers may not understand if the dataset works a lot or the proposed method. The writing and explanation in this paper need to be improved. This paper involves extensive training, but details are largely missing from the current paper.

Questions

The writing lacks sufficient detail and explanation, resulting in quite some questions. For example, what are the computational resources used? The provided materials do not include codes related to the proposed pre-training method. No information about whether the pre-trained model or the pre-training dataset will be released to make the work transparent.

Rating

3

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

This works involves load forecasting for power system and individual users in real-world applications, which would affect power system operation, its safety, and societal impact as a whole. These should be discussed.

Summary

This paper proposed a foundation model, PowerPM, to model electricity time series data, providing a large-scale off-the-shelf model for power systems. PowerPM consists of a temporal encoder and a hierarchical encoder with a self-supervised pretraining framework. The authors have tested PowerPM in Demand-side Management, Grid Stability, and Consumer Behavior Analysis, showing their advantages against other common time series LLM models, such as GPT4, TimeLLM, etc.

Strengths

1. They have tested their work with real data and showing competitive results; 2. It is the first model that considers temporal dependency and hierarchical dependency simultaneously; 3. Comprehensive model analyses.

Weaknesses

The writing should be improved, several places are missing space between words. It could be improved or make the paper more convincing if the author could compare their framework with some state-of-art techniques. However, it is understandable that the research question/application objectives are more comprehensive in this paper.

Questions

1. For power systems, have you considered the power systems constraints, such as power network structures, physic laws of Kirchhoff's circuit laws? 2. With renewable energy sources, have you considered of including weather time series data along with ETS data?

Rating

7

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors has discussed the limitations of their work.

Summary

This paper learns a generic representation of electricity time series data. The proposed PowerPM model is composed of a temporal encoder and a hierarchical encoder.

Strengths

The results shown in the table exhibit good numerical results of the proposed model.

Weaknesses

It is unclear where the performance gain comes from. And the proposed architecture is a combination of several previous techniques, so it is hard to identify the technical contributions. Moreover, it is not well motivated on the usage of developed models for many power system tasks. As far as the reviewer is aware of, current statistical methods and machine learning-based methods can already give good load forecasting results. The real challenges come from noisy data inputs, incomplete number of features, or small-region level forecasting. Yet this paper only looks into a very general forecasting problem, and does not reveal the key challenges and special properties of energy forecasting. The grid stability notion is quite misleading. In standard power system tasks, stability notion is with respect to either frequency or voltage, and is more related to the transient states of the systems. While in this paper, the stability is more like a phenomenon or behavior analysis. In addition, the paper needs to discuss the computational costs related to both training and inference, as utility and grid operators normally are not equipped with enough computation capabilities to use large foundation models.

Questions

Can the authors elaborate the settings of the freeze version? What are the effects of contrastive learning? The figure 4 looks like a software snapshot. Can the authors explain the major information conveyed in this figure?

Rating

3

Confidence

3

Soundness

2

Presentation

2

Contribution

1

Limitations

The paper describes techniques limited to energy forecasting tasks, and it is not clear if the methods can be applied to other tasks.

Thank you

We wholeheartedly appreciate and deeply cherish your efforts in helping us to strengthen the paper and your recognition of our work.

Clarification on Ethical Approval

Thank you for your valuable feedback. We appreciate your attention to detail and ethical considerations. We would like to address your concerns and provide further information regarding the dataset and model application. ## **1. Privacy protection** * **Consent Process in dataset collecation process**: State Grid adopts the mode of questionnaire survey to obtain users' labels to ensure that users have the right to know about this research. The consent process was conducted by **State Grid Corporation of China in Zhejiang province** using a standardized consent form approved by the IRB. Users were informed about the purpose of the study, the data that would be collected, and their rights as research participants. * **How privacy is protected in the dataset**: The information data and the label we receive about personal privacy is anonymous, which is provided by State Grid Corporation of China in Zhejiang province. The specific encryption process is specially handled by the Data privacy Department of the State Grid Corporation of China in Zhejiang province, and we and the model only have access to the encrypted data. ## **2. Bias mitigation** To mitigate bias, we take the following approach to data set processing: * **Inclusive Sampling**: Pretrain datasets include data from underrepresented groups to ensure that the dataset is as representative as possible of the entire user population. * **Geographical Diversity**: Pretrain datasets from various geographical locations to capture different usage patterns that may be influenced by local climate, culture, and infrastructure. * **Temporal Diversity**: Include data collected over different times of the day, week, and year to capture seasonal and daily usage variations. These method ensures that the pretrian datasets includes a variety of user behaviors and scenarios helps in training PowerPM that perform well across different populations. This is critical in power systems where users can vary widely in their consumption patterns, economic status, and access to technology. ## **3. Environmental Impact and Human Rights Impact** * **Environmental Impact**: It is worth noting that our model only provides an analysis and insight on resource scheduling and utilization. Specific policies need to be formulated by the decision-making departments of State Grid Corporation of China in Zhejiang province, and the release of its policies needs to be reviewed and approved by the State Environmental Protection Administration and the State Energy Administration, and its impact on environmental resources is guaranteed to a certain extent. * **Human Rights Impact**: State Grid Corporation of China in Zhejiang province will implment a system where human oversight is involved in the decision-making process, especially when it comes to sensitive classifications. For the detection of electricity theft, there will be a special staff to inspect the line and electricity meter on the spot. For the detection of elderly living alone, it is only used as a reference for community care statistics, and relevant government departments will also go to the scene to confirm the results. Most importantly, all above results will be fed back into the model, which is constantly optimized to ensure accurate detection of sensitive classifications. ## **4. Security Measures** Concurrently, we have formulated a plan for the responsible disclosure of any security vulnerabilities identified during the research or deployment of PowerPM. We are committed to adhering to the best practices in cybersecurity, engaging in transparent communication with security researchers, relevant stakeholders, and potentially affected users. Our process ensures that the scope of information disclosure is limited until the vulnerabilities are addressed, preventing malicious exploitation. We remain vigilant in monitoring and assessing the security of the model, ensuring that all security measures are continuously strengthened and updated over time. ## **5. Transparency and Reproducibility** More detailed information about the methodology can be found in Appendix D, and to ensure the reproducibility, we plan to promote the release of our dataset in the following three stages as shown in rebuttal for **Reviewer W2Rh** **Q3: Our planned steps for the dataset and pre-trained model release** ## **6. Ethical Review Boards** The data collection and experiments conducted in our work have been approved by the Institutional Review Board (IRB) and passed ethical review. The data has been effectively licensed by State Grid Corporation of China in Zhejiang province, all user-related information is encrypted, and all downstream tasks are only used for State Grid supply planning and internal analysis.

Clarification on Ethical Approval

Thank you for your valuable feedback. We appreciate your attention to detail and ethical considerations. We would like to address your concerns and provide further information regarding the dataset and model application. ## **1. Users' consent**: We ensure that all users agree to share data, and the user's personal information is encrypted through **State Grid Corporation of China in Zhejiang province** : Informed consent was obtained from all electricity users before their data was used for research purposes. State Grid adopts the mode of questionnaire survey to obtain users' labels to ensure that users have the right to know about this research. The consent process was conducted by **State Grid Corporation of China in Zhejiang province** using a standardized consent form approved by the IRB. Users were informed about the purpose of the study, the data that would be collected, and their rights as research participants. ## **2.Research involving human subjects / Safety and security** In task prediction, the prediction result is only used to analyze the user's behavior. Although it will affect the electricity deployment and dispatching of the power grid, it is worth noting that the influence of the electricity purchasing behavior of the power grid is only used to reduce the waste of electricity procurement. In order to ensure the electricity use of users, the power grid needs to continuously purchase more electricity than the users in the region can use. Our prediction is only to reduce the excess part, so as to achieve energy saving, emission reduction and efficiency improvement. If the prediction deviation leads to the loss of electricity use by users, the grid will have a complete power dispatching mechanism to ensure people's power consumption. These will avoi the dunfair availability of power. ## **3. How the users' data was deanonymized** We are very sorry that the data we have access to has been anonymized, and all our tasks are based on anonymized data, and the data anonymization process is carried out by **State Grid Corporation of China in Zhejiang province**, and we have no right to know. Thanks for your reminder, we will update above information in our paper.