ChatGPT as Research Scientist: Probing GPT's Capabilities as a Research Librarian, Research Ethicist, Data Generator and Data Predictor

Significance Though scientists widely adopt them, the promise of general-purpose AI systems to facilitate science has been largely untested. In four studies, we examine the capabilities of ChatGPT across several tasks intrinsic to the scientific process. ChatGPT is a poor (but improving) curator of scientific articles. It is a surprisingly good research ethicist, detecting violations of statistical best practices and evolving open science protocols. Its ability to simulate known results may herald useful abilities in data generation and theory building. However, the chatbot had little success predicting highly novel data, highlighting its limited ability to surmise things outside its training data. Beyond merely testing large language models, these studies produce several insights into the nature of machine intelligence.