Large language models (LLMs), including ChatGPT, can extract profitable trading signals from the sentiment in news text. However, backtesting such strategies poses a challenge because LLMs are trained on many years of data, and backtesting produces biased results if the training and backtesting periods overlap. This bias can take two forms: a look-ahead bias, in which the LLM may have specific knowledge of the stock returns that followed a news article, and a distraction effect, in which general knowledge of the companies named interferes with the measurement of a text’s sentiment. The authors investigate these sources of bias through trading strategies driven by the sentiment of financial news headlines. They compare trading performance based on the original headlines with debiased strategies in which they remove the relevant company’s identifiers from the text. In-sample (within the LLM training window), the authors find, surprisingly, that the anonymized headlines outperform, indicating that the distraction effect has a greater impact than look-ahead bias. This tendency is particularly strong for larger companies—companies about which the authors expect an LLM to have greater general knowledge. Out-of-sample, look-ahead bias is not a concern but distraction remains possible. The authors’ proposed anonymization procedure is therefore potentially useful in out-of-sample implementation, as well as for debiased backtesting.