We consider the problem of a training data proof, where a data creator or owner wants to demonstrate to a third party that some machine learning model was trained on their data. Training data proofs play a key role in recent lawsuits against foundation models trained on Web-scale data. Many prior works suggest to instantiate training data proofs using membership inference attacks. We argue that this approach is statistically unsound: to provide convincing evidence, the data creator needs to demonstrate that their attack has a low false positive rate, i.e., that the attack's output is unlikely under the null hypothesis that the model was not trained on the target data. Yet, sampling from this null hypothesis is impossible, as we do not know the exact contents of the training set, nor can we (efficiently) retrain a large foundation model. We conclude by offering three paths forward, by showing that membership inference on special canary data, watermarked training data, and data extraction attacks can be used to create sound training data proofs.