Tools for Verifying Neural Models' Training Data

Summary

This paper proposes a protocol for verifying that a model trainer submits data and learned weights, and a verifier checks whether the weights are correctly learned from the submitted data. Such a protocol is useful for trustworthy AI. The paper defines the problem of Proof-of-Trainind-Data (PoTD). It is inspired by the previous work on "Proof-of-Learning," but the setting of PoTD requires more. The paper gives a formal definition of PoTD protocol and argues that the protocol needs to satisfy some conditions to achieve the guarantee in a practical setting.

Strengths

The motivation of the paper is interesting. The proposed PoTD protocol seems reasonable.

Weaknesses

1. The paper does not explain the motivation why we need the PoTD protocol and its impact. The paper says that some attacks can be treated by existing Proof-of-Learning methods. It seems trivial since PoTD poses stronger requirements, as written in the paper. The paper should discuss the problems of existing PoL methods and how the proposed protocol can solve them. 2. The only attack that the existing method cannot deal with but the proposed method can is a data subtraction attack. How important to deal with such an attack is not explained enough. Moreover, experimental results show how the proposed method performs but do not compare with baseline methods. Therefore, experiments are not enough to show the effectiveness of the proposed methods. 3. The presentation of the paper is hard to follow. The paper seems to consist of fragments of texts relationships between them are unclear. For example, There is a formal definition of PoTD protocol in section 2, but the definition is never mentioned in the subsequent sections. Therefore, it is hard to judge whether the definition is reasonable or not. The memorization heuristic introduced in Section 3.2 seems overly complex. If we use $-L(d, W)$ instead of M(d, W) in (3), then the values of PBQ and FBQ would not change since the first term of (2) does not depend on data d and has no effect when evaluating $\Delta_M(d^\prime, W) > \Delta_M(d, W)$.

Questions

n/a

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

The paper mentions to limitations in the end of the paper.

Summary

The authors propose Proof of Training Data (PoTD), a variant of Proof-of-Learning (PoL) protocols that focuses on training set attacks, rather than the training algorithm itself. A valid PoTD protocol should be able to, at least in theory, spot when a machine learning model has been trained on a different training set than the one declared by the learner. As in PoL, the learner is required to provide a full transcript of the training process, including training data, code and intermediate checkpoints. Unfortunately, the task of verifying a training transcript is as computationally intensive as re-training the model from scratch. For this reason, the authors propose several heuristic strategies for PoTD, which rely on the fact that stochastic gradient ascent tends to first memorize and then forget the data it observes in each batch.

Strengths

From the methodology standpoint, I truly appreciate the memorization heuristics proposed in the papers. In particular, they seems to be able to defend against a large number of different threat models.

Weaknesses

The PoTD protocol proposed by the authors have considerable overlap with the existing PoL proposals. I am inclined to see it as a variant of these existing efforts, rather than a novel, independent idea. Furthermore, the defenses proposed in the paper are heuristics and have been tested on large language models only. The authors mention that their techniques may work differently on other neural architectures, learning tasks and training procedures. Therefore, I believe more experimentation is needed to confirm their usefulness. Additionally, the PoTD protocol requires the learner to disclose their complete learning process. Thus, there is no way to protect the intellectual property of the learner. The authors are honest about this limitation, and claim it will be addressed as future work. However, I feel this makes the paper weaker. The data addition attack presented in lines 256-267 seems the most interesting scenario to me. It is unfortunate that the heuristic technique proposed by the authors cannot defend against it. The writing style of the paper could be improved. I report here some specific examples. Line 6, "and flag if the model specific harmful or beneficial data sources" is not a syntactically-correct English sentence. Line 30, the authors start discussing how to solve proof-of-training-data before giving a precise enough definition that the reader can follow. Line 87, c2 is missing from the definition of V (compare with Line 82). Also, why is the probability taken over c1, when c1 does not appear in any of the terms? Line 172-173, please number all equations. Section 3.3 is very dense and many important details are left for future work. Footnotes 4-9 occupy almost a quarter of the page and contain important information. I would prefer having them merged with the main text. Line 225, "on trained" should be "trained on". Section 5 introduces new concepts, new attacks, new defenses, new notation. Since this happens so late in the paper, it ends up being a bit overwhelming. Why not explicitly organise the whole paper by threat model?

Questions

Would it be possible to bypass the memorization heuristic by changing both the model and the data? Theoretical work has shown that only a small portion of a large neural network is important for achieving good predictive performance (lottery tickets). Once a learner has trained a good model on dataset D*, it should be possible to add (a large number of) redundant neurons and retrain those on D. Would this be a valid attack on PoTD? The Equation between lines 172 and 173 is not obvious at first sight, and should be clarified. Also, please number all equation in the paper.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

I commend the authors for beign upfront about the limitations of their work. All my concerns have been listed above.

Summary

The paper presents a novel protocol called Proof-of-Training-Data, which a third party auditor can verify the data used to train a model. Here, the auditor will require training data, training code, and intermediate checkpoints. Experiments on two language models have demonstrated that known attacks from the Proof-of-Learning literature can be caught by this new protocol.

Strengths

This paper attempts to tackle an important security problem on trained neural network models. The proposed heuristics (i.e., memorization-based tests) is appealing and can efficiently catch spoofed checkpoints using a small amount of data. The paper is adequately structured and solid experiments have been carried out to empirically justify the effectiveness of the proposed protocol.

Weaknesses

The concept of Proof-of-Learning has been well studied. Although the authors have discussed various Proof-of-Learning literature in the related work and the experiments section, it is still not immediately clear to me why we need this brand-new protocol (Proof-of-Training-Data). If I understand correctly, the authors are attempting to solve an even harder problem where the adversaries can have more computing power.

Questions

- Line 77, could there be a comparison between the formal formulation of Proof-of-Learning and Proof-of-Training-Data? - Section 4, as most of the Proof-of-Learning experiments have been done on image datasets like CIFAR, I am curious if PoTD could perform similarly well on those image datasets.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors have adequately addressed the limitations.

Summary

This paper describes techniques and tools that can be used for verifying the "provenance" of large neural models, to evaluate their risks. These techniques and tools are part of "protocols" used by a model trainer to convince a "verifier" that the training data was used to produce the model parameters. The authors show experimentally that their prescribed procedures can catch a variety of known attacks from the "proof-of-learning" literature.

Strengths

* The paper addresses an increasingly important problem, as large neural models are becoming very popular, and advances practical techniques that can be used by regulators to check the provenance of large models. * The authors present convincing evaluation using GPT-2, demonstrating that the proposed procedures are effective in catching a variety of attacks, such as glue-ing and interpolation as well as data addition and subtraction.

Weaknesses

* I find that the use of "proofs" in the title and throughout the paper is misleading as the authors do not present techniques that amount to an actual proof. * I think it is great that the techniques presented by the authors are practical and effective wrt several attacks but I wonder if this is enough for regulators. I mean they would possibly need stronger guarantees for such techniques. * It is unclear to me how the verification strategies presented in section 3 relate to definition 1. The authors should work on adding a theorem that clearly states that their strategies achieve the desired properties, i.e., the verifier accepts/rejects true winesses/spoofs with the desired probabilities. * The technical contribution beyond "proof-of-learning" is unclear.

Questions

Please see above.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

In conclusion this is very interesting work but may be too preliminary for publication.

Thank you for the response

Thank you for the response. The claims in the response are all strong, and I understand that the proposed method has many interesting properties compared with PoL methods. My question is, "Why did the authors not put these things in the original paper?" I think the paper's introduction does not clearly show the motivation of the PoTD. It says PoTD is "inspired by PoL (line 33)", and "provide several verification strategies ... all published attacks in the PoL literature (line 37)". Moreover, the paper also says PoTd is a "stricter requirement than POL" (line 52). However, the introduction says nothing about why we need PoTD instead of PoL. These things made it hard for me to understand the position of PoTD. I will raise my score, but I still think this paper needs a major revision to improve the presentation by including the material shown in the response.

Thanks

Thanks a lot for the clarification!