Optimistic Verifiable Training by Controlling Hardware Nondeterminism

Summary

The authors present a novel approach to achieving identical results in model training across different GPU types and introduce a verifiable training scheme. To achieve this, they: - proposed a technique for two parties training the same model on different GPU types to achieve identical results by sharing rounding decisions. - presented a verifiable training scheme which uses a Merkle tree to store model weights for efficient comparison between a trainer and an auditor. - conducted experiments demonstrating the scalability of the approach to ResNet-50 and GPT-2 across three different NVIDIA GPU architectures (A40, Titan XP, RTX 2080 Ti). - proposed methods to reduce the storage cost via efficient encoding of rounding logs and an adaptive threshold mechanism to minimize the amount of rounding decisions logged. - compared their approach with existing methods, including proof-based systems, and the results show that their approach is storage and time-efficient.

Strengths

- The paper introduces a novel method for achieving identical training results across different GPU types by sharing rounding decisions, which could significantly enhance reproducibility in machine learning. The use of a verifiable training scheme based on a well-established verification game adds a layer of trust and transparency to the training process, making it more reliable for sensitive applications. The methods to reduce storage costs via efficient encoding of rounding logs and an adaptive threshold mechanism address practical concerns related to resource usage. This in itself is a huge contribution. - The experiments using foundation models like ResNet-50 and GPT-2 across multiple GPU architectures showcases the robustness and practicality of the proposed approach. The paper also provides thorough comparisons with existing methods, highlighting the improved storage and time efficiency of the proposed approach, which strengthens the case for its adoption.

Weaknesses

- The authors implemented their verifiable training method entirely on top of the PyTorch framework, using torch version 1.13.1. Given that PyTorch has since released version 2.3.1, there may be compatibility issues or inaccuracies if the method is implemented on the updated version, i.e., using an older version could affect the claim that their method achieves perfect training replication of the two used models, if someone else tries to implement their approach using the updated version, or another framework. - While the paper includes a comparison with existing methods, the authors assume certain metrics from the baselines due to the unavailability of specific information. This may affect the fairness of the comparisons, especially as the reported improvements might only be valid for the given scenarios (and would be different for other scenarios).

Questions

Please attend to weaknesses above.

Rating

6

Confidence

3

Soundness

4

Presentation

4

Contribution

3

Limitations

The authors present the limitation of their work in the limitation section.

Summary

Machine learning is increasingly compute power consuming, and as a result, clients may delegate the training to external parties with high computation power. A challenge is how to verify the external parties is training the model as promised. This work examines a way of auditing by having a trusted third party to verify the outcome. In particular, it investigates the impact of discrepancy in hardware to the third party's auditing outcome.

Strengths

The overall approach of the paper is sound and the assessments are refreshing.

Weaknesses

My main concern of the work is the motivation. In particular, while hardware nondeterminism is a problem, is it really the bottleneck to such auditing? The existence of a trusted third-party is not very obvious to me already: if it exists, we can just delegate the training to this third party; if it doesn't have the compute power to verify all training process of the trainer, then the trainer can still acts suspiciously when not audited in the round. The setting of the work is hardly convincing to me. ============= After Author Response ================================== I appreciate the response from the authors. The response cleared some of my doubts over the setting. Coming from a ML security/privacy background, I still don't buy the existence of a trusted third party. However, I appreciate the contribution of this auditing process as a first step in a non adversarial environment. Therefore, I've raised my score accordingly.

Questions

As mentioned in the weakness box, the setting really confuses me. Please elaborate/justify the scenario in the author response. Thanks! The technical contribution of mitigating the gap due to hardware difference is interesting. I feel such a technique should have a more impactful and realistic corresponding setting in ML.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are adequately addressed.

Summary

This paper studies the problem of verifying the correctness of the training process of model. In particular, the user who lacks sufficient resources pays a service provider with sufficient resources to train models. Then a trusted third-party auditor will check whether the training process is legit. The proposed approach rounds after intermediate computation steps, and stores rounding decisions based on an adaptive thresholding procedure, to successfully control for nondeterminism among different GPUs and also saves the storage space. Empirical results show that the proposed approach can run more efficiently than the previous baselines while also using smaller storage space, which makes it valuable for verifying larger models.

Strengths

1. The problem of verifying the correctness of the model training process is an important problem and leveraging a third-party auditor can be one possible way to scale these techniques to larger models. 2. The proposed approach outperforms the existing baselines both in terms of verification efficiency and usage of storage space.

Weaknesses

There is a lack of description on the relation and difference of this work and Teutsch & Reitwießner (2019). I think the authors will benefit by clarifying the difference and as well as emphasizing their novelty.

Questions

There is no direct outstanding questions, but I would encourage the authors to focus on the writing to highlight the contribution more explicitly.

Rating

7

Confidence

1

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are well addressed in this work.

Thanks for the response

I've updated my score accordingly.