ReproducedPapers.org: Openly teaching and structuring machine learning reproducibility

We present this http URL: an open online repository for teaching and structuring machine learning reproducibility. We evaluate doing a reproduction project among students and the added value of an online reproduction repository among AI researchers. We use anonymous self-assessment surveys and obtained 144 responses. Results suggest that students who do a reproduction project place more value on scientific reproductions and become more critical thinkers. Students and AI researchers agree that our online reproduction repository is valuable.

cs.CY cs.CV Computer Science Education

Paper

References (31)

01Black Magic in Deep Learning: How Human Skill Impacts Network TrainingK. Anand, Ziqi Wang, M. Loog et al.2020 · British Machine Vision Conference · 17 citations In Library

02dagger: A Python Framework for Reproducible Machine Learning Experiment OrchestrationMichela Paganini, Jessica Zosa Forde2020 · arXiv.org · 8 citations In Library

03Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOLogan Engstrom, Andrew Ilyas, Shibani Santurkar et al.2020 · arXiv.org · 335 citations In Library

04Code replicability in computer graphicsNicolas Bonneel, D. Coeurjolly, Julie Digne et al.2020 · ACM Transactions on Graphics · 29 citations

05A Metric Learning Reality CheckKevin Musgrave, Serge J. Belongie, Ser-Nam Lim2020 · European Conference on Computer Vision · 539 citations In Library

06A Step Toward Quantifying Independently Reproducible Machine Learning ResearchEdward Raff2019 · Neural Information Processing Systems · 165 citations In Library

07Are we really making much progress? A worrying analysis of recent neural recommendation approachesMaurizio Ferrari Dacrema, P. Cremonesi, D. Jannach2019 · ACM Conference on Recommender Systems · 688 citations In Library

08We need to talk about standard splitsKyle Gorman, Steven Bedrick2019 · Annual Meeting of the Association for Computational Linguistics · 131 citations

09Research for practiceZachary Chase Lipton, J. Steinhardt2019 · Communications of the ACM · 9 citations

10Do ImageNet Classifiers Generalize to ImageNet?B. Recht, R. Roelofs, Ludwig Schmidt et al.2019 · International Conference on Machine Learning · 2.3k citations In Library

11The Neural Hype and Comparisons Against Weak BaselinesJimmy J. Lin2019 · SIGIR Forum · 154 citations

12The Diagrammatic AI Language (DIAL): Version 0.1G. Marshall, A. Freitas2018 · arXiv.org · 3 citations In Library

Scroll for more · 19 remaining

Similar papers

Reproducibility as a Mechanism for Teaching Fairness, Accountability, Confidentiality, and Transparency in Artificial IntelligenceAna Lučić, Maurits Bleeker, Sami Jullien et al.2021 · arXiv (Cornell University) · 15 citations In Library

Reproducibility as a Mechanism for Teaching Fairness, Accountability, Confidentiality, and Transparency in Artificial IntelligenceAna Lučić, Maurits Bleeker, Sami Jullien et al.2022 · Proceedings of the AAAI Conference on Artificial Intelligence In Library

Survey Research With AI2026 · Advances in computational intelligence and robotics book series In Library

Detailed Implementation of a Reproducible Machine Learning-Enabled WorkflowKenneth Schackart, Heidi J. Imker, Charles E. Cook2024 · Data Science Journal · 5 citations In Library

Black Box or Open Science? Assessing Reproducibility-Related Documentation in AI ResearchFlorian Koenigstorfer, Armin Haberl, Dominik Kowald et al.2024 · Proceedings of the ... Annual Hawaii International Conference on System Sciences/Proceedings of the Annual Hawaii International Conference on System Sciences In Library

AutoReproduce: Automatic AI Experiment Reproduction with Paper LineageXuanle Zhao, Zilin Sang, Yuxuan Li et al.2025 · Annual Meeting of the Association for Computational Linguistics · 20 citations In Library

Reproducibility of Machine Learning: Terminology, Recommendations and Open IssuesRiccardo Albertoni, Sara Colantonio, Piotr Skrzypczyński et al.2023 · arXiv (Cornell University) · 17 citations In Library

Module 8: Reproducibility primer for AI-driven researchSimone Kopeinik, Dominik Kowald, Tony Ross-Hellauer2023 · Open MIND In Library

Module 8: Reproducibility primer for AI-driven researchStefania Amodeo2026 · OSF Preprints (OSF Preprints) In Library

Mission Reproducibility: An Investigation on Reproducibility Issues in Machine Learning and Information Retrieval ResearchMoritz Staudinger, B. Kern, Tomasz Miksa et al.2024 · IEEE International Conference on e-Science · 2 citations In Library

Source of record

arXiv

44B

Welcome.

This panel sticks with you. Pick anything below and it opens right beside it, so you can dig through 60,000-plus records without ever losing your spot here.

Travel 44B

LibraryPapers, policy, standards, statute LabsEvery organization building AI ModelsIntelligence, price, and speed BenchmarksThe evaluation catalog IncidentsAI failures and harms SearchOne field across everything DashboardThe state of AI accountability in NY 44B RegistryThe Art. 44-B compliance portal