NeurIPS 2020 NLC2CMD Competition: Translating Natural Language to Bash Commands

The NLC2CMD Competition hosted at NeurIPS 2020 aimed to bring the power of\nnatural language processing to the command line. Participants were tasked with\nbuilding models that can transform descriptions of command line tasks in\nEnglish to their Bash syntax. This is a report on the competition with details\nof the task, metrics, data, attempted solutions, and lessons learned.\n

cs.CL Natural Language Processing Techniques Topic Modeling Computer Science Linguistics

Paper

References (42)

01Measuring Coding Challenge Competence With APPSDan Hendrycks, Steven Basart, Saurav Kadavath et al.2021 · NeurIPS Datasets and Benchmarks · 1.2k citations In Library

02Extracting Training Data from Large Language ModelsNicholas Carlini, Florian Tramèr, Eric Wallace et al.2020 · USENIX Security Symposium · 3.2k citations In Library

03Semantic Evaluation for Text-to-SQL with Distilled Test SuiteRuiqi Zhong, Tao Yu, D. Klein2020 · Conference on Empirical Methods in Natural Language Processing · 192 citations In Library

04Photon: A Robust Cross-Domain Text-to-SQL SystemJichuan Zeng, Xi Victoria Lin, S. Hoi et al.2020 · Annual Meeting of the Association for Computational Linguistics · 58 citations In Library

05Unsupervised Translation of Programming LanguagesM. Lachaux, Baptiste Rozière, Lowik Chanussot et al.2020 · Neural Information Processing Systems · 579 citations In Library

06CLAI: A Platform for AI Skills on the Command LineMayank Agarwal, Jorge J. Barroso, Tathagata Chakraborti et al.2020 · arXiv.org · 9 citations In Library

07Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine LearningPeter Henderson, Jie Hu, Joshua Romoff et al.2020 · arXiv.org · 729 citations In Library

08Energy Usage Reports: Environmental awareness as part of algorithmic accountabilityKadan Lottick, Silvia Susai, Sorelle A. Friedler et al.2019 · arXiv.org · 91 citations In Library

09Exploring the Limits of Transfer Learning with a Unified Text-to-Text TransformerColin Raffel, Noam Shazeer, Adam Roberts et al.2019 · Journal of machine learning research · 27k citations In Library

10CodeSearchNet Challenge: Evaluating the State of Semantic Code SearchHamel Husain, Hongqiu Wu, Tiferet Gazit et al.2019 · arXiv.org · 1.4k citations In Library

11Semantic program alignment for equivalence checkingBerkeley R. Churchill, Oded Padon, Rahul Sharma et al.2019 · ACM-SIGPLAN Symposium on Programming Language Design and Implementation · 104 citations

12Energy and Policy Considerations for Deep Learning in NLPEmma Strubell, Ananya Ganesh, A. McCallum2019 · Annual Meeting of the Association for Computational Linguistics · 3.6k citations In Library

Scroll for more · 30 remaining

Similar papers

A Transformer-based Approach for Translating Natural Language to Bash CommandsQuchen Fu, Zhongwei Teng, Jules White et al.2021 · 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA) · 13 citations In Library

NL2CMD: An Updated Workflow for Natural Language to Bash Commands TranslationQuchen Fu, Zhongwei Teng, Marco Georgaklis et al.2023 · arXiv (Cornell University) In Library

Natural Language to Bash Command Translation Using Large Language Models with Agentic External Tool FeedbackMehmet Eren Yeşilyurt, Ömer Özgür Tanrıöver2025 In Library

NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating SystemXi Victoria Lin, Chenglong Wang, Luke Zettlemoyer et al.2018 · International Conference on Language Resources and Evaluation · 214 citations In Library

LLM-Supported Natural Language to Bash TranslationFinnian Westenfelder, Erik Hemberg, Miguel Tulla et al.2025 · arXiv (Cornell University) · 13 citations In Library

Efficient Constituency Tree based Encoding for Natural Language to Bash TranslationShikhar Bharadwaj, Shirish Shevade2022 · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · 5 citations In Library

Explainable Natural Language to Bash Translation using Abstract Syntax TreeShikhar Bharadwaj, Shirish Shevade2021 In Library

A Survey of Natural-Language Driven Command-Line Assistants and Intelligent Shell SystemsProf. Nikhil Agrawal, Anushka Patil, Nikhil Kurkure et al.2025 · Zenodo (CERN European Organization for Nuclear Research) In Library

A Survey of Natural-Language Driven Command-Line Assistants and Intelligent Shell SystemsProf. Nikhil Agrawal, Anushka Patil, Nikhil Kurkure et al.2025 · Zenodo (CERN European Organization for Nuclear Research) In Library

A Survey of Natural-Language Driven Command-Line Assistants and Intelligent Shell SystemsProf. Nikhil Agrawal, Anushka Patil, Nikhil Kurkure et al.2025 · Zenodo (CERN European Organization for Nuclear Research) In Library

Source of record

arXiv

44B

Welcome.

This panel sticks with you. Pick anything below and it opens right beside it, so you can dig through 60,000-plus records without ever losing your spot here.

Travel 44B

LibraryPapers, policy, standards, statute LabsEvery organization building AI ModelsIntelligence, price, and speed BenchmarksThe evaluation catalog IncidentsAI failures and harms SearchOne field across everything DashboardThe state of AI accountability in NY 44B RegistryThe Art. 44-B compliance portal