Code quality is and will be a crucial factor while developing new software\ncode, requiring appropriate tools to ensure functional and reliable code.\nMachine learning techniques are still rarely used for software engineering\ntools, missing out the potential benefits of its application. Natural language\nprocessing has shown the potential to process text data regarding a variety of\ntasks. We argue, that such models can also show similar benefits for software\ncode processing. In this paper, we investigate how models used for natural\nlanguage processing can be trained upon software code. We introduce a data\nretrieval pipeline for software code and train a model upon Java software code.\nThe resulting model, JavaBERT, shows a high accuracy on the masked language\nmodeling task showing its potential for software engineering tools.\n