Machine Learning in the Wild: Early Evidence of Non-Compliant ML-Automation in Open-Source Software

The increasing availability of Machine Learning (ML) models, particularly foundation models, enables their use across a range of downstream applications, from scenarios with missing data to safety-critical contexts. This, in principle, may contravene not only the models'terms of use, but also governmental principles and regulations. This paper presents a preliminary investigation into the use of ML models by 173 open-source projects on GitHub, spanning 16 application domains. We evaluate whether models are used to make decisions, the scope of these decisions, and whether any post-processing measures are taken to reduce the risks inherent in fully autonomous systems. Lastly, we investigate the models'compliance with established terms of use. This study lays the groundwork for defining guidelines for developers and creating analysis tools that automatically identify potential regulatory violations in the use of ML models in software systems.