I Know Which LLM Wrote Your Code Last Summer: LLM generated Code Stylometry for Authorship Attribution

As code generated by Large Language Models (LLMs) becomes more common, identifying the specific model behind each sample is increasingly important. This paper presents the first systematic study of LLM authorship attribution for C programs. We release CodeT5-Authorship, a novel LLM that uses only the encoder layers from the original CodeT5 encoder-decoder architecture. Our model’s encoder output (first token) is passed through a two-layer classification head with GELU activation and dropout, producing a probability distribution over possible authors. To evaluate our approach, we introduce LLM-AuthorBench, a benchmark of 32,000 compilable C programs generated by eight state-of-the-art LLMs across diverse tasks. We compare our model to seven traditional ML classifiers and eight fine-tuned transformer models, including BERT, RoBERTa, CodeBERT, ModernBERT, DistilBERT, DeBERTa-V3, Longformer, and LoRA-fine-tuned Qwen2-1.5B. In binary classification, our model achieves 97.56 % accuracy in telling apart source code written by related models such as GPT-4.1 and GPT-4o, and 95.40 % accuracy for multi-class attribution among five leading LLMs (Gemini 2.5 Flash, Claude 3.5 Haiku, GPT-4.1, Llama 3.3, and DeepSeek-V3). To support open science, we release the CodeT5-Authorship architecture, the LLM-AuthorBench dataset, and all relevant Google Colab scripts on GitHub: https://github.com/LLMauthorbench/.