LLMs in Code Vulnerability Analysis: A Proof of Concept

Context: Traditional software security analysis methods struggle to keep pace with the scale and complexity of modern codebases, requiring intelligent automation to detect, assess, and remediate vulnerabilities more efficiently and accurately. Objective: This paper explores the incorporation of code-specific and general-purpose Large Language Models (LLMs) to automate critical software security tasks, such as identifying vulnerabilities, predicting the severity and access complexity rate, and generating fixes as a proof of concept. Method: We evaluate five pairs of the most recent LLMs, which contain both code-based and general-purpose open-source models, on two recognized C/C++ vulnerability datasets, namely Big-Vul and Vul-Repair. Additionally, we show comparisons between fine-tuning and prompt-based approaches. Results: The results show that fine‑tuning uniformly outperforms both zero-shot and few‑shot approaches across all tasks and models. Notably, code‑specialized models excel in zero‑shot and few‑shot settings on complex tasks, but general‑purpose models remain nearly as effective. Discrepancies among CodeBLEU, CodeBERTScore, BLEU, and ChrF highlight the inadequacy of the current metrics for measuring repair quality. Conclusions: Through this study, we contribute to the resilience of the current software security community by investigating the potential of advanced LLMs.