The explosive growth of large language models (LLMs) has transformed the landscape of natural language generation, unlocking incredible capabilities—but also creating new avenues for misuse. This study delves into the darker side of LLMs, specifically their potential to generate highly convincing social engineering content, such as phishing schemes and impersonation tactics. By exploring the techniques of prompt engineering and fine-tuning, we demonstrate how these models can be manipulated to produce surprisingly realistic and contextually accurate malicious content. Our experiments show that open-source models can be exploited with alarming success, while fine-tuning further sharpens their ability to deceive. On the other hand, commercial models equipped with advanced safeguards do better at blocking harmful inputs—but even these systems are not foolproof. Drawing on these findings, we propose a risk framework and offer actionable recommendations for securing LLMs in real-world applications. This research underscores the urgent need for strong oversight, proactive security measures, and continuous adaptation to mitigate the dual-use risks posed by this powerful technology.
Paper
Full text
Fine-Tuning Language Models for Social Engineering: A Technical Feasibility Study
Semantic Scholar · 2025
Abstract
The explosive growth of large language models (LLMs) has transformed the landscape of natural language generation, unlocking incredible capabilities—but also creating new avenues for misuse. This study delves into the darker side of LLMs, specifically their potential to generate highly convincing social engineering content, such as phishing schemes and impersonation tactics. By exploring the techniques of prompt engineering and fine-tuning, we demonstrate how these models can be manipulated to produce surprisingly realistic and contextually accurate malicious content. Our experiments show that open-source models can be exploited with alarming success, while fine-tuning further sharpens their ability to deceive. On the other hand, commercial models equipped with advanced safeguards do better at blocking harmful inputs—but even these systems are not foolproof. Drawing on these findings, we propose a risk framework and offer actionable recommendations for securing LLMs in real-world applications. This research underscores the urgent need for strong oversight, proactive security measures, and continuous adaptation to mitigate the dual-use risks posed by this powerful technology.