Evaluating the Efficacy of Prompt-Engineered Large Multimodal Models Versus Fine-Tuned Vision Transformers in Image-Based Security Applications

The success of Large Language Models (LLMs) has spurred the rise of Large Multimodal Models (LMMs), which integrate multiple modalities, such as text and images, to address complex data analysis tasks. As these black-box models gain popularity due to their ease of use and adaptability, there is growing interest in understanding their potential to replace or complement task-specific models in domain-specific applications. This article evaluates the applicability and effectiveness of prompt-engineered LMMs, specifically LLaVA, BakLLaVA, Moondream, Gemini 1.5 Flash, and GPT-4o, compared to fine-tuned Vision Transformer (ViT) models in addressing cybersecurity challenges of varying complexity. Our study examines three distinct tasks: (1) detecting visual triggers indicative of potential backdoors in two scenarios (digit recognition and traffic sign classification), (2) identifying phishing attempts from Web site screenshots, and (3) classifying malware based on visual representations. The results reveal that prompt-engineered LMMs perform competitively on tasks with visually evident or moderately complex features, such as trigger detection and phishing classification, with GPT-4o and Gemini 1.5 Flash demonstrating superior performance among LMMs. However, for the highly specialized task of malware classification, LMMs exhibit notable limitations when relying solely on prompting. Fine-tuning GPT-4o significantly improves its performance, yet it still lags behind fine-tuned ViT models, which consistently achieve higher accuracy across all tasks. While ViTs deliver superior precision and robustness, they require substantial resources for training, fine-tuning, and maintenance. In contrast, LMMs provide flexibility and ease of deployment, making them an appealing alternative for scenarios where resource constraints or rapid implementation are critical. This study highlights the tradeoffs between these approaches, emphasizing that while ViTs are indispensable for high precision, specialized applications, LMMs offer a scalable and versatile solution for less complex or resource-limited tasks.