Are prompts all you need? Evaluating prompt-based Large Language Models (LLM)s for software requirements classification

Context: Requirements classification is a fundamental task in software engineering that involves classifying natural language requirements into predefined classes, such as functional and non-functional requirements. Accurate classification is critical to the success of software projects, as it helps reduce development risks and improve overall software quality. However, most existing classification models rely on supervised learning approaches, which demand large, annotated datasets that are resource-intensive, time-consuming, and dependent on domain-specific expertise to acquire. Moreover, these models often struggle with generalizability, as they typically require retraining or fine-tuning for each new classification task. Objective: This study aims to address the challenge of dataset scarcity in requirements classification by investigating the effectiveness of prompt-based LLMs. Method: We conducted an empirical benchmarking study to evaluate and statistically compare the performance of multiple prompt-based LLMs and prompting techniques across several requirements classification tasks on two English requirements classification benchmark datasets: PROMISE and SecReq. The evaluation incorporated four prompting techniques: zero-shot, few-shot, persona, and chain of thought (CoT) prompts. For each task, we assessed and statistically compared the performance of different model–prompt configurations. We then compared the best-performing prompt-based LLM configurations on each task against the performance of the state-of-the-art fine-tuned transformer-based model. Results: The results demonstrate that prompt-based LLMs, particularly when paired with few-shot prompting, can achieve classification performance comparable to or even exceeding that of the state-of-the-art fine-tuned transformer-based model. Furthermore, augmenting few-shot prompts with either persona alone or a combination of persona and CoT can potentially improve model performance. Conclusion: This study highlights the potential of prompt-based LLMs as a practical and scalable alternative to traditional supervised approaches for requirements classification. By reducing the dependency on large annotated datasets, these models provide a flexible solution to overcoming data scarcity and enhancing the generalizability of automated requirements classification tools.