AISum4DCR: A Multi-Topic Dialog Summarization Approach for Developer Chat Rooms Integrating AI Chain and Prompt Engineering
Developer chat rooms (DCR) serve as critical platforms for collaborative software development, and summarizing dialogs facilitates knowledge retrieval and sharing. Due to sparse labeled data and dynamic topic shifts in dialogs, existing models fail to generalize effectively and struggle to preserve special contents (e.g., code snippets and links), which degrades summary quality. To address these challenges, we propose AISum4DCR, an automated summarization approach that integrates AI chain and prompt engineering with large language models (LLMs). AISum4DCR leverages natural language processing techniques to integrate hybrid semantic features from dialog sentences, code snippets, and links. It then segments conversation threads by topic and performs hierarchical clustering. After that, AISum4DCR performs segmented summarization through the AI chain, and finally merges the summaries. Experimental results on Gitter dialogs demonstrate that AISum4DCR outperforms baseline models (BART-CNN, T5-SAM, and ChatGPT) in both usefulness (F1-score improvement of 18.7% in downstream classification tasks) and readability (human-rated score of 3.01/4.0).
Paper
Full text
AISum4DCR: A Multi-Topic Dialog Summarization Approach for Developer Chat Rooms Integrating AI Chain and Prompt Engineering
OpenAlex · Topic Modeling · 2025
Abstract
Developer chat rooms (DCR) serve as critical platforms for collaborative software development, and summarizing dialogs facilitates knowledge retrieval and sharing. Due to sparse labeled data and dynamic topic shifts in dialogs, existing models fail to generalize effectively and struggle to preserve special contents (e.g., code snippets and links), which degrades summary quality. To address these challenges, we propose AISum4DCR, an automated summarization approach that integrates AI chain and prompt engineering with large language models (LLMs). AISum4DCR leverages natural language processing techniques to integrate hybrid semantic features from dialog sentences, code snippets, and links. It then segments conversation threads by topic and performs hierarchical clustering. After that, AISum4DCR performs segmented summarization through the AI chain, and finally merges the summaries. Experimental results on Gitter dialogs demonstrate that AISum4DCR outperforms baseline models (BART-CNN, T5-SAM, and ChatGPT) in both usefulness (F1-score improvement of 18.7% in downstream classification tasks) and readability (human-rated score of 3.01/4.0).