Reinforcement learning for question answering in programming domain using public community scoring as a human feedback

This study explores improving GPT Neo 125M in programming-focused Community Question Answering (CQA) using Reinforcement Learning from Human Feedback (RLHF) and Stack Overflow scores. We utilized two reward model training strategies with Proximal Policy Optimization (PPO), achieving enhancements comparable to GPT Neo's 2.7B model. The research introduces an auxiliary scoring mechanism, revealing the limitations of traditional linguistic metrics for programming responses. It highlights the need for domain-specific evaluation methods and the challenges in applying RLHF to programming CQA, contributing to the advancement of Large Language Models (LLMs) with human feedback.