LLM-Powered Nuanced Video Attribute Annotation for Enhanced Recommendations

This paper presents a case study of deploying Large Language Models (LLMs) as an advanced "annotation" mechanism to achieve nuanced content understanding (e.g., discerning content "vibe") at scale within an industrial short-form video recommendation system. Traditional machine learning classifiers for content understanding face protracted development cycles and a lack of deep, nuanced comprehension. The "LLM-as-annotators" approach addresses these by significantly shortening development times and enabling the annotation of subtle attributes. This work details an end-to-end workflow encompassing: (1) iterative definition and robust evaluation of target attributes, refined by offline metrics and online A/B testing; (2) scalable offline bulk annotation of video corpora using LLMs with multimodal features, optimized inference, and knowledge distillation for broad application; and (3) integration of these rich annotations into the online recommendation serving system, for example, through personalized restricted retrieval. Experimental results demonstrate the efficacy of this approach, with LLMs outperforming human raters in offline annotation quality for nuanced attributes and yielding significant improvements in user participation and satisfied consumption in online A/B tests. The study provides insights into designing and scaling production-level LLM pipelines for content annotation, highlighting the adaptability and benefits of LLM-based multimodal content understanding for enhancing video discovery, user satisfaction, and the overall effectiveness of modern recommendation systems.

Paper

Similar papers

© 2026 NYSGPT2525 LLC