Evaluating the Use of LLMs for Documentation to Code Traceability

Large Language Models (LLMs) offer new potential for automating documentation-to-code traceability, yet their capabilities and limitations remain underexplored. We evaluate three LLMs (Claude Sonnet, GPT-4o, and o3-mini) on documentation-to-code traceability across diverse software projects. We create two datasets from open-source systems (Unity Catalog and Crawl4AI), curated to minimize training-data overlap with the evaluated models, and derive a 27-project benchmark from CoDocBench and TransArc to study generalizability across established codebases. Our experiments assess three capabilities: (1) trace link identification, (2) relationship explanation quality, and (3) multi-step chain reconstruction, while comparing LLMs against TF-IDF, BM25, and CodeBERT baselines. On the curated datasets, the best LLM attains F1-scores of 79.4% and 80.4%, and on the Established Projects benchmark, LLMs reach average F1 up to 92.3%, substantially outperforming traditional baselines except in a single dense system. Fully correct explanations cover 42.9%–71.1% of links, while partial correctness exceeds 97%, and chain endpoints are almost always recovered, though intermediate steps are often imperfect. We also show that task framing, particularly one-to-many matching, is crucial for recall, whereas naive many-to-many prompts achieve around 5% F1 due to enumeration bias. Overall, LLMs are promising assistants for trace discovery when combined with human oversight and careful tool design.

Paper

Similar papers

© 2026 NYSGPT2525 LLC