Multi-Stage Retrieval for Operational Technology Cybersecurity Compliance Using Large Language Models: A Railway Casestudy

Operational Technology Cybersecurity (OTCS) continues to be a dominant challenge for critical infrastructure such as rail. As these systems are vulnerable to malicious attacks, effective documentation and development processes are essential to perform compliance assessments and proactively protect systems. In addition to maintaining compliance, it is also necessary to periodically evaluate the design documentation to account for the evolution of cybersecurity standards and threats. In this paper, a system is proposed that uses Large Language Models (LLMs) and multi-stage retrieval to enhance the compliance verification process. Initially, a basic architecture named baseline compliance architecture (BCA) is evaluated to answer OTCS compliance queries. After this, the extended approach is defined, which is called parallel compliance architecture (PCA). The performance of OpenAI-gpt-4o and Claude-3.5-haiku models is compared when used in the pipeline against the relative performance of the BCA. A discussion of their responses is provided and, through empirical evaluation, it is observed that PCA can score 0.1 points higher in correctness and 0.55 points higher in reasoning than BCA.