Real-time Spatial Retrieval Augmented Generation for Urban Environments

The proliferation of Generative Artificial Intelligence (AI), especially Large Language Models, presents transformative opportunities for urban applications through Urban Foundation Models. However, base models face limitations, as they only contain the knowledge available at the time of training, and updating them is both time-consuming and costly. Retrieval Augmented Generation (RAG) has emerged in the literature as the preferred approach for injecting contextual information into Foundation Models. It prevails over techniques such as fine-tuning, which are less effective in dynamic, real-time scenarios like those found in urban environments. However, traditional RAG architectures, based on semantic databases, knowledge graphs, structured data, or AI-powered web searches, do not fully meet the demands of urban contexts. Urban environments are complex systems characterized by large volumes of interconnected data, frequent updates, real-time processing requirements, security needs, and strong links to the physical world. This work proposes a real-time spatial RAG architecture that defines the necessary components for the effective integration of generative AI into cities, leveraging temporal and spatial filtering capabilities through linked data. The proposed architecture is implemented using FIWARE, an ecosystem of software components to develop smart city solutions and digital twins. The design and implementation are demonstrated through the use cases of a tourism assistant and for the management of traffic lights and luminaires in the city of Madrid. The use cases serve to validate the correct integration of Foundation Models through the proposed RAG architecture. They also enable the analysis of current model limitations, such as their inability to handle large volumes of information, even when it fits within their context window, and the high latency of Large Language Models caused by transformer-based architectures, which generate output token by token.