Large language models are known to suffer from the hallucination problem in\nthat they are prone to output statements that are false or inconsistent,\nindicating a lack of knowledge. A proposed solution to this is to provide the\nmodel with additional data modalities that complements the knowledge obtained\nthrough text. We investigate the use of visual data to complement the knowledge\nof large language models by proposing a method for evaluating visual knowledge\ntransfer to text for uni- or multimodal language models. The method is based on\ntwo steps, 1) a novel task querying for knowledge of memory colors, i.e.\ntypical colors of well-known objects, and 2) filtering of model training data\nto clearly separate knowledge contributions. Additionally, we introduce a model\narchitecture that involves a visual imagination step and evaluate it with our\nproposed method. We find that our method can successfully be used to measure\nvisual knowledge transfer capabilities in models and that our novel model\narchitecture shows promising results for leveraging multimodal knowledge in a\nunimodal setting.\n