The Difficulty of Novelty Detection in Open-World Physical Domains: An Application to Angry Birds

Detecting and responding to novel situations in open-world environments is a\nkey capability of human cognition and is a persistent problem for AI systems.\nIn an open-world, novelties can appear in many different forms and may be easy\nor hard to detect. Therefore, to accurately evaluate the novelty detection\ncapability of AI systems, it is necessary to investigate how difficult it may\nbe to detect different types of novelty. In this paper, we propose a\nqualitative physics-based method to quantify the difficulty of novelty\ndetection focusing on open-world physical domains. We apply our method in the\npopular physics simulation game Angry Birds, and conduct a user study across\ndifferent novelties to validate our method. Results indicate that our\ncalculated detection difficulties are in line with those of human users.\n