Contemporary deep-learning object detection methods for autonomous driving\nusually assume prefixed categories of common traffic participants, such as\npedestrians and cars. Most existing detectors are unable to detect uncommon\nobjects and corner cases (e.g., a dog crossing a street), which may lead to\nsevere accidents in some situations, making the timeline for the real-world\napplication of reliable autonomous driving uncertain. One main reason that\nimpedes the development of truly reliably self-driving systems is the lack of\npublic datasets for evaluating the performance of object detectors on corner\ncases. Hence, we introduce a challenging dataset named CODA that exposes this\ncritical problem of vision-based detectors. The dataset consists of 1500\ncarefully selected real-world driving scenes, each containing four object-level\ncorner cases (on average), spanning more than 30 object categories. On CODA,\nthe performance of standard object detectors trained on large-scale autonomous\ndriving datasets significantly drops to no more than 12.8% in mAR. Moreover, we\nexperiment with the state-of-the-art open-world object detector and find that\nit also fails to reliably identify the novel objects in CODA, suggesting that a\nrobust perception system for autonomous driving is probably still far from\nreach. We expect our CODA dataset to facilitate further research in reliable\ndetection for real-world autonomous driving. Our dataset will be released at\nhttps://coda-dataset.github.io.\n