Image-based tracking of medical instruments is an integral part of surgical\ndata science applications. Previous research has addressed the tasks of\ndetecting, segmenting and tracking medical instruments based on laparoscopic\nvideo data. However, the proposed methods still tend to fail when applied to\nchallenging images and do not generalize well to data they have not been\ntrained on. This paper introduces the Heidelberg Colorectal (HeiCo) data set -\nthe first publicly available data set enabling comprehensive benchmarking of\nmedical instrument detection and segmentation algorithms with a specific\nemphasis on method robustness and generalization capabilities. Our data set\ncomprises 30 laparoscopic videos and corresponding sensor data from medical\ndevices in the operating room for three different types of laparoscopic\nsurgery. Annotations include surgical phase labels for all video frames as well\nas information on instrument presence and corresponding instance-wise\nsegmentation masks for surgical instruments (if any) in more than 10,000\nindividual frames. The data has successfully been used to organize\ninternational competitions within the Endoscopic Vision Challenges 2017 and\n2019.\n