Machine learning (ML) is increasingly being used in image retrieval systems\nfor medical decision making. One application of ML is to retrieve visually\nsimilar medical images from past patients (e.g. tissue from biopsies) to\nreference when making a medical decision with a new patient. However, no\nalgorithm can perfectly capture an expert's ideal notion of similarity for\nevery case: an image that is algorithmically determined to be similar may not\nbe medically relevant to a doctor's specific diagnostic needs. In this paper,\nwe identified the needs of pathologists when searching for similar images\nretrieved using a deep learning algorithm, and developed tools that empower\nusers to cope with the search algorithm on-the-fly, communicating what types of\nsimilarity are most important at different moments in time. In two evaluations\nwith pathologists, we found that these refinement tools increased the\ndiagnostic utility of images found and increased user trust in the algorithm.\nThe tools were preferred over a traditional interface, without a loss in\ndiagnostic accuracy. We also observed that users adopted new strategies when\nusing refinement tools, re-purposing them to test and understand the underlying\nalgorithm and to disambiguate ML errors from their own errors. Taken together,\nthese findings inform future human-ML collaborative systems for expert\ndecision-making.\n