Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure

As machine learning systems move from computer-science laboratories into the\nopen world, their accountability becomes a high priority problem.\nAccountability requires deep understanding of system behavior and its failures.\nCurrent evaluation methods such as single-score error metrics and confusion\nmatrices provide aggregate views of system performance that hide important\nshortcomings. Understanding details about failures is important for identifying\npathways for refinement, communicating the reliability of systems in different\nsettings, and for specifying appropriate human oversight and engagement.\nCharacterization of failures and shortcomings is particularly complex for\nsystems composed of multiple machine learned components. For such systems,\nexisting evaluation methods have limited expressiveness in describing and\nexplaining the relationship among input content, the internal states of system\ncomponents, and final output quality. We present Pandora, a set of hybrid\nhuman-machine methods and tools for describing and explaining system failures.\nPandora leverages both human and system-generated observations to summarize\nconditions of system malfunction with respect to the input content and system\narchitecture. We share results of a case study with a machine learning pipeline\nfor image captioning that show how detailed performance views can be beneficial\nfor analysis and debugging.\n