MEASURING MUTUAL UNDERSTANDING IN HUMAN-COMPUTER CONVERSATION

Technical solutions are described for analyzing a natural language conversation-generating machine. The computer implemented method includes identifying, from a conversation log, a plurality of adjacency pairs. The method further includes determining, from the adjacency pairs, a number of adjacency pairs with outcome success indicators. The method further includes determining, from the adjacency pairs, a number of adjacency pairs with outcome failure indicators. The method further includes computing a mutual understanding score for the machine by computing a ratio of the number of adjacency pairs with outcome success indicators and the number of adjacency pairs with outcome failure indicators.