REMOVING SUBSTITUTION RULES BASED ON USER INTERACTIONS

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for removing substitution rules. According to one implementation, a method includes identifying a plurality of revised search queries associated with a substitution rule, wherein the substitution rule is used to generate the revised search queries from each of a plurality of original search queries to include a substitute query term for an original query term included in the original search queries; for each of the revised search queries, accessing query log data that identifies (i) a set of search results generated by the respective revised search query, and (ii) a particular, user-selected search result in the set of search results; determining a quality metric for the substitution rule using the query log data; determining that the quality metric does not satisfy a threshold; and in response to determining that the quality metric does not satisfy the threshold, removing the substitution rule.