Measuring user satisfaction level is a challenging task, and a critical\ncomponent in developing large-scale conversational agent systems serving the\nneeds of real users. An widely used approach to tackle this is to collect human\nannotation data and use them for evaluation or modeling. Human annotation based\napproaches are easier to control, but hard to scale. A novel alternative\napproach is to collect user's direct feedback via a feedback elicitation system\nembedded to the conversational agent system, and use the collected user\nfeedback to train a machine-learned model for generalization. User feedback is\nthe best proxy for user satisfaction, but is not available for some ineligible\nintents and certain situations. Thus, these two types of approaches are\ncomplementary to each other. In this work, we tackle the user satisfaction\nassessment problem with a hybrid approach that fuses explicit user feedback,\nuser satisfaction predictions inferred by two machine-learned models, one\ntrained on user feedback data and the other human annotation data. The hybrid\napproach is based on a waterfall policy, and the experimental results with\nAmazon Alexa's large-scale datasets show significant improvements in inferring\nuser satisfaction. A detailed hybrid architecture, an in-depth analysis on user\nfeedback data, and an algorithm that generates data sets to properly simulate\nthe live traffic are presented in this paper.\n