Conventional spoken language understanding systems consist of two main\ncomponents: an automatic speech recognition module that converts audio to a\ntranscript, and a natural language understanding module that transforms the\nresulting text (or top N hypotheses) into a set of domains, intents, and\narguments. These modules are typically optimized independently. In this paper,\nwe formulate audio to semantic understanding as a sequence-to-sequence problem\n[1]. We propose and compare various encoder-decoder based approaches that\noptimize both modules jointly, in an end-to-end manner. Evaluations on a\nreal-world task show that 1) having an intermediate text representation is\ncrucial for the quality of the predicted semantics, especially the intent\narguments and 2) jointly optimizing the full system improves overall accuracy\nof prediction. Compared to independently trained models, our best jointly\ntrained model achieves similar domain and intent prediction F1 scores, but\nimproves argument word error rate by 18% relative.\n