Understanding a medical conversation between a patient and a physician poses\na unique natural language understanding challenge since it combines elements of\nstandard open ended conversation with very domain specific elements that\nrequire expertise and medical knowledge. Summarization of medical conversations\nis a particularly important aspect of medical conversation understanding since\nit addresses a very real need in medical practice: capturing the most important\naspects of a medical encounter so that they can be used for medical decision\nmaking and subsequent follow ups.\n In this paper we present a novel approach to medical conversation\nsummarization that leverages the unique and independent local structures\ncreated when gathering a patient's medical history. Our approach is a variation\nof the pointer generator network where we introduce a penalty on the generator\ndistribution, and we explicitly model negations. The model also captures\nimportant properties of medical conversations such as medical knowledge coming\nfrom standardized medical ontologies better than when those concepts are\nintroduced explicitly. Through evaluation by doctors, we show that our approach\nis preferred on twice the number of summaries to the baseline pointer generator\nmodel and captures most or all of the information in 80% of the conversations\nmaking it a realistic alternative to costly manual summarization by medical\nexperts.\n