Summarization systems make numerous "decisions" about summary properties\nduring inference, e.g. degree of copying, specificity and length of outputs,\netc. However, these are implicitly encoded within model parameters and specific\nstyles cannot be enforced. To address this, we introduce HydraSum, a new\nsummarization architecture that extends the single decoder framework of current\nmodels to a mixture-of-experts version with multiple decoders. We show that\nHydraSum's multiple decoders automatically learn contrasting summary styles\nwhen trained under the standard training objective without any extra\nsupervision. Through experiments on three summarization datasets (CNN, Newsroom\nand XSum), we show that HydraSum provides a simple mechanism to obtain\nstylistically-diverse summaries by sampling from either individual decoders or\ntheir mixtures, outperforming baseline models. Finally, we demonstrate that a\nsmall modification to the gating strategy during training can enforce an even\nstricter style partitioning, e.g. high- vs low-abstractiveness or high- vs\nlow-specificity, allowing users to sample from a larger area in the generation\nspace and vary summary styles along multiple dimensions.\n