A large and diverse set of measurements are regularly collected during a\npatient's hospital stay to monitor their health status. Tools for integrating\nthese measurements into severity scores, that accurately track changes in\nillness severity, can improve clinicians ability to provide timely\ninterventions. Existing approaches for creating such scores either 1) rely on\nexperts to fully specify the severity score, or 2) train a predictive score,\nusing supervised learning, by regressing against a surrogate marker of severity\nsuch as the presence of downstream adverse events. The first approach does not\nextend to diseases where an accurate score cannot be elicited from experts. The\nsecond approach often produces scores that suffer from bias due to\ntreatment-related censoring (Paxton, 2013). We propose a novel ranking based\nframework for disease severity score learning (DSSL). DSSL exploits the\nfollowing key observation: while it is challenging for experts to quantify the\ndisease severity at any given time, it is often easy to compare the disease\nseverity at two different times. Extending existing ranking algorithms, DSSL\nlearns a function that maps a vector of patient's measurements to a scalar\nseverity score such that the resulting score is temporally smooth and\nconsistent with the expert's ranking of pairs of disease states. We apply DSSL\nto the problem of learning a sepsis severity score using a large, real-world\ndataset. The learned scores significantly outperform state-of-the-art clinical\nscores in ranking patient states by severity and in early detection of future\nadverse events. We also show that the learned disease severity trajectories are\nconsistent with clinical expectations of disease evolution. Further, using\nsimulated datasets, we show that DSSL exhibits better generalization\nperformance to changes in treatment patterns compared to the above approaches.\n