Accurate automatic organ segmentation is an important yet challenging problem\nfor medical image analysis. The pancreas is an abdominal organ with very high\nanatomical variability. This inhibits traditional segmentation methods from\nachieving high accuracies, especially compared to other organs such as the\nliver, heart or kidneys. In this paper, we present a holistic learning approach\nthat integrates semantic mid-level cues of deeply-learned organ interior and\nboundary maps via robust spatial aggregation using random forest. Our method\ngenerates boundary preserving pixel-wise class labels for pancreas\nsegmentation. Quantitative evaluation is performed on CT scans of 82 patients\nin 4-fold cross-validation. We achieve a (mean $\\pm$ std. dev.) Dice Similarity\nCoefficient of 78.01% $\\pm$ 8.2% in testing which significantly outperforms the\nprevious state-of-the-art approach of 71.8% $\\pm$ 10.7% under the same\nevaluation criterion.\n