Clinical trials are essential for drug development but often suffer from\nexpensive, inaccurate and insufficient patient recruitment. The core problem of\npatient-trial matching is to find qualified patients for a trial, where patient\ninformation is stored in electronic health records (EHR) while trial\neligibility criteria (EC) are described in text documents available on the web.\nHow to represent longitudinal patient EHR? How to extract complex logical rules\nfrom EC? Most existing works rely on manual rule-based extraction, which is\ntime consuming and inflexible for complex inference. To address these\nchallenges, we proposed DeepEnroll, a cross-modal inference learning model to\njointly encode enrollment criteria (text) and patients records (tabular data)\ninto a shared latent space for matching inference. DeepEnroll applies a\npre-trained Bidirectional Encoder Representations from Transformers(BERT) model\nto encode clinical trial information into sentence embedding. And uses a\nhierarchical embedding model to represent patient longitudinal EHR. In\naddition, DeepEnroll is augmented by a numerical information embedding and\nentailment module to reason over numerical information in both EC and EHR.\nThese encoders are trained jointly to optimize patient-trial matching score. We\nevaluated DeepEnroll on the trial-patient matching task with demonstrated on\nreal world datasets. DeepEnroll outperformed the best baseline by up to 12.4%\nin average F1.\n