Mixed Integer Linear Programs (MILP) are well known to be NP-hard problems in general, and therefore, tackling and using their solution in online applications is a great challenge. Some Machine Learning techniques have been proposed in the literature to alleviate their computational burden. Unfortunately, these techniques report that a non-negligible percentage of the resulting machine-learning-based solutions are infeasible. By linking Mathematical Optimization and Machine Learning, this paper proposes an offline-online strategy to solve MILPs in a fast and reliable manner. The offline step seeks to identify the so-called support constraints of past instances of the target MILP and uses them to train a Machine Learning model of our choice. This model is subsequently used online to generate a reduced MILP that is significantly easier to solve. Through synthetic and real-life MILPs, we show that our approach dramatically decreases the risk of obtaining solutions from the reduced MILP that are infeasible in the original formulation without any extra cost in terms of computational time. Hence, our methodology is appropriate for online applications where feasibility is of particular importance.