IT Infrastructure Downtime Preemption using Hybrid Machine Learning and NLP

Abstract

IT Infrastructure Management and server down- time have been an area of exploration by researchers and indus- try experts, for over a decade. Despite the research on web server downtime, system failure and fault prediction, etc., there is a void in the field of IT Infrastructure Downtime Manage- ment. Downtime in an IT Infrastructure can cause enormous fi- nancial, reputational and relationship losses for customer and vendor. Our attempt is to address this gap by developing an in- novative architecture which predicts IT Infrastructure failure. We have used a hybrid approach of human-machine interaction through Big Data, Machine Learning, NLP and IR. We sourced real-time machine, operating system, application logs and un- structured case notes into an algorithm for multi-dimensional symptoms mining, using iterative deepening depth-first search, traversal to create transactions for Sequential Pattern Mining of symptoms to events. It went through multiple statistical tests and review from technology experts, to create and update a dy- namic Pattern Dictionary. This dictionary is used for training unsupervised and supervised classification models of machine learning, namely SVM and Random Forrest to score and pre- dict new logs in a real time mode. The approach is also dynamic to use unsupervised clustering methods to give directions to the technicians on future or unknown pattern of errors or fault, to constantly update the Pattern Dictionary and improve classifi- cation for new IT products. General Terms—Experimentation, Algorithms, Service Sup- port, Technology, Research.