06-16, 10:15–10:55 (Europe/London), Salisbury
The analysis of logging messages is a big challenge because of their massive number, different origins and unspecify formats. These challenges can be partially address with NLP techniques and ultimately detect, predict or even maybe avoid incidents. What we demonstrate is complete monitoring solution. That includes clustering and uncovering warning-incident correlation with a Hawkes model. This model was previously successfully applied for earthquake predictions based on aftershocks. The Hawkes process model is well-defined mathematically and can process a large volume of data.
To continuously keep track of digital assets such as servers, network devices, and software programs log messages are produced. In general, they are massive stream of data and analysis of them is a big challenge especially because the format is unspecify (left for engineers), many shortcuts and names/numbers, etc. These challenges can be partially address with Natural Language Processing, but technical language is not completely natural. Fortunately, the common practice is to have three standard levels of logs: information, warnings and errors. The last one leads to incidents that we want to avoid, predict or as early as possible detect.
What we propose and shortly demonstrate is complete monitoring solution. That includes cutting edge natural language processing techniques, clustering and at the end warning-error correlation with a Hawkes process. This model was previously successfully applied for earthquake predictions based on aftershocks and capturing the dynamics of order books in finance. The Hawkes process model is well-defined mathematically and can process a large volume of data to uncover Granger causal structures in data if implemented appropriately. We show how Hawkes processes help answer the question ‘what caused what’ within the IT infrastructure.
Presentation outline:
- Speaker introduction (2 minutes)
- Introduction of problem (5 minutes)
- Recap: Natural Language Processing (3 minutes)
- Demo: Clustering using Drain 3 (5 minutes)
- Mathematics behind Hawkes processes (5 minutes)
- Demo: Incident prediction using Hawkes processes (5 minutes)
- Summary and Questions (5 minutes)
No previous knowledge expected
Product Lead of Data Scientist team in IT monitoring department of ING. PhD in Physics, BSc in Computer Science.