STREAMING AND SAMPLING IN REAL-TIME LOG ANALYSIS

Technologies are described herein for incorporating streaming and/or sampling in real-time log analysis. Representative samples of log data are extracted from the log files on a number of monitored hosts and streamed in real-time to log processors for processing. The log processors accumulate and process the representative samples of log data, and track a data completeness value representing an indication of a proportion of total log data represented by the representative samples received. The representative samples of log data are merged and collated. Estimated metrics are calculated from the merged and collated representative samples and the data completeness, and the estimated metrics are published to consumers in near real-time.