SYSTEM FOR DYNAMIC BATCHING AT VARYING GRANULARITIES USING MICRO-BATCHING TO ACHIEVE BOTH NEAR REAL-TIME AND BATCH PROCESSING CHARACTERISTICS

This disclosure relates generally to automated workflow for data management, and specifically to efficient data manipulation and information extraction techniques used by a versatile data analytics platform. The information extraction techniques include micro-batching and provenance tracking, as well as integration of provenance-driven policies to improve the efficacy of the analytics. Embodiments disclosed here integrate the concept of real-time or near-real-time data stream processing and event/batch processing into one data processing layer, allowing a single processing definition to be applied at different granularities.