SYSTEMS AND METHODS FOR SCHEDULING DATA FLOW EXECUTION BASED ON AN ARBITRARY GRAPH DESCRIBING THE DESIRED DATA FLOW

The data transformation system in one embodiment, comprises a capability to receive data, a data destination and a capability to store transformed data, and a data transformation pipeline that constructs complex end-to-end data transformation functionality by pipelining data flowing from one or more sources to one or more destinations through various interconnected nodes for transforming the data as it flows. Each component in the pipeline possesses predefined data transformation functionality, and the logical connections between components define the data flow pathway in an operational sense.The data transformation pipeline enables a user to develop complex end-to-end data transformation functionality by graphically describing and representing, via a GUI,a desired data flow from one or more sources to one or more destinations through various interconnected nodes (graph). Each node in the graph selected by the user represents predefined data transformation functionality, and connections between nodes define the data flow pathway.