SYSTEM AND METHOD FOR DATA TRANSFER USING A RECOVERABLE DATA PIPE

A system for data transfer using a recoverable data pipe includes a data producer, one or more data consumers, a storage device and a data pipe manager. The data producer may be configured to append updated data blocks to the storage device via a producer storage input/output (I/O) channel. The data consumers may be configured to read data blocks from the storage device via consumer storage I/O channels. The data pipe manager may be configured to maintain metadata identifying data blocks of the storage device that have been read by each data consumer, and to release backing storage corresponding to the data blocks that have been read by all data consumers.