SCALABLE AND EXTENDABLE STREAM PROCESSING

An extensible architecture that enables developers to focus solely on the domain-specific nature of the stream processing algorithm to be implemented. It is positioned as an intermediary component between streaming data feeds and stream algorithms, thereby aggregating demand on data sources and hiding the complexity involved in managing active connections to different data sources. Per-algorithm stream throttling is provided so that individual algorithms do not get overloaded; thus, ensuring that algorithms receive fresh items from the data feeds to which the algorithms subscribe. Feed items can be discarded when an algorithm is not able to process the items in realtime to ensure that feed items are sampled at the fastest processing rate of the algorithm. Thus, a single instance of an algorithm can handle an entire data stream. Moreover, redundancy can be achieved by running the same configuration on multiple machines.