Stream processing lies in the backbone of modern businesses, being employed for mission critical applications such as real-time fraud detection, car-trip fare calculations, traffic management, and stock trading. Large-scale applications are executed by scale-out stream processing systems on thousands of long-lived operators, which are subject to failures. Recovering from failures fast and consistently are both top priorities, yet they are only partly satisfied by existing fault tolerance methods due to the strong assumptions these make.
In particular, prior solutions fail to address consistency in the presence of nondeterminism, such as calls to external services, asynchronous timers and processing-time windows.
We describe Clonos, a fault tolerance approach that achieves fast, local operator recovery with exactly-once guarantees and high availability by instantly switching to passive standby operators. Clonos enforces causally consistent recovery, including output deduplication, by tracking nondeterminism within the system through causal logging. Clonos is transparent to the user and highly configurable both in terms of resources and guarantees.
Pedro Silvestre is currently a PhD student in the Large-Scale Data & Systems Group https://lsds.doc.ic.ac.uk/ at Imperial College London http://www.imperial.ac.uk/, under the supervision of Dr. Peter Pietzuch https://www.doc.ic.ac.uk/~prp/. His research interests include Stream Processing Systems, Edge Computing and building specialized dataflow systems for other problem-spaces such as Reinforcement Learning. Before Imperial, Pedro was a Research Engineer at the TU Delft’s Web Information Systems Group https://www.tudelft.nl/ewi/over-de-faculteit/afdelingen/software-technology/web-information-systems/, working on High-Availability for Stream Processing and a student at the NOVA School of Science and Technology https://www.fct.unl.pt/en.