Distributed streaming dataflow systems have evolved into scalable and fault-tolerant production-grade systems. Their applicability has departed from the mere analysis of stream- ing windows and complex-event processing, and now includes cloud applications and machine learning inference. Although the advancements in the state management of streaming systems have contributed significantly to their maturity, the internal state of streaming operators has been so far hidden from external applications. However, that internal state can be seen as a materialized view that can be used for analytics, monitoring, and debugging.
In this work we argue that exposing the internal state of streaming systems to outside applications by making it queryable, opens the road for novel use cases. To this end, In this talk I will introduce S-QUERY: an approach and reference architecture where the state of stream processors can be queried - either live or through snapshots, achieving different isolation levels. I will show how this new capability can be implemented in an existing open-source stream processor, and how queryable state can affect the performance of such a system.
Jim Verheijde recently joined IMC as a software engineer. Prior to this he received his master degree in computer science and bachelor degree in computer science engineering from Delft University of Technology. Jim also completed a minor at the National University of Singapore and joined the summer school program at Tsinghua university in China. During his studies In Delft, Jim took part in Forze for multiple years where he helped design and build the software for the world’s fastest hydrogen race car.