While OLTP engines excel at maintaining invariants under transactional workloads, and OLAP engines excel at ad-hoc analytics, the relational database is not presently an excellent tool for maintaining the results of computations as data change. This space is currently occupied largely by microservices, bespoke tools that suffer from all the problems you might expect of systems that do not provide many of the ACID properties, and which anecdotally consume engineering departments with their maintenance overhead. Materialize, then, is high-performance dataflow implementation of an incremental view maintenance engine for SQL92 queries, designed to remove as much of this pain as possible.
We’ll discuss what changes in a relational engine stack with this shift in responsibilities, as well as new and powerful idioms that emerge in a push-based SQL systems that simply don’t make sense in traditional polling systems.
Frank McSherry is Chief Scientist at Materialize, Inc. Prior to that, Frank did a fair bit of relatively public work on dataflow systems, first at MSR Silicon Valley (RIP) and most recently ETH Zurich, with a bit of pro bono work in between. He also did some work on differential privacy back in the day, and has one of the more confrontational laptops in the business.