There is a pressing need to execute an increasingly diverse set of workloads efficiently on modern hardware. Wouldn’t it be nice if we could apply powerful cross-domain optimizations and easily adjust to the fast moving target modern hardware is today. Unfortunately, many high-performance engines struggle to embrace this exciting opportunity, often because they are not flexible or extendible enough. Their monolithic internal designs require a high effort in order to support additional workloads transparently or to adapt swiftly to the changes in the underlying hardware. With LingoDB we build an open source, compiling data processing engine to meet these needs and expectations. We break up previously monolithic components such as query plans, query optimizers, and query executers to construct an open and flexible engine that can not only execute complex SQL queries, but also user-defined algorithms and operators and makes cross-domain optimizations feasible. Through code generation and transparent optimizations such as auto-parallelization, LingoDB ensures that even user-defined operators are executed efficiently on modern hardware, while keeping the implementation effort low.
Michael is a second year PhD student at the Technical University of Munich. Supervised by Jana Giceva, he is researching on novel architectures for database engines using compiler technology.