Data Management Systems increasingly abandon monolithic architectures in favor of compositions of off-the-shelf components. Storage layers like Parquet and Arrow are combined with kernels like Velox and RocksDB, and optimizers like Calcite. The interfaces between these components are, however, the same as 30 years ago: highly efficient but rigid. This rigidity obstructs the adoption of novel ideas and techniques such as hardware acceleration, adaptive processing, learned optimization or serverless execution in real-world systems.
To address this impasse, I propose a novel approach to database composition inspired by early compiler-construction research: partial query evaluation. Under this paradigm, components communicate using a unified representation for data, code, execution plans and any combination thereof. I present an implementation of the approach in a new system called BOSS and illustrate how BOSS achieves a fully composable design that is effective, elegant and virtually overhead-free.
Holger Pirk is Associate Professor/Senior Lecturer in Computing, at Imperial College London. His research is at the intersection of data management, compilers and computer architecture: targeting new applications like visualization, games, IoT and AI as well as new platforms like compilers, GPUs or FPGAs as well all hardware-conscious algorithms, new data processing paradigms, algebraic optimizations, cost models and code generation techniques. De did his PhD years in the Database Architectures group at CWI in Amsterdam resulting in a PhD from the University of Amsterdam in 2015. Before joining Imperial, he was a Postdoc at the Database group at MIT CSAIL.