Part of my history inside InfiniDB/ColumnStore

Several years ago there was a fork of the unreleased MySQL 6.0 called Drizzle. It was designed to be a lightweight, cloud/web/UTF8 first database server with a microkernel style core. I worked for a while as one of the core developers of Drizzle until the corporate sponsor I worked for ceased funding its development.

Fast-forward to 2016 and I start working on MariaDB ColumnStore and one of the biggest surprises to me is that it incorporated part of Drizzle! Specifically the BSD licensed MySQL/MariaDB compatible client library called libdrizzle.

ColumnStore’s MariaDB plugin gets the entire query plan tree for a query and passes it on to its internal processes to break it up into parts that can be worked on in parallel. Since this doesn’t happen inside MariaDB server it needs a way to get at data that is not part of ColumnStore (such as InnoDB) for joins and subqueries.

This is where libdrizzle comes in, ColumnStore builds a query which can get the data required from the non-ColumnStore tables, makes a new client connection, executes the query and joins the result internally to the ColumnStore tables.

I suspect libdrizzle was originally used due to its BSD license which would make it easier for Calpont (the InfiniDB company) to have a commercial and Open Source version of InfiniDB. As well as its libmysqlclient-like API and MySQL compatibility.

So, where does my history come in? One of the things I worked on the most when I was working on the Drizzle project is libdrizzle. I had no clue that it was being used in this way which was an interesting surprise!

Unfortunately libdrizzle 2.0 (the version used in ColumnStore) is no longer developed. The only version that is being maintained is a fork of a more recent version Brian Aker and I created whilst we worked for HP called libdrizzle-redux. This was an attempt at simplifying the API and adding support for features that libdrizzle never supported. Development continues on GitHub by sociomantic labs GmbH. I have helped them get off the ground with the project and am happy a part of my work lives on.

Libdrizzle-redux has a very different API so we couldn’t just drop-in replace it. As much as I liked libdrizzle I didn’t want to continue maintaining the 2.0 version (I have fixed some things in the version ColumnStore 1.0 uses). In addition we didn’t really have a need for a BSD licensed connector in ColumnStore so for ColumnStore 1.1 we are switching to use the internal client connector that comes with MariaDB.

It was a little bittersweet removing a large chunk of my history from ColumnStore’s codebase but I’m glad that socimantic have continued to build a community around my legacy.