This is the first part of many posts in a series of blog posts where I want to document how the MepSQL packages were built. By doing that, I will also end up covering the MariaDB build system (which this is based on), some of BuildBot, Amazon EC2 cloud and packaging DEBs and RPMs just in general, so it could be interesting from many perspectives. In this first part I'll simply scribble some notes about reviewing the OpenSuse Build System, Launchpad PPA service vs using your own servers and automating the builds with BuildBot.

Originally I just wanted to work on some new ideas on the automated build and QA system used by MariaDB. But since leaving Monty Program I didn't have access to any of those servers anymore, so as a first step I had to look into what alternatives there are for building binary packages for many operating systems and hardware platforms. In fact, this was another thing I had wanted to learn more about for a while. For instance Michal Hrušecký uses OpenSuse Build Service to build both MySQL and MariaDB packages for all RPM based distributions in the blink of an eye - I was interested to find out what's behind that magic.

After another week of hacking on MepSQL the DEB files for Ubuntu are now available.(MepSQL is my new "just a hobby" MySQL fork project.)

The Download page has instructions on how to install the packages with a simple apt-get install command. Debian packages will appear soon as they are now easy to add - I mostly just need to add new Amazon images for each.

Some time ago Stephen O'Grady and Brian Aker had an interesting Blogo-dialogue about what they call the "Cambrian Explosion" of open source development. The Cambrian Explosion means that we increasingly see forks of projects being developed in different directions, where traditionally we are used to open source development happening in relatively hierarchical and easy to follow upstream and downstream relationships. This is exactly what happens in the MySQL community currently, where in total there is more progress than ever before, but that progress is divided among several competing forks, none of which is strictly in an upstream-downstream hierarchy with each other.

I used to be a bit frustrated about this state of affairs, believing that if at least most of the forks could co-operate on a common tree, we would see even faster progress. But when I left MariaDB some months ago, I realized that the situation is what it is and since all existing MySQL variants are associated with a commercial vendor, there wasn't an obvious choice for me to continue contributing, which I still want to do. So I thought I might as well embrace this Cambrian Explosion thing and just publish my own fork as a contribution to the community.

And this is where I today announce the new MepSQL project and the immediate availability of MepSQL 5.1.52-alpha1.

In November a Mark Schonewille posted a blog on when you can't and cannot use the GPL version of MySQL together with your closed source application. The post was a result of actually talking to an Oracle lawyer which makes it valuable information. Unfortunately Mark's blog is now offline (it seems he didn't renew his domain registration?)

This is just a repost of the disappeared blog post. (The small print allows me to copy it verbatim.) There is no commentary from myself, except that what Mark wrote is the same I also heard Oracle say a year ago. That Oracle is being consistent on this point is very welcome and deserves to be kept available online.

Update 2011-06-23: Mark comments that his blog is still online, but at a new address: https://qery.us/tl

I often wonder what's behind the increased trend behind Hadoop and other NoSQL technologies. I realize if you're Yahoo that such technology makes sense. I don't get why everyone else wants to use it.

Reading Stephen O'Grady's self-review of his predictions for 2010 for the first time gave me some insights into how such people think:

Democratization of Big Data

Consider that RedMonk, a four person analyst shop, has the technical wherewithal to attack datasets ranging from gigabytes to terabytes in size. Unless you’re making institutional money, budgets historically have not permitted this. The tools of Big Data have never been more accessible than they are today.

My recent account of The State of MySQL forks seems to have gotten quite a lot of attention. I promised to follow up with a separate piece about Drizzle and also PostgreSQL, as the other major open source database, so I'd better keep that promise now.

Just a short note again on HandlerSocket developments: PHP bindings to the HandlerSocket client library have been spotted in the wild: https://code.google.com/p/php-handlersocket/ (I'd like to credit someone, but I have no idea who user "avuenta" is on Google Code. Is it you Dathan?)

I found this on Dathan Vance Pattishall's blog. I haven't personally had time to try it out yet as I'm still deep into my current project (of which I hope to blog in January).

"...php-handlersocket which is a PECL type version (C driver with exposure to PHP). It does the job but needs some work that I'm doing now."

Just wanted to highlight that Percona Server has now added HandlerSocket to its most recent release, being the first "MySQL fork/distribution" to ship it in easy to consume binary downloads.

HandlerSocket brings NoSQL to MySQL, and does so with a vengeance! It was developed at DeNa, by Akira Higuchi and is already used in production in their MySQL servers. The announcement on my former collague Yoshinori Matsunobu's blog flaunts a 7x performance improvement over the standard SQL interface in MySQL. The most astonishing part is that their MySQL is now faster than Memcached, even if the latter doesn't store anything to disk, so with this NoSQL-for-MySQL solution it makes sense to remove the caching layer completely!

Giuseppe "The Data Charmer" Maxia recently posted his take on the MySQL forks. I had been pondering whether to do the same, and seeing that what I planned to write will nicely complement Giuseppe's article, I was inspired to follow him into the same topic. Note that last Spring I created a Map of MySQL forks in preparation for Monty's keynote at the MySQL user conference. So let's see how things have evolved. I'll look into MySQL ecosystem as a whole and the forks separately.

The post is long, but the key takeaway is that despite the challenges, the combined development seen in the MySQL ecosystem is probably stronger than ever, the current situation is hard for an outsider to grasp but manageable, and if a few more obstacles can be overcome, we are looking into a very bright future indeed. There are more than 100 engineers (how much more?) working full time on the mysql code base (including both developers, QA, build engineers...). This development effort is an order of magnitude higher than other open source databases I'm aware of, in particular PostgreSQL and Drizzle. Often the open source project with most momentum and mass will come out as the winner, no matter what challenges it may seemingly be facing, and this is the case with MySQL too.

Some time ago I was asked to do a study of our most popular open source projects to assess 1) what governance models are out there and 2) if the governance model has any effect on the project's success (such as size of developer community) on the one hand and on the other hand on the business of the related vendor(s). Some of the results are quite remarkable and have general applicability, so I wanted to share them here:

(Small updates done on 2011-07-14. OpenJDK size clarified on 2012-05-21.)