Announcing MepSQL, continuing the "Cambrian Explosion" of MySQL forks
Some time ago Stephen O'Grady and Brian Aker had an interesting Blogo-dialogue about what they call the "Cambrian Explosion" of open source development. The Cambrian Explosion means that we increasingly see forks of projects being developed in different directions, where traditionally we are used to open source development happening in relatively hierarchical and easy to follow upstream and downstream relationships. This is exactly what happens in the MySQL community currently, where in total there is more progress than ever before, but that progress is divided among several competing forks, none of which is strictly in an upstream-downstream hierarchy with each other.
I used to be a bit frustrated about this state of affairs, believing that if at least most of the forks could co-operate on a common tree, we would see even faster progress. But when I left MariaDB some months ago, I realized that the situation is what it is and since all existing MySQL variants are associated with a commercial vendor, there wasn't an obvious choice for me to continue contributing, which I still want to do. So I thought I might as well embrace this Cambrian Explosion thing and just publish my own fork as a contribution to the community.
What you'll find inside
There were several things I've wanted to hack on for a long time, which came together nicely in the MepSQL project. (I will blog about all of these in more detail in separate blog posts going forward - even hope to speak about this project at (un)conferences in 2011.)
- BuildBot for automated integration testing. Arguably the single most important reason MySQL quality has finally improved in the recent releases is the Pushbuild system MySQL engineers use internally at Oracle. This was developed some years ago by a Kristian Nielsen. The idea of continuous integration testing is that when a developer commits new code, it will automatically be run against the test-suite which by now has thousands of tests, and the developer then gets a report of failing tests. The Pushbuild system is not publicly available, but MariaDB uses an open source system based on BuildBot. The MariaDB build and QA system was developed by... Kristian Nielsen! Needless to say, I was always a big fan of Kristian's work when working with him, and I wanted to spend some time of my paternity leave working hands on with this system myself. Building MepSQL was an excellent task for such a project.
- Since I no longer work at Monty's company, I didn't have access to the servers where the buildbot installation actually runs. So a part of the project was to take the MariaDB build system and migrate it so I could run it on Amazon's EC2 cloud. This resulted in a new Launchpad project MepSQL Bakery. If you want to build some MySQL derived branch yourself, you can now do so for only a few bucks charged to your credit card!
- Kristian sometimes complained that the management of 70 (!) different KVM images used to build and test MariaDB for all the different Linux distributions and other operating systems out there requires more manual work than it should. To automate this in the MepSQL system I created a shell script used for bootstrapping a virtual server instance on boot. So when halfway through the project I had to add a build dependency to all of the BuildBot slaves, I didn't need to touch my AMI images at all, rather I just added one line of code to the shell script I used. I call this bootstrapping script "Loitsut" and this will be spun off as an independent project as this method is useful for anyone running servers in EC2 or any other cloud too.
- The first task was just to migrate all of this to EC2 and build MariaDB itself. But from there I raised the bar and wanted to build something more useful - after all MariaDB downloads are already delivered by MariaDB. So I pointed the system at the MySQL at Facebook repository which is what Mark Callaghan's team at Facebook publishes. These features typically find their ways into the commercial MySQL variants quite well nowadays, given some months or even years. But the newest features were always available only in this source code repository. So my hope is that providing binary packages of this MySQL fork (and soon also documenting them!) more adventurous MySQL users will be able to easily try them out, discuss them and just get excited about the great work the guys at Facebook keep on doing! I believe this is the first time anyone is offering this code as binary downloads, and this is my way of continuing to contribute to the MySQL ecosystem.
- Finally, while the first release is an exact clone of the mysqlatfacebook Launchpad project, there were some small fixes needed to makefiles and BUILD/compile-dist in order to build downloadable tar files (and deb files) instead of just going directly to make install.
Today the downloads work, but are still somewhat rough around the edges. I'm releasing all this following the "release early, release often" mantra of open source development. So in the short term there is a list of obvious tasks to follow:
- DEB packages are ready and will be available after testing.
- RPM packages will be built after that.
- There are some things still to fix in the current packages, it seems for instance a default buildbot installation produces packages with too little permissions, so MySQL doesn't start unless you first do chmod -R a+rx. There are other things too.
- The way the installation packages as well as the running server identifies themselves still needs spit and polish, now it just says "5.1.52" which is not correct. (A result of this is, you probably cannot upgrade from the current packages, but must uninstall them when the next packages are available.)
- The packages should be GPG signed and checksums generated.
- All the new features need to be documented.
- There might be a nicer website, especially when there is more content.
- Finally one might test whether 32 bit platforms would work, or even windows.
- After all this is done, there are still dozens of plugins, patches and utilities floating around the MySQL community that can be added for easy consumption. (Of course, nothing prevents anyone from developing new code either!)