When I was younger and had lots of free time, I used to do video editing as a hobby. At that time I developed a rule that is true for many projects in general (it was also true for writing a book some years later). The rule is: When you think you are 90% done, you are only 50% done. With video-editing, this meant that when the video was more or less ready, you are still 50% away from the final goal of actually having a master copy on tape. The latter 50% would be spent on checking ending credits, watching through the video a couple of times, and in those time, rendering even simplest of effects. Using a Windows PC for video editing was in those times a shaky effort in itself, so even when mastering you had to sit there and watch through the whole tape to make sure there were no glitches.

Producing a MariaDB release has been a similar process. In our company meeting in August we were discussing "final steps" to produce a final Beta, then Release Candidate, then production release. As I blogged then, the progress has been documented on a daily basis on the askmonty.org wiki.

Prioritizing features

We thought the list of features to include into the release was short.
Even so, we eventually pushed about half of them to a future release. For instance the Virtual Columns feature was done in itself, but seemed to expose a memory overrun in other parts of the server. Tracking this was difficult, because it did not happen in a debug build. Similarly a customer sponsored an optimization that avoids unnecesary flushing of the keycache and while this was done in time, pending final review it missed the train too. Some other features were pushed into the future just due to lack of time.

We did however get in 2 Percona patches as planned, so MariaDB does include some great additions that have been out in the MySQL community for some time now. Virtual Columns will go into the 5.2 branch, and in Monty's inbox we've found some further performance improvements done by MySQL users that we'll start looking at too.

Just one final merge...

One innocent bullet point on the todo list was to merge in the newest MySQL version, as well as XtraDB and PBXT. This was not a one day job...

Somehow we managed to start the merge from MySQL from a more or less random point in the MySQL 5.1 Launchpad tree. This led to a long list of merge failures and bugs that needed to be fixed, since of course between releases the MySQL tree, like any repositary, may or may not be fully stable. Quite soon we realized that we were merging something that was between MySQL 5.1.37 and 5.1.38, and the missing changesets to 5.1.38 were then added. Things got better from there.

I'm told the delta between MySQL and MariaDB is already 26.000 lines, a lot of it is in whitespace or comments. (This does not count storage engines Maria, XtraDB or PBXT.) A good target for cleanup work in our next release cycle... We also make sure to submit back our fixes to MySQL, to minimize divergence, it makes life easier for people on both sides.

The bigger problem was that XtraDB 6 was developed against MySQL 5.1.36, so merging it against a MariaDB with 5.1.38 was not easy.

All in all we spent at least 4 man weeks just fixing merge issues. But we are learning from our mistakes, and most importantly, documented them.

PBXT

If I remember correctly, merging the PBXT was relatively painless and the Primebase team was quick to respond to small issues that came up. We did notice that it did not compile on PowerPC (Mac OS X), and Paul McCullagh confirmed that this was not a supported platform for PBXT.

Testing, testing... and first it has to build.

It turns out that MySQL 5.1.38 happened to be a release with quite a lot of changes. Most notably Sun has suddenly included InnoDB plugin in the midst of a stable release series. (Considering the major benefit to MySQL users,
I actually understand the decision, it just happened to create additional surprises to us.) Possibly related to that (?) 5.1.38 also changed how it builds all storage engines, so we had to apply the same fix both to XtraDB and PBXT too.

Of course Buildbot runs did report all kinds of test-suite errors, so another batch of man weeks was spent fixing various test failures. Many test failures happened on Windows only. To our surprise they also happened in the stock MySQL 5.1.38 (which is a GA series after all). We were however able to find patches for most of those (on their way into 5.1.39?) or fix them ourselves, until we were down to a small number of test failures which were considered safe to ignore.

Developers told me that Windows always got less attention in MySQL, so for instance replication on Windows may never have been that heavily tested. Slightly worrying, but actually believable.

Installation packages

Personally I had completely underestimated the complexity of actually building TAR, ZIP, RPM, DEB and Windows installer packages. After all, developers build and do test runs every day. In any case, the Sun MySQL team produces builds steadily on a monthly basis. It turns out the knowledge how to do that doesn't really exist outside of Sun.

Lots of technical MySQL users know how to checkout the latest source code from launchpad, build it and play with it. Also lots of users know how to download the source code tar balls of MySQL releases, build it and play with it. What turned out to be non-trivial is 1) the step to create the source tar ball, from the repository and 2) even more the step to create a DEB or RPM from the source tar ball. Maybe the scripts to do this exist in the MySQL source codes, maybe they don't, but at least we did not know how to do that. Producing installation packages was more or less a Sun proprietary domain.

Thankfully OurDelta has been doing this task for the 5.0 series of MySQL. They had so far not produced 5.1 builds, but thanks to close co-operation with Kristian from Monty Program and Arjen from Open Query, we were able to remove all problems so that we now have produced builds for MariaDB 5.1.38.

This situation reminds me of other "open in license only" discussions, such as seems to be the case for Google's Android. The producers of MySQL and Android didn't put lots of effort into actually making it easy for others to excercise their GPL rights to build their own versions of these Open Source products. This is in contrast to such projects like Linux, where every kid knows how to compile their own kernel, even if they wouldn't know how to actually write a single line of C.

Anyway, thanks to the experience of the OurDelta project, this didn't block us for more than a week or two. For MariaDB, we will of course make sure that there are both scripts and instructions readily available so that anyone can produce their own builds if they want to.

Especially important since we of course welcome feedback and contributions also in this area, there are some popular Linuxes that OurDelta scripts don't yet cover, but we certainly hope MariaDB packages will become available for them sooner or later! (Yes, this is an invitation to help.)

Caring about Windows users

I personally believe that a key reason to MySQL's success (for instance compared to PostgreSQL) was that it also supported Windows (which PostgreSQL does as of recently). After all, many web developers
- especially in the 90's - would use Windows for their desktop, while deploying on Linux servers. So they needed both. So even if we won't initially support all of the MySQL supported platforms, MariaDB
certainly needs to support Windows if it wants to be a MySQL replacement. This topic was widely discussed in comments to my previous MariaDB related blog post.

With this background in mind, our Windows support started from a relatively low point: After merging XtraDB, MariaDB crashed on startup for Windows. It seems that nodoby had ever used XtraDB on Windows before that. Also in MariaDB, we just hadn't gotten around to running a Buildbot slave on Windows. Also as I told above, we eventually found out that even the stock MySQL would have failures particular to Windows.

While we did fix most of the Windows issues, one thing we don't have is a Windows installer. (The one Sun has for MySQL is not Open Source, it turns out...) So while we are still committed to supporting Windows, we have now decided that it is not proper to delay releasing the Linux binaries anymore and we have now decided to release the Linux binaries as soon as possible, while still continuing to create the Windows binaries.

Is there a happy end...

So this was a long story. (apologies) Does it end well?

It seems so. We now have debs and rpms and they are undergoing final smoke testing and sanity checking. If there are no surprises, you can look for a "recommended for public testing" release on Monty's blog any day now.

PS: Just in case you're wondering: https://www.theanswerbank.co.uk/Phrases-and-Sayings/article/it-isnt-over-until-the-fat-lady-sings/