Some time ago Stephen O'Grady and Brian Aker had an interesting Blogo-dialogue about what they call the "Cambrian Explosion" of open source development. The Cambrian Explosion means that we increasingly see forks of projects being developed in different directions, where traditionally we are used to open source development happening in relatively hierarchical and easy to follow upstream and downstream relationships. This is exactly what happens in the MySQL community currently, where in total there is more progress than ever before, but that progress is divided among several competing forks, none of which is strictly in an upstream-downstream hierarchy with each other.
I used to be a bit frustrated about this state of affairs, believing that if at least most of the forks could co-operate on a common tree, we would see even faster progress. But when I left MariaDB some months ago, I realized that the situation is what it is and since all existing MySQL variants are associated with a commercial vendor, there wasn't an obvious choice for me to continue contributing, which I still want to do. So I thought I might as well embrace this Cambrian Explosion thing and just publish my own fork as a contribution to the community.
And this is where I today announce the new MepSQL project and the immediate availability of MepSQL 5.1.52-alpha1.
What you'll find inside
There were several things I've wanted to hack on for a long time, which came together nicely in the MepSQL project. (I will blog about all of these in more detail in separate blog posts going forward - even hope to speak about this project at (un)conferences in 2011.)
- BuildBot for automated integration testing. Arguably the single most important reason MySQL quality has finally improved in the recent releases is the Pushbuild system MySQL engineers use internally at Oracle. This was developed some years ago by a Kristian Nielsen. The idea of continuous integration testing is that when a developer commits new code, it will automatically be run against the test-suite which by now has thousands of tests, and the developer then gets a report of failing tests. The Pushbuild system is not publicly available, but MariaDB uses an open source system based on BuildBot. The MariaDB build and QA system was developed by... Kristian Nielsen! Needless to say, I was always a big fan of Kristian's work when working with him, and I wanted to spend some time of my paternity leave working hands on with this system myself. Building MepSQL was an excellent task for such a project.
- Since I no longer work at Monty's company, I didn't have access to the servers where the buildbot installation actually runs. So a part of the project was to take the MariaDB build system and migrate it so I could run it on Amazon's EC2 cloud. This resulted in a new Launchpad project MepSQL Bakery. If you want to build some MySQL derived branch yourself, you can now do so for only a few bucks charged to your credit card!
- Kristian sometimes complained that the management of 70 (!) different KVM images used to build and test MariaDB for all the different Linux distributions and other operating systems out there requires more manual work than it should. To automate this in the MepSQL system I created a shell script used for bootstrapping a virtual server instance on boot. So when halfway through the project I had to add a build dependency to all of the BuildBot slaves, I didn't need to touch my AMI images at all, rather I just added one line of code to the shell script I used. I call this bootstrapping script "Loitsut" and this will be spun off as an independent project as this method is useful for anyone running servers in EC2 or any other cloud too.
- The first task was just to migrate all of this to EC2 and build MariaDB itself. But from there I raised the bar and wanted to build something more useful - after all MariaDB downloads are already delivered by MariaDB. So I pointed the system at the MySQL at Facebook repository which is what Mark Callaghan's team at Facebook publishes. These features typically find their ways into the commercial MySQL variants quite well nowadays, given some months or even years. But the newest features were always available only in this source code repository. So my hope is that providing binary packages of this MySQL fork (and soon also documenting them!) more adventurous MySQL users will be able to easily try them out, discuss them and just get excited about the great work the guys at Facebook keep on doing! I believe this is the first time anyone is offering this code as binary downloads, and this is my way of continuing to contribute to the MySQL ecosystem.
- Finally, while the first release is an exact clone of the mysqlatfacebook Launchpad project, there were some small fixes needed to makefiles and BUILD/compile-dist in order to build downloadable tar files (and deb files) instead of just going directly to make install.
Going forward
Today the downloads work, but are still somewhat rough around the edges. I'm releasing all this following the "release early, release often" mantra of open source development. So in the short term there is a list of obvious tasks to follow:
- DEB packages are ready and will be available after testing.
- RPM packages will be built after that.
- There are some things still to fix in the current packages, it seems for instance a default buildbot installation produces packages with too little permissions, so MySQL doesn't start unless you first do chmod -R a+rx. There are other things too.
- The way the installation packages as well as the running server identifies themselves still needs spit and polish, now it just says "5.1.52" which is not correct. (A result of this is, you probably cannot upgrade from the current packages, but must uninstall them when the next packages are available.)
- The packages should be GPG signed and checksums generated.
- All the new features need to be documented.
- There might be a nicer website, especially when there is more content.
- Finally one might test whether 32 bit platforms would work, or even windows.
- After all this is done, there are still dozens of plugins, patches and utilities floating around the MySQL community that can be added for easy consumption. (Of course, nothing prevents anyone from developing new code either!)
- Log in to post comments
- 49002 views
I think official MySQL,
I think official MySQL, Percona Server or MariaDB are all much better sources for this than the Facebook patch. The target consumer for the Facebook patch is MySQL, Percona Server or MariaDB. The target consumer is not an end user.
But I like all of the other things you have done here.
What to include in MepSQL
Yes, I picked that out from our previous exchange too. I suppose the explanation is the same as with most open source projects: I wanted to look more closely into your work, so that's what I built. If there are others like me, they'll find it interesting, if not, then not.
It was also about doing something suitably challenging: Just buidling one of the already existing forks was easy (in the process I built both MariaDB and MySQL with the system). I wanted to take a "raw" source base and see if I can make it build. And I could, but there was 4-5 late nights of tweaking to be done, so it was a great learning experience.
Wrt to MySQL, Percona and MariaDB I'm with you. I think what I've relased here is supposed to be somewhere between you and them. Like a tech preview of something that is expected to become mainstream later down the road. For instance if one looks at semi-sync replication, it is now, what, 2-3 years since Google developed it and only now is it available in MySQL 5.5. And even now I actually don't see any blogs of people using it or trying it out, discussing pro's, con's and tradeoffs (I remember Johan Andersson trying it, that's all). So a useful thing that MepSQL could potentially become is a kind of bridge between you writing the code and the feature becoming mainstream via the vendor provided forks. It could speed up that process.
But in the end, this is just about something I (and potentially other like minded people?) want to do. Once I'm done with the MySQL at Facebook code, I'll pull in some new code from somewhere else and play with that and provide it as packages as a "side effect".
As an example of what I mean
As an example of what I mean above, it is not many days/weeks of work for someone like Percona to cherry pick a feature from your branch if a customer asks (and pays) for it. So a scenario could be that someone plays with MepSQL, likes something, then sponsors it to be swiftly included in Percona Server, then runs it in production.
MepSQL is a hobby project, without 24/7 support guarantees or end of life policies. The intent is not to compete with 3 already existing vendor led forks - that would be pointless. The intent is to add something more.
Hi! I'd take a serious look
Hi!
I'd take a serious look at Jenkins (ie formally Hudson), http://jenkins-ci.org/. Buildbot is a real pain to setup and maintain, the guys who wrote Jenkins have an excellent interface, and have really done a lot of work to polish it up.
Cheers,
-Brian
BuildBot vs Jenkins
I'm aware of Jenkins (née Hudson), but it wasn't considered for this project. The main motivation for that was that I wanted to stick with whatever MariaDB was using so that the things I do could potentially be copied back to MariaDB.
Other than that I'm interested to learn more about Jenkins, but to actually start using it I would only do it in sync together with the MariaDB team. (At least that's the thinking for now...)
Good luck!
While a single person is probably not enough to hold up a real fork, and while I will most probably not use a one man's fork on production, I fail to understand the rejection of the idea that someone should want to work on his own fork of MySQL. It's as if there's an automatic "Evil" mark on anyone not working on the baseline.
I would expect a single man to actually achieve good results, with possibly cleaning up the build system, or adding interesting patches and plugins. These may later propagate back to the baseline or to other forks.
So this is my Thumbs Up and encouragement on behalf of myself as a community member. Good luck!
Who said this was evil? Its
Who said this was evil? Its as if you are putting words in our mouths.
The build system has been cleaned up 5.5. Someone did an awesome job with cmake.
I don't expect a single person to accomplish much on such a large project. A smaller scope -- improving packaging, automated testing, writing plugins, enhancing HandlerSocket -- would be interesting to me.
Mark, by all means, not referring to you :)
I'm referring to the people who immediately vote "thumb down" on planet MySQL. Just look at the amount of "votes down".
I apologize if I wasn't clear about that and if I made you feel my writing was targeted at you.
re: Good luck!
Well, history will tell us whether this is a one man project or not :-) But the intent is not to be a fork in the sense of going in a different direction, I've built MepSQL to have a platform where I can participate in the MySQL community, not to go away from it!
At least for now I also probably wouldn't use this in production anywhere that I can influence. Otoh I'm interested in possibly including it in comparative benchmarks, learn about how slocket works, etc... That said, I wouldn't be surprised to see adventurous geeks run it in production one day too :-) If Domas can run this code on Wikipedia, why should we stop others from trying their luck with the same?
Btw, where have you seen such "one man" criticism?
I did not see a "one man" criticism
I saw a "MySQL fork" criticism, as I see often enough. The "one man" is my own personal observation.
You have a good point about Domas' workings in Wikipedia. I'm not too familiar with his work, I'll admit to my shame.
Ahh, the thumbs down on
Ahh, the thumbs down on planetmysql... I learned to ignore those already when I was at MariaDB, but to be honest I'm a bit surprised to see them again.
At MariaDB we used to get routinely 6 thumbs down for anything posted. It was within 24 hours and only on office time (so if you posted on a weekend, the 24 hours would count from Monday). After a round of Oracle layoffs the number of thumbs down decreased, which again confirmed that it was just handful in the MySQL team routinely voting down what they saw as competition. They didn't even read the blogs (or understand them?), since a few of my (positive) MySQL Cluster related blogs got the same thumbs down treatment too :-) After more people left the MySQL team, this phenomenon disappeared.
The interesting question with these thumbs are that hypothetically it could be the MariaDB team now doing the same. Would be kinda ironic! We'll never know.
One thing we do know is that the thumbs feature on the planet has proven to be a source of much negative energy in the community. Those who actively use it - especially the downvoting - tend to vote against their competition, rather than about their content. Since most readers don't vote at all, this behavior tends to dominate. MySQL could do a service to itself by just removing the feature, since it doesn't work as intended.
Maybe MariaDB should figure
Maybe MariaDB should figure out what things Domas needs and port them to MariaDB. I am surprised this has not been done given their interesting in seeing MariaDB used by WP.