This is the first part of many posts in a series of blog posts where I want to document how the MepSQL packages were built. By doing that, I will also end up covering the MariaDB build system (which this is based on), some of BuildBot, Amazon EC2 cloud and packaging DEBs and RPMs just in general, so it could be interesting from many perspectives. In this first part I'll simply scribble some notes about reviewing the OpenSuse Build System, Launchpad PPA service vs using your own servers and automating the builds with BuildBot.
Originally I just wanted to work on some new ideas on the automated build and QA system used by MariaDB. But since leaving Monty Program I didn't have access to any of those servers anymore, so as a first step I had to look into what alternatives there are for building binary packages for many operating systems and hardware platforms. In fact, this was another thing I had wanted to learn more about for a while. For instance Michal Hrušecký uses OpenSuse Build Service to build both MySQL and MariaDB packages for all RPM based distributions in the blink of an eye - I was interested to find out what's behind that magic.
The OpenSuse Build Service is a free service that allows you to submit an RPM source package (and separately a Debian source package to build Debs for Debian and Ubuntu) and it will automatically build packages for recent versions of all the popular (and even not so popular anymore) RPM based distributions. It also hosts the repository which is another important service - a co-located server is actually quite expensive to rent if the project in question is just a hobby of a private person.
While OBS is an OpenSuse creation and service, the code is available as open source, and in fact other projects like MeeGo are using it for their own build service.
The only thing missing with the OBS is one pretty important one: I didn't find a way to also test the packages it produces. So I could use OBS to build packages, but then to see if they actually work I'd still need access to an instance of all versions of Red Hat, Fedora, CentOS, Suse, Mandriva, Debian and Ubuntu - in both 32-bit and 64-bit variations even! Since publishing untested binaries isn't really an option, this omission in OBS is quite severe and for me - a showstopper.
Launchpad provides Personal Package Archives service, where you can submit a DEB source package and get packages built for all recent Ubuntu versions. Also PPA will host the debs so people can download them.
While it's understandable that Canonical focuses on making it easy to build packages for Ubuntu, it's not sufficient as a general solution. PPA also doesn't include a testing service, just like OBS. I had a look at how the guys at Canonical build the official Ubuntu debs, and it seems they run "make test" as part of the build. This however seems pointless to me: yes, it tests that the MySQL code passes the tests, but it doesn't test the very thing they should be testing: whether their own packaging script works!
So the conclusion on reviewing OBS and PPA: Please include a possibility to test whether the packages actually work, that would make your service much more usable!
Update: Andreas Jaeger comments below, that OBS already does a test install of the packages but doesn't then run any tests (that's already something!) and the implementation of actually testing the installation is in a planning and discussion phase (URL below).
So this left me with the option of staying with the BuildBot based system running on my own servers, since it is the only alternative where I can also configure the system to actually test the packages being built. This introduced a new problem: As I didn't have a server available, should I buy one or should I use the trendy cloud services? This is the topic for the next post.
PS: A word about Jenkins is in order: In addition to BuildBot I'm aware of many open source projects using Jenkins (formerly known as Hudson) for the same purpose - Drizzle is one of them. From the point of view I had in this post Jenkins can be considered equivalent to BuildBot: Both include running the system on your own servers. One consideration I had was that "all else being equal" I would choose to use the same system as MariaDB uses, so that any improvements could potentially also benefit MariaDB (and any other MySQL fork as well). Of course, Drizzle is also a MySQL fork so theoretically I could have still gone with Jenkins, but with Drizzle being more different than the other MySQL forks are from each other, the likelihood for more synergy was larger with deriving from the MariaDB system. (Plus, I was already familiar with that system.)
- Log in to post comments
- 25118 views
Testing?
You tripped over an important missing point in OBS but it's not as bad as you think ;)
You can run as part of packaging in OBS the testsuite - and most packages do this. We are also thinking how to handle this more general, for details check http://en.opensuse.org/openSUSE:Build_Service_Concept_QA
Any ideas - and patches ;) - on how to improve are welcome!
Good to know it is being
Good to know it is being worked on! I noticed Canonical guys doing "make test" before the binary is then packaged into debs. But to me that's useless: By this time I have already separately run these very tests so that I'm confident I should create packages now. What I need to test is the rpm and deb packages - do they install, does MySQL start, does it work? MariaDB does that with it's BuildBot setup. With my MepSQL setup those tests are actually disabled - I want to change the MariaDB tests so that packages are actually fetched from the staging apt/yum repository (which MariaDB doesn't have yet).
OBS installs packages as part
OBS installs packages as part of the building, so it builds the software, runs it's testsuite (if the spec file says so), packages everything together and then installs the packages. So, that basic test is done. You're right, it does not test whether really everything is setup and MepSQL would work indeed...
What you want could be done with an extra test package - and the ideas our developers and QA guys discuss (see URL I gave).
I had a look at the URL, but
I had a look at the URL, but it looked like something in the planning stage still.
If the packages are at least installed, that's at least something. It still leaves some areas untested, for instance I've built packages that install fine but MySQL doesn't start. (Upstart related...)
By test package you mean I could run simple tests from the post install script in a dummy package? Or do you refer to the ideas yet to be implemented?
Test package
With a test package I mean a package that instead of building software, tests the installed system. So, it would require the MySQL package, check that it has started, and run tests. The created package would then contain only the log file of the test instead of software itself.
Sounds like an excellent
Sounds like an excellent idea!
Now that I think about it, knowing that OBS at least does a test install, I could have abused the current system to implement basic tests just to know that MySQL server is running and can be connected to. I could add a dummy package which depends on all the other ones, so it is installed last, and from the pre- or post-install scripts I could do something like "mysql -e "SELECT 1 FROM TEST" test;" and then fail the script if this fails.
Re: automated buildbot testing of *.rpm
I'm currently actively switching to automated testing
if full distro installs, headed for upgrades and then --rollback
testing under buildbot's.
What I'm doing is here
http://eastham.rpm5.org:8011
for Mandriva-derived distros (I've done perhaps 20 waterfalls
across various distros from RHEL4/s390x to rolling distros
where the packages change daily/hourly while boring in
on the process that will be needed to keep up.
If you already know how to use buildbot's, what I'm
doing is remarkable only in the fact that it involves
Gb's of packages stored locally in order to avoid
the time cost of downloading huge mirrors with modest bandwidth.
If you want to attempt mepSQL installs/upgrades under my
waterfalls down the road (like May or June) I can likely accomodate.
ATM, the link above is targeted at a May 15th end-of-three year
project called Mancoosi, which needs the waterfalls for package
--rollback testing. After that the waterfalls will likely be available
for other purposes.
hth
73 de Jeff
Interesting idea...
Interesting idea. What is your motivation here? a) to test the building of a distribution or b) to test rpm5 itself with lots of input data?
I've yet to actually add the rpm based distros to the MepSQL buildbot, but that is only some weeks away (2 tasks before that, and this is a moonlight project). MepSQL server as a MySQL fork is just one thing, there's quite a lot of interesting utilities out there that aren't currently packaged by anyone. June as a timeframe of getting to that point sounds good, I might be in touch!
I have learned important
I have learned important stuff by reading this article.
Buildbot infrastructure for *.rpm upgrade testing
Largely FYI (because none of the code is finished yet) is now
standalone in CVS here
cvs -d ':pserver:anonymous@rpm5.org:/v/rpm/cvs' get mancoosi
Note active dead-run sprint development. I gotta deploy *something*
in the next week to start collecting data. SO there ain't much of
interest to see.
But the technique(s) of using buildbot's to automate upgrade testing
are perfectly sane and obvious even if there's a whopping amount of
storage needed with "bloated" linux distro mirrors that have to be
locally cached. I filled up ~1Tb of *.rpm crud in ~2 weeks with rsync.
Sure I could wade in an clean up the mess ... cheaper/easier to
buy another disk because there's no way to change mirror infrastructure,
and its rather costly (for me) to maintain custom rsync scripts and
that assumes I know what I need (which isn't the case in "dependency hell")