Yesterday I wrote down the approach used in the MepSQL build system to parameterize the TAR package name produced. Today I will follow up with how the same was done for building DEBs. The motivation is to create a system that can be used flexibly to create packages of any MySQL fork, with any brand name: mepsql-server-*.deb, percona-server-server-*.deb, mariadb-server-*.deb or even just mysql-server-*.deb (which I might do some day).

While yesterday's tricks with the TAR files were rather straightforward, with the process of building DEBs this turns out to be much more challenging. But not to worry, like my former collague Bernhard Ocklin used to say: This is software, anything is of course possible.

I used the debian source archive from Ubuntu as a starting point, as official MySQL doesn't build DEBs at all. When you correctly unpack the weird format this .diff.gz file is in, you end up with a debian/ directory which is supposed to reside at the root of your MySQL source tree. (Note: Not the bzr source tree, but the one unpacked from a released source tar file, the one we built yesterday.) This directory has quite a lot of content:


$ ls -F|cat
additions/
apparmor-profile
changelog
compat
control
copyright
libmysqlclient16.dirs
libmysqlclient16.docs
libmysqlclient16.files
libmysqlclient16.postinst
libmysqlclient-dev.dirs
...
mysql-server-5.1.config
mysql-server-5.1.dirs
mysql-server-5.1.docs
mysql-server-5.1.files
mysql-server-5.1.lintian-overrides
mysql-server-5.1.logcheck.ignore.paranoid
mysql-server-5.1.logcheck.ignore.server
mysql-server-5.1.logcheck.ignore.workstation
mysql-server-5.1.mysql-server.logrotate
mysql-server-5.1.mysql.upstart
mysql-server-5.1.NEWS
mysql-server-5.1.postinst
mysql-server-5.1.postrm
mysql-server-5.1.preinst
mysql-server-5.1.prerm
mysql-server-5.1.py
mysql-server-5.1.README.Debian
mysql-server-5.1.templates
mysql-server-core-5.1.dirs
mysql-server-core-5.1.files
mysql-testsuite.dirs
mysql-testsuite.files
patches/
po/
README.Maintainer
rules
source.lintian-overrides
watch


For each DEB, there are files named *.dirs and *.files which are lists of what files from the MySQL build should be put into that package, as well as pre- and post-installation scripts. You also see other Debian/Ubuntu specific files to be added, like an upstart script. There are additions and patches to be applied to the upstream source TAR.

The control file lists the meta-data for each package to build, descriptions and dependencies. And there is the rules file, which is like a makefile. It is what makes everything happen.

In all of these files, the string "mysql" or "MySQL" is found 3855 times:


$ grep -i mysql * */*|wc -l
3855


Most refer to file names and directories which needn't be changed. Almost half are in the translations:


$ grep -i mysql po/*|wc -l
1747


While it was not done in MepSQL, most of the strings found in the po translations could potentially be "rebranded" to your fork name.

Out of these, the string "mysql" was replaced in 28 places in various files.


$ grep XmysqlforkX * */*|wc -l
28


And 37 filenames:


$ ls *XmysqlforkX*|wc -l
37


Rather than changing the string "mysql" to "mepsql", a generic placeholder "XmysqlforkX" was inserted in those places.

Finally, the control file contains so many references to package names, 50 to be precise:


$ grep mepsql control.mepsql |wc -l
50


...and also considering the fact that when creating a fork one might actually want to edit or rewrite the description texts which are in this file, another approach was needed. Rather than writing XmysqlforkX everywhere in the file, it is possible to just have the complete file customized for your own fork, and save it as control.mepsql (where "mepsql" matches the value of $MYSQLFORK).

In the MepSQL Bakery scripts these template files are stored in a directory debian-5.1/. (The 5.1 is the MySQL version number to build.) Before the build starts, they are copied to the actual debian/ directory, and in the process all occurences of XmysqlforkX are replaced with the value of $MYSQLFORK. The sed and other shell magic needed was eventually quite straightforward, once I had figured out what needs to be changed where:


# New approach: filenames and file contents in the template debian-5.1 dir may
# contain the string "XmysqlforkX". This should be replaced with value of $MYSQLFORK.
mkdir debian
for f in $(find "${BAKERY_BASE_DIR}/debian-${MYVER}" -type f); do
   # Manipulate filename
   newf=$(basename $f | sed "s/XmysqlforkX/$MYSQLFORK/g")
   # Manipulate file contents, save with new filename
   sed "s/XmysqlforkX/$MYSQLFORK/g" $f > debian/$newf
done
# Other directories can be copied as is, there's nothing to manipulate
cp -rf ${BAKERY_BASE_DIR}/debian-${MYVER}/additions ${BAKERY_BASE_DIR}/debian-${MYVER}/patches ${BAKERY_BASE_DIR}/debian-${MYVER}/po debian/
# Except sed is also needed for po/POTFILES.in
sed "s/XmysqlforkX/$MYSQLFORK/g" ${BAKERY_BASE_DIR}/debian-${MYVER}/po/POTFILES.in > debian/po/POTFILES.in
# Finally, it is possible to specify your own files that are not manipulated with
# sed, rather used as is. For instance, you can have control.mepsql, control.mariadb
# etc... and the following will use the file whose suffix matches $MYSQLFORK
for f in $(find "debian/" -name "*.$MYSQLFORK"); do
   # $f includes debian/, but basename strips it away for the second argument.
   mv -f $f debian/$(basename "$f" ".$MYSQLFORK")
done


Going forward...

I have not yet built any RPM files in MepSQL. I expect that the $MYSQLFORK trick will be easier to do in the .spec file than it was in the rather complex debian build system. It remains to be seen.

Working on a build system for MySQL forks, let alone blogging about it, isn't perhaps the sexiest thing there is. But in the work presented in this pair of blog posts I've tried to introduce some new thinking into how packages are built, with the motivation to enable easy code sharing and co-maintenance across the different forks we are working on. I hope that this could encourage others to think of similar ideas in other parts of the code base, and I will of course continue to do so myself too.

Next week I will return with one more post that gives a high level overview of all the layers, from buildbot automation to makefiles, that you come across when building your own MySQL source tree with the MepSQL Bakery.