As I hinted yesterday I've tried to automate the deployment of a sharded MongoDB cluster in Amazon. It's unnecessarily difficult (rumor has it 10gen is doing something about it in the future) but it's doable with appropriate amount of persistence.

There are many reasons why this was more difficult than it should be. Creating a database cluster is always harder than launching, for example, a Tomcat server. A web server is stateless, so you just use some tool to say "launch 5 EC2 servers and install these RPMs on all of them". Easy to do with puppet, chef and a bash script. A database cluster on the other hand is not a bunch of independent cowboys, rather the nodes need to talk to each other and in the case of sharding or master-slave replication (both of which MongoDB does) they need to agree on different roles.

On Amazon by default you get a different hostname and ip address every time. This makes for example the default puppet approach useless, which is based on identifying the nodes by hostname. The only tool I've found that really addresses this problem is Amazon CloudFormation. It allows you to describe a deployment without having to know in advance their hostnames. CloudFormation sucks in 15 other ways, but currently it's the only tool I know that is usable from this perspective.

MongoDB manual already provides CloudFormation templates to deploy a MongoDB Replica Set. As you can see, it contains functionality like Fn:GetAtt where you can reference the hostname of another node:


"cat /tmp/replicaSetConfigInit.js\n",
"config = {_id: \"", { "Ref" : "ReplicaSetName" } ,"\", members : [",
"{_id : 0, host:\"$HOSTNAME:27017\"},",
"{_id : 1, host:\"", { "Fn::GetAtt" : ["ReplicaSetMember1", "Outputs.ReplicaSetMemberName"] },":27017\"},",
"{_id : 2, host:\"", { "Fn::GetAtt" : ["ReplicaSetMember2", "Outputs.ReplicaSetMemberName"] },":27017\"}",
"]};\n",
"rs.initiate(config);\n",
"EOF\n",


I could write a separate blog post, maybe even a small book, on all the flaws of CloudFormation, but in the above snippet you already see one antipattern: Embedding a bash script as strings in a json file is a terrible idea. It's also unnecessary, but seems to be widespread practice. Sysadmins are terrible coders. Anyways...

Alas, nobody has posted templates to also deploy shards, but it was simple enough to use the above examples to also deploy mongos and mongoc nodes and then have a top-level template that orchestrates the building of a whole sharded cluster. My template launches 3 shards, so that's a total of 15 nodes. In production it would include a backup node per shard, so 18 nodes. (Sorry, I would like to post my templates here, but the process of getting approval to publish them is harder than for someone else to just rewrite them.)

Except for one thing. At least in the 2.2 version the MongoDB rpm's do not actually contain init scripts or config files for starting mongoc and mongos processes. WTF? It almost makes me think sharding isn't seriously supported. So in my template I have to manually copy such files into place too from a git repo I googled (mongos.init, mongoc.init).

This worked to launch a 15 node MongoDB cluster. And it would have been great except for one thing. What happens when a node eventually fails and needs to be replaced? Well, CloudFormation is completely oblivious to such things. So you have to copy paste from the AutoScalingKeepAtNSample.template posted on the AWS CloudFormation templates page. Note: The point here is not to do any scaling, rather the AutoScaling group is configured to keep the amount of nodes at exactly 3 - and this includes when one node terminates, another is started to replace it.

AutoScaling has another limitation: In an AutoScaling group, all nodes are clones. They get exactly the same meta data, as far as I could see. Well jeez... The need for nodes to do different things was the reason to use CloudFormation in the first place. Now we are back to square one.

In the end I concluded I need to take matters into my own hands, as the AWS tooling is still inadequate on this point, alternatively you can also blame MongoDB for not making this very easy. So I wrote a bash script to the rescue. It's only a proof of concept, and for example will not recover from error conditions.

So what you need to do to deploy a sharded MongoDB with AWS tooling is the following:

1. Using AWS CloudFormation we will deploy each Replica Set as an AutoScaling group. (Need for AutoScaling is a workaround due to CloudFormation limitations.)
2. A Replica Set is launched as N nodes that belong to the AutoScaling groups. N >= 3. The N nodes are identical with each other. (AutoScaling limitation.)
3. To create a Replica Set, we need to pick one of the nodes as the first one to call rs.initialize() and he will then connect the other nodes to himself. Note that due to the previous point, it is not trivial to pick one node that will execute a different sequence of actions. (MongoDB limitation - creating a replica set is not symmetric operation across all of the members.)
4. A starting node can use the config database, available via MongoS nodes, to check whether a shard already exists or whether he should initialize and add it. If the shard already exists in the cluster, he can also find out a hostname of a current member and connect to it to join the shard. (MongoDB limitation - for example Galera would allow to specify a list of potentially available members.)
5. If the shard does not yet exist, the node can proceed to call rs.initialize() and after that call sh.addShard(). However, a race condition exists: rs.initialize() is a very slow operation so the risk is high that all N nodes would proceed to initialize themselves. Further, there is no easy way to undo replica set initialization, so if two or more nodes would call rs.initialize() at the same time, the latter is essentially useless after that. (MongoDB limitation.)
6. To ensure that only one node proceeds to initialization, nodes have to compete for a lock. We write this lock into our own collection in the config database on MongoS nodes. (MongoDB limitation)

The shell script is attached to this post.

It's not pretty, and everyone involved should feel a bit embarrassed.

Getting to this point has taken me about 2 months of work (more in calendar time). The main reason it is so slow is CloudFormation. It is really easy to make mistakes with it, and each iteration is really slow (30-60 minutes per iteration, in this case).

In practice we have pretty much decided to abandon CloudFormation and will use one of some 3rd party tools to deploy our stack instead. (One of Cloudify, juju, Scalr, Severalnines, etc...) The bash script was really finished just to understand the problem and prove it can be solved, but I don't currently conisder CloudFormation a maintainable approach.