Geert made us aware that MySQL Cluster now provides the possibility to disable arbitration in order to use an external arbitration mechanism. This is a really important feature, because... well, not really, but only because I was the one who designed it :-)

Coming up with the concept and the two parameters Arbitration=WaitExternal and ArbitrationTimeout=n took a few weeks of discussion. Once we agreed on how to do it, I think JonasMagnus coded it in 20 minutes on the mezzanine floor of the Hyatt, Santa Clara. After that MySQL conference I soon resigned from Sun, so I had now idea what then happened to this feature.

Arbitration is used as a "tie-breaker" in some error scenarios in any cluster. In a cluster, if one of the nodes go down, then the rest of the cluster continues. But if the network and not the nodes is at fault, then you need to decide which nodes should stop and which can continue to be the cluster. In this case you might need an arbitrator to make the decision.

Geert explains this concept well, and how the new parameters can be used. However, it seems the use case when these should be used is not apparent to him. The problem is not if you don't have a server for the arbitrator. In that case you should just get another server, yes, exactly like Geert says.

The problem these parameters solves is in fact the opposite: if you have too many arbitrators.

Consider a simple 2-node cluster with nodes A and B, and the management node M. MySQL Cluster is a fully functional standalone cluster, so you don't need to use some external clustering like Linux Heartbeat or Solaris Cluster. In fact, you cannot do that.

Now consider that on servers A and B you are also running an operating system level cluster like those. In the customer case it was Solaris Cluster that was managing their JBoss server. If the network fails and you have a split brain scenario, then both the MySQL Cluster and the Solaris Cluster will need to execute their arbitration algorithm.

So suppose that MySQL Cluster chooses to run node A, and shutdown node B. Fine, this is how it's supposed to work.

But then suppose that Solaris Cluster chooses the opposite, and shutdowns node A. There is a 50% chance of this happening. You are now left with node A powered down, and Solaris and JBoss running on node B but no MySQL Cluster on node B.

The conclusion of this is that if you have multiple clustering software on the same servers, they actually need to be aware of each other and choose the same nodes when arbitrating.

When thinking about this problem we considered various messaging mechanisms we could create between Solaris Cluster and MySQL Cluster, but they either felt very complex or would not be feasible to implement in the Solaris Cluster framework. In the end we realized the simplest solution is just what was created now. If you set Arbitration=WaitExternal then instead of actually doing some arbitration, the MySQL Cluster will just halt for a specified number of seconds. The idea is that during this period the other clustering software (like Solaris Cluster) can then power down (or actually panic) whichever nodes it chooses. After the specified time has passed, the surviving MySQL Cluster nodes will just continue to run from where they left off.

So it is worth reiterating that the point of these parameters is not that you should use some external cluster software for arbitration, they are there only if you really must coordinate with an already existing arbitrator.