[Clustering/JBoss] - Cluster is loosing his master
by bernd.koecke
Hi everyone,
we have the problem that our cluster sometimes looses his master and only restarting the 'right' node solves it. We are using jboss 4.0.2 but the code which causes the problem is present up to 4.2.0.CR1. I can reproduce it as follows:
cluster with two nodes: node_a, node_b, both jboss-4.0.2
node_a is master, clusterview is node_a:1199,node_b:1199
we have a high workload on both nodes
node_b has problems and doesn't shut down normally, we have to kill him with kill -9 and restart it immediately
node_bnew (node_bn) rejoins the cluster before node_a recognices that node_bold (node_bo) is realy dead
result is the current clusterview: node_a:1199, node_b(o):1199,node_b(n):1199
after a very short time, node_a sends a new clusterview without node_b(o), but node_b(n) doesn't remove his dead predecessor
on node_bn the old node_bo still remains in the replicants list of org.jboss.ha.framework.server.DistributedReplicantManagerImpl and this is never changed.
when now node_a is restarted normaly, node_bn is the first node in the clusterview and JGroups knows he is the master, but the DistributedReplicantManagerImpl not
only when node_bn is restarted, too, the 'right' node from above, all works fine again
otherwise we don't have a master in our cluster. This means e.g. that isMasterReplica of HAJNDI returns 'false' on all nodes and code which should run on the master is never triggered
The reason for this behaviour comes from generateUniqueNodeName in class ClusterPartition. It uses <node_name>:<jndi_port> if the JNDI-Subsystem is started and <node_name>:<generated UID> otherwise. The first form is not unique after a restart of a node and it is set as additional data in JGroups and then used as a key in some HashMaps. Because of this ambiguous key you see a node in the clusterview twice and because of the implementation of DistributedReplicantManagerImpl.setCurrentState and HAPartitionImpl.getDeadMembers the dead old node is never removed from the replicants list of DistributedReplicantsManagerImpl. And this means, that for all deployed services the call to isMasterReplica is false, even if node_bn is the master.
I solved the problem by commenting out the return statement in ClusterPartition.generateUniqueNodeName which returns the name with JNDI port and then the method generates a UID and gives back <node_name>:<generated UID>. This is unique, but the drawback is, that now the clusterview looks a little bit ugly.
In my opinion this is a bug, the unique node name is not unique. But may be it should not be unique across node restarts. Would it be possible, to make it a config option in ClusterPartition to always use the UID, which is off by default? Then if someone want's a complete unique node name can configure it and for all others nothing changes.
After fixing this, I get a new minor problem: There is a deadlock condition in the locking order of the synchronized blocks in DistributedReplicantManagerImpl. I had to move the notifyKeyListeners outside of the synchronized(replicants)-block in purgeDeadMemebers and remember the changed keys in an ArrayList. All other methods have a locking order of localReplicants -> replicants and only purgeDeadMembers uses replicants -> localReplicants. I see that in newer versions of this class the collections from the concurrent-package are used, so this problem may not be present there.
This posting is long enough ;), so if someone wants additional data, like line numbers, diffs etc. I can post it here. It would be nice, if the config option could be integrated and I would be glad to help to do this :).
Best regards
Bernd
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4035942#4035942
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4035942
19 years