Hi all, I am the sysadmin working with Rishabh on this.
I'd like to add that during the Tomcat startup we see two SYN packets from node A to node B being dropped by the local firewall (i.e. iptables) on an apparently random port (e.g. DPT=58317), another six from node A to node B being dropped using a different apparently random port (e.g. DPT=48800), and finally two SYN packets from node B to node A (i.e. in the remote firewall, e.g. DPT=53675).
We got everything to work by following the instructions in the following document: http://community.jboss.org/wiki/RunTwoInstanceOfJBossInFedora which suggests we just allow all TCP and UDP ports between the two nodes. That is, however, not ideal from our security policy perspective!