[JBoss JIRA] Created: (JBAS-6807) Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed)

[JBoss JIRA] Created:...

[JBoss JIRA] Created: (JBAS-6750)...

Radoslav Husar (JIRA)

Monday, 20 April 2009 Mon, 20 Apr '09

12:37 p.m.

Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed) ------------------------------------------------------------------------ Key: JBAS-6807 URL: https://jira.jboss.org/jira/browse/JBAS-6807 Project: JBoss Application Server Issue Type: Bug Security Level: Public (Everyone can see) Components: Clustering Affects Versions: JBossAS-5.1.0.CR1 Environment: any, perf lab, brno lab Reporter: Radoslav Husar Assignee: Brian Stansberry Priority: Blocker A cluster refuses to add more members. Problem detected on Friday 17th/4. http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... Coordinator fails to flush: 12:27:33,209 WARN [GMS] GMS flush by coordinator at 10.34.32.226:55576 failed Joining node logs: 12:33:41,025 WARN [GMS] join(10.34.32.232:39410) sent to 10.34.32.226:55576 timed out (after 3000 ms), retrying Command used to startup: java -server -Xms256m -Xmx1024m -XX:MaxPermSize=256m -Dxb.builder.useUnorderedSequence=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djgroups.udp.ip_ttl=1 -Djava.net.preferIPv4Stack=true -Djboss.messaging.ServerPeerID=`hostname | cut -c "6"` -jar /tmp/jboss-5.1.0.CR1/bin/run.jar -c all -b `hostname` -Djboss.partition.name=SoakTestPartition -Dcluster.route.name=`hostname` -u 234.1.79.154 -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

Show replies by date

Radoslav Husar (JIRA)

Monday, 20 April Mon, 20 Apr

12:41 p.m.

New subject: [JBoss JIRA] Commented: (JBAS-6807) Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed)

[ https://jira.jboss.org/jira/browse/JBAS-6807?page=com.atlassian.jira.plug... ] Radoslav Husar commented on JBAS-6807: -------------------------------------- Brian, as we talked about possible JGroups problem, I downgraded JGroups from 2.6.9 GA (in 5_x) to 2.6.8 GA (in Beta1) but that resulted in the same situation.

...

Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed) ------------------------------------------------------------------------ Key: JBAS-6807 URL: https://jira.jboss.org/jira/browse/JBAS-6807 Project: JBoss Application Server Issue Type: Bug Security Level: Public(Everyone can see) Components: Clustering Affects Versions: JBossAS-5.1.0.CR1 Environment: any, perf lab, brno lab Reporter: Radoslav Husar Assignee: Brian Stansberry Priority: Blocker A cluster refuses to add more members. Problem detected on Friday 17th/4. http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... Coordinator fails to flush: 12:27:33,209 WARN [GMS] GMS flush by coordinator at 10.34.32.226:55576 failed Joining node logs: 12:33:41,025 WARN [GMS] join(10.34.32.232:39410) sent to 10.34.32.226:55576 timed out (after 3000 ms), retrying Command used to startup: java -server -Xms256m -Xmx1024m -XX:MaxPermSize=256m -Dxb.builder.useUnorderedSequence=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djgroups.udp.ip_ttl=1 -Djava.net.preferIPv4Stack=true -Djboss.messaging.ServerPeerID=`hostname | cut -c "6"` -jar /tmp/jboss-5.1.0.CR1/bin/run.jar -c all -b `hostname` -Djboss.partition.name=SoakTestPartition -Dcluster.route.name=`hostname` -u 234.1.79.154

-- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

Brian Stansberry (JIRA)

8:31 p.m.

New subject: [JBoss JIRA] Updated: (JBAS-6807) Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed)

[ https://jira.jboss.org/jira/browse/JBAS-6807?page=com.atlassian.jira.plug... ] Brian Stansberry updated JBAS-6807: ----------------------------------- Fix Version/s: JBossAS-5.1.0.CR1 Affects Version/s: (was: JBossAS-5.1.0.CR1) This may very well be a manifestation of the memory related issues affecting all sorts of AS Branch_5_x hudson testsuite runs over the weekend.

...

Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed) ------------------------------------------------------------------------ Key: JBAS-6807 URL: https://jira.jboss.org/jira/browse/JBAS-6807 Project: JBoss Application Server Issue Type: Bug Security Level: Public(Everyone can see) Components: Clustering Environment: any, perf lab, brno lab Reporter: Radoslav Husar Assignee: Brian Stansberry Priority: Blocker Fix For: JBossAS-5.1.0.CR1 A cluster refuses to add more members. Problem detected on Friday 17th/4. http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... Coordinator fails to flush: 12:27:33,209 WARN [GMS] GMS flush by coordinator at 10.34.32.226:55576 failed Joining node logs: 12:33:41,025 WARN [GMS] join(10.34.32.232:39410) sent to 10.34.32.226:55576 timed out (after 3000 ms), retrying Command used to startup: java -server -Xms256m -Xmx1024m -XX:MaxPermSize=256m -Dxb.builder.useUnorderedSequence=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djgroups.udp.ip_ttl=1 -Djava.net.preferIPv4Stack=true -Djboss.messaging.ServerPeerID=`hostname | cut -c "6"` -jar /tmp/jboss-5.1.0.CR1/bin/run.jar -c all -b `hostname` -Djboss.partition.name=SoakTestPartition -Dcluster.route.name=`hostname` -u 234.1.79.154

Radoslav Husar (JIRA)

Tuesday, 21 April Tue, 21 Apr

3:40 p.m.

New subject: [JBoss JIRA] Commented: (JBAS-6807) Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed)

[ https://jira.jboss.org/jira/browse/JBAS-6807?page=com.atlassian.jira.plug... ] Radoslav Husar commented on JBAS-6807: --------------------------------------

...

I believe r87459 on Branch_5_x, from last Thur, is my last commit that includes anything that would be relevant to these tests. So Rado if you want you can try a build from that revision, plus using the ejb3 core jar I sent you last night.

Yes Brian, reverting to r87459 resolves the JBoss cluster connecting up, but there is some bug in the connector / client and when I call the EJB from the client, it fails in null pointer exception. But that was fixed in the next revisions as of late Friday. So I guess im still stuck regarding the testing.

...

Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed) ------------------------------------------------------------------------ Key: JBAS-6807 URL: https://jira.jboss.org/jira/browse/JBAS-6807 Project: JBoss Application Server Issue Type: Bug Security Level: Public(Everyone can see) Components: Clustering Environment: any, perf lab, brno lab Reporter: Radoslav Husar Assignee: Brian Stansberry Priority: Blocker Fix For: JBossAS-5.1.0.CR1 A cluster refuses to add more members. Problem detected on Friday 17th/4. http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... Coordinator fails to flush: 12:27:33,209 WARN [GMS] GMS flush by coordinator at 10.34.32.226:55576 failed Joining node logs: 12:33:41,025 WARN [GMS] join(10.34.32.232:39410) sent to 10.34.32.226:55576 timed out (after 3000 ms), retrying Command used to startup: java -server -Xms256m -Xmx1024m -XX:MaxPermSize=256m -Dxb.builder.useUnorderedSequence=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djgroups.udp.ip_ttl=1 -Djava.net.preferIPv4Stack=true -Djboss.messaging.ServerPeerID=`hostname | cut -c "6"` -jar /tmp/jboss-5.1.0.CR1/bin/run.jar -c all -b `hostname` -Djboss.partition.name=SoakTestPartition -Dcluster.route.name=`hostname` -u 234.1.79.154

Radoslav Husar (JIRA)

3:46 p.m.

New subject: [JBoss JIRA] Updated: (JBAS-6807) Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed)

[ https://jira.jboss.org/jira/browse/JBAS-6807?page=com.atlassian.jira.plug... ] Radoslav Husar updated JBAS-6807: --------------------------------- Attachment: (was: graph_image.php)

...

Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed) ------------------------------------------------------------------------ Key: JBAS-6807 URL: https://jira.jboss.org/jira/browse/JBAS-6807 Project: JBoss Application Server Issue Type: Bug Security Level: Public(Everyone can see) Components: Clustering Environment: any, perf lab, brno lab Reporter: Radoslav Husar Assignee: Brian Stansberry Priority: Blocker Fix For: JBossAS-5.1.0.CR1 Attachments: ipth.png A cluster refuses to add more members. Problem detected on Friday 17th/4. http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... Coordinator fails to flush: 12:27:33,209 WARN [GMS] GMS flush by coordinator at 10.34.32.226:55576 failed Joining node logs: 12:33:41,025 WARN [GMS] join(10.34.32.232:39410) sent to 10.34.32.226:55576 timed out (after 3000 ms), retrying Command used to startup: java -server -Xms256m -Xmx1024m -XX:MaxPermSize=256m -Dxb.builder.useUnorderedSequence=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djgroups.udp.ip_ttl=1 -Djava.net.preferIPv4Stack=true -Djboss.messaging.ServerPeerID=`hostname | cut -c "6"` -jar /tmp/jboss-5.1.0.CR1/bin/run.jar -c all -b `hostname` -Djboss.partition.name=SoakTestPartition -Dcluster.route.name=`hostname` -u 234.1.79.154

Radoslav Husar (JIRA)

3:46 p.m.

New subject: [JBoss JIRA] Updated: (JBAS-6807) Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed)

[ https://jira.jboss.org/jira/browse/JBAS-6807?page=com.atlassian.jira.plug... ] Radoslav Husar updated JBAS-6807: --------------------------------- Attachment: graph_image.php ipth.png More info: I left 2 "broken" JBoss instances in the cluster and the Brno lab admin noticed that there has been going 20 mbps traffic to everyone in the same virtual LAN. As can be seen from the images, 2 nodes are sending 10 mbps traffic through multicast. 10mbps is quite a lot if we expect them to only send heartbeats :)

...

Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed) ------------------------------------------------------------------------ Key: JBAS-6807 URL: https://jira.jboss.org/jira/browse/JBAS-6807 Project: JBoss Application Server Issue Type: Bug Security Level: Public(Everyone can see) Components: Clustering Environment: any, perf lab, brno lab Reporter: Radoslav Husar Assignee: Brian Stansberry Priority: Blocker Fix For: JBossAS-5.1.0.CR1 Attachments: ipth.png A cluster refuses to add more members. Problem detected on Friday 17th/4. http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... Coordinator fails to flush: 12:27:33,209 WARN [GMS] GMS flush by coordinator at 10.34.32.226:55576 failed Joining node logs: 12:33:41,025 WARN [GMS] join(10.34.32.232:39410) sent to 10.34.32.226:55576 timed out (after 3000 ms), retrying Command used to startup: java -server -Xms256m -Xmx1024m -XX:MaxPermSize=256m -Dxb.builder.useUnorderedSequence=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djgroups.udp.ip_ttl=1 -Djava.net.preferIPv4Stack=true -Djboss.messaging.ServerPeerID=`hostname | cut -c "6"` -jar /tmp/jboss-5.1.0.CR1/bin/run.jar -c all -b `hostname` -Djboss.partition.name=SoakTestPartition -Dcluster.route.name=`hostname` -u 234.1.79.154

Radoslav Husar (JIRA)

3:48 p.m.

New subject: [JBoss JIRA] Updated: (JBAS-6807) Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed)

[ https://jira.jboss.org/jira/browse/JBAS-6807?page=com.atlassian.jira.plug... ] Radoslav Husar updated JBAS-6807: --------------------------------- Attachment: graph_image.gif

...

Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed) ------------------------------------------------------------------------ Key: JBAS-6807 URL: https://jira.jboss.org/jira/browse/JBAS-6807 Project: JBoss Application Server Issue Type: Bug Security Level: Public(Everyone can see) Components: Clustering Environment: any, perf lab, brno lab Reporter: Radoslav Husar Assignee: Brian Stansberry Priority: Blocker Fix For: JBossAS-5.1.0.CR1 Attachments: graph_image.gif, ipth.png A cluster refuses to add more members. Problem detected on Friday 17th/4. http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... Coordinator fails to flush: 12:27:33,209 WARN [GMS] GMS flush by coordinator at 10.34.32.226:55576 failed Joining node logs: 12:33:41,025 WARN [GMS] join(10.34.32.232:39410) sent to 10.34.32.226:55576 timed out (after 3000 ms), retrying Command used to startup: java -server -Xms256m -Xmx1024m -XX:MaxPermSize=256m -Dxb.builder.useUnorderedSequence=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djgroups.udp.ip_ttl=1 -Djava.net.preferIPv4Stack=true -Djboss.messaging.ServerPeerID=`hostname | cut -c "6"` -jar /tmp/jboss-5.1.0.CR1/bin/run.jar -c all -b `hostname` -Djboss.partition.name=SoakTestPartition -Dcluster.route.name=`hostname` -u 234.1.79.154

Brian Stansberry (JIRA)

4:17 p.m.

New subject: [JBoss JIRA] Commented: (JBAS-6807) Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed)

[ https://jira.jboss.org/jira/browse/JBAS-6807?page=com.atlassian.jira.plug... ] Brian Stansberry commented on JBAS-6807: ---------------------------------------- Do you see this is if you run the current head of Branch_5_x, without any modified ejb3-core.jar?

...

Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed) ------------------------------------------------------------------------ Key: JBAS-6807 URL: https://jira.jboss.org/jira/browse/JBAS-6807 Project: JBoss Application Server Issue Type: Bug Security Level: Public(Everyone can see) Components: Clustering Environment: any, perf lab, brno lab Reporter: Radoslav Husar Assignee: Brian Stansberry Priority: Blocker Fix For: JBossAS-5.1.0.CR1 Attachments: graph_image.gif, ipth.png A cluster refuses to add more members. Problem detected on Friday 17th/4. http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... Coordinator fails to flush: 12:27:33,209 WARN [GMS] GMS flush by coordinator at 10.34.32.226:55576 failed Joining node logs: 12:33:41,025 WARN [GMS] join(10.34.32.232:39410) sent to 10.34.32.226:55576 timed out (after 3000 ms), retrying Command used to startup: java -server -Xms256m -Xmx1024m -XX:MaxPermSize=256m -Dxb.builder.useUnorderedSequence=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djgroups.udp.ip_ttl=1 -Djava.net.preferIPv4Stack=true -Djboss.messaging.ServerPeerID=`hostname | cut -c "6"` -jar /tmp/jboss-5.1.0.CR1/bin/run.jar -c all -b `hostname` -Djboss.partition.name=SoakTestPartition -Dcluster.route.name=`hostname` -u 234.1.79.154

Radoslav Husar (JIRA)

5:38 p.m.

New subject: [JBoss JIRA] Commented: (JBAS-6807) Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed)

[ https://jira.jboss.org/jira/browse/JBAS-6807?page=com.atlassian.jira.plug... ] Radoslav Husar commented on JBAS-6807: -------------------------------------- Current revision, Last Changed Rev: 87638 Last Changed Date: 2009-04-21 16:36:49 +0200 (Tue, 21 Apr 2009) pretty much the same thing happening (this time i dont see the excessive traffic as before but that was only 2 nodes, so maybe that is why).

...

Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed) ------------------------------------------------------------------------ Key: JBAS-6807 URL: https://jira.jboss.org/jira/browse/JBAS-6807 Project: JBoss Application Server Issue Type: Bug Security Level: Public(Everyone can see) Components: Clustering Environment: any, perf lab, brno lab Reporter: Radoslav Husar Assignee: Brian Stansberry Priority: Blocker Fix For: JBossAS-5.1.0.CR1 Attachments: graph_image.gif, ipth.png A cluster refuses to add more members. Problem detected on Friday 17th/4. http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... Coordinator fails to flush: 12:27:33,209 WARN [GMS] GMS flush by coordinator at 10.34.32.226:55576 failed Joining node logs: 12:33:41,025 WARN [GMS] join(10.34.32.232:39410) sent to 10.34.32.226:55576 timed out (after 3000 ms), retrying Command used to startup: java -server -Xms256m -Xmx1024m -XX:MaxPermSize=256m -Dxb.builder.useUnorderedSequence=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djgroups.udp.ip_ttl=1 -Djava.net.preferIPv4Stack=true -Djboss.messaging.ServerPeerID=`hostname | cut -c "6"` -jar /tmp/jboss-5.1.0.CR1/bin/run.jar -c all -b `hostname` -Djboss.partition.name=SoakTestPartition -Dcluster.route.name=`hostname` -u 234.1.79.154

Radoslav Husar (JIRA)

5:57 p.m.

New subject: [JBoss JIRA] Commented: (JBAS-6807) Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed)

...

Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed) ------------------------------------------------------------------------ Key: JBAS-6807 URL: https://jira.jboss.org/jira/browse/JBAS-6807 Project: JBoss Application Server Issue Type: Bug Security Level: Public(Everyone can see) Components: Clustering Environment: any, perf lab, brno lab Reporter: Radoslav Husar Assignee: Brian Stansberry Priority: Blocker Fix For: JBossAS-5.1.0.CR1 Attachments: graph_image.gif, ipth.png A cluster refuses to add more members. Problem detected on Friday 17th/4. http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... Coordinator fails to flush: 12:27:33,209 WARN [GMS] GMS flush by coordinator at 10.34.32.226:55576 failed Joining node logs: 12:33:41,025 WARN [GMS] join(10.34.32.232:39410) sent to 10.34.32.226:55576 timed out (after 3000 ms), retrying Command used to startup: java -server -Xms256m -Xmx1024m -XX:MaxPermSize=256m -Dxb.builder.useUnorderedSequence=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djgroups.udp.ip_ttl=1 -Djava.net.preferIPv4Stack=true -Djboss.messaging.ServerPeerID=`hostname | cut -c "6"` -jar /tmp/jboss-5.1.0.CR1/bin/run.jar -c all -b `hostname` -Djboss.partition.name=SoakTestPartition -Dcluster.route.name=`hostname` -u 234.1.79.154

Bela Ban (JIRA)

6:34 p.m.

New subject: [JBoss JIRA] Commented: (JBAS-6807) Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed)

[ https://jira.jboss.org/jira/browse/JBAS-6807?page=com.atlassian.jira.plug... ] Bela Ban commented on JBAS-6807: -------------------------------- Could you capture a bit of that traffic ? Use wireshark if possible, then we can look at the JGroups messages (with Richard's JGroups module). Also, you probably need to run this with some cats enabled at TRACE, e.g. org.jgroups.protocols.pbcast.FLUSH, but I'll let Vladimir comment on that. He can then take a better look. I assume this issue is reproduceable ?

...

Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed) ------------------------------------------------------------------------ Key: JBAS-6807 URL: https://jira.jboss.org/jira/browse/JBAS-6807 Project: JBoss Application Server Issue Type: Bug Security Level: Public(Everyone can see) Components: Clustering Environment: any, perf lab, brno lab Reporter: Radoslav Husar Assignee: Brian Stansberry Priority: Blocker Fix For: JBossAS-5.1.0.CR1 Attachments: graph_image.gif, ipth.png A cluster refuses to add more members. Problem detected on Friday 17th/4. http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... Coordinator fails to flush: 12:27:33,209 WARN [GMS] GMS flush by coordinator at 10.34.32.226:55576 failed Joining node logs: 12:33:41,025 WARN [GMS] join(10.34.32.232:39410) sent to 10.34.32.226:55576 timed out (after 3000 ms), retrying Command used to startup: java -server -Xms256m -Xmx1024m -XX:MaxPermSize=256m -Dxb.builder.useUnorderedSequence=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djgroups.udp.ip_ttl=1 -Djava.net.preferIPv4Stack=true -Djboss.messaging.ServerPeerID=`hostname | cut -c "6"` -jar /tmp/jboss-5.1.0.CR1/bin/run.jar -c all -b `hostname` -Djboss.partition.name=SoakTestPartition -Dcluster.route.name=`hostname` -u 234.1.79.154

Brian Stansberry (JIRA)

6:37 p.m.

New subject: [JBoss JIRA] Assigned: (JBAS-6807) Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed)

[ https://jira.jboss.org/jira/browse/JBAS-6807?page=com.atlassian.jira.plug... ] Brian Stansberry reassigned JBAS-6807: -------------------------------------- Assignee: Vladimir Blagojevic (was: Brian Stansberry) Vlad, I'm assigning this one to you. Per above comments Rado is still seeing this with a build of the current head of JBoss AS Branch_5_x (https://svn.jboss.org/repos/jbossas/branches/Branch_5_x/). The current hudson testsuite runs for that branch now show the tests running normally, so the non-clustering-related AS issues that led me to suggest reverting to r87459 are resolved. So if we're still seeing this it looks more like a JGroups issue, or at least something that needs Vladimir expertise. :)

...

Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed) ------------------------------------------------------------------------ Key: JBAS-6807 URL: https://jira.jboss.org/jira/browse/JBAS-6807 Project: JBoss Application Server Issue Type: Bug Security Level: Public(Everyone can see) Components: Clustering Environment: any, perf lab, brno lab Reporter: Radoslav Husar Assignee: Vladimir Blagojevic Priority: Blocker Fix For: JBossAS-5.1.0.CR1 Attachments: graph_image.gif, ipth.png A cluster refuses to add more members. Problem detected on Friday 17th/4. http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... Coordinator fails to flush: 12:27:33,209 WARN [GMS] GMS flush by coordinator at 10.34.32.226:55576 failed Joining node logs: 12:33:41,025 WARN [GMS] join(10.34.32.232:39410) sent to 10.34.32.226:55576 timed out (after 3000 ms), retrying Command used to startup: java -server -Xms256m -Xmx1024m -XX:MaxPermSize=256m -Dxb.builder.useUnorderedSequence=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djgroups.udp.ip_ttl=1 -Djava.net.preferIPv4Stack=true -Djboss.messaging.ServerPeerID=`hostname | cut -c "6"` -jar /tmp/jboss-5.1.0.CR1/bin/run.jar -c all -b `hostname` -Djboss.partition.name=SoakTestPartition -Dcluster.route.name=`hostname` -u 234.1.79.154

Vladimir Blagojevic (JIRA)

7:11 p.m.

New subject: [JBoss JIRA] Commented: (JBAS-6807) Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed)

[ https://jira.jboss.org/jira/browse/JBAS-6807?page=com.atlassian.jira.plug... ] Vladimir Blagojevic commented on JBAS-6807: ------------------------------------------- Radoslav, can you first try it back in the lab where everything use to work, the network environment you are very familiar with...

...

Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed) ------------------------------------------------------------------------ Key: JBAS-6807 URL: https://jira.jboss.org/jira/browse/JBAS-6807 Project: JBoss Application Server Issue Type: Bug Security Level: Public(Everyone can see) Components: Clustering Environment: any, perf lab, brno lab Reporter: Radoslav Husar Assignee: Vladimir Blagojevic Priority: Blocker Fix For: JBossAS-5.1.0.CR1 Attachments: graph_image.gif, ipth.png A cluster refuses to add more members. Problem detected on Friday 17th/4. http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... Coordinator fails to flush: 12:27:33,209 WARN [GMS] GMS flush by coordinator at 10.34.32.226:55576 failed Joining node logs: 12:33:41,025 WARN [GMS] join(10.34.32.232:39410) sent to 10.34.32.226:55576 timed out (after 3000 ms), retrying Command used to startup: java -server -Xms256m -Xmx1024m -XX:MaxPermSize=256m -Dxb.builder.useUnorderedSequence=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djgroups.udp.ip_ttl=1 -Djava.net.preferIPv4Stack=true -Djboss.messaging.ServerPeerID=`hostname | cut -c "6"` -jar /tmp/jboss-5.1.0.CR1/bin/run.jar -c all -b `hostname` -Djboss.partition.name=SoakTestPartition -Dcluster.route.name=`hostname` -u 234.1.79.154

Brian Stansberry (JIRA)

Thursday, 23 April Thu, 23 Apr

12:45 a.m.

New subject: [JBoss JIRA] Assigned: (JBAS-6807) Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed)

[ https://jira.jboss.org/jira/browse/JBAS-6807?page=com.atlassian.jira.plug... ] Brian Stansberry reassigned JBAS-6807: -------------------------------------- Assignee: Radoslav Husar (was: Vladimir Blagojevic) Rado, can you try this again? We found some code that was handling a group JMX notification by resending it async to the group. With that fixed this problem should go away.

...

Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed) ------------------------------------------------------------------------ Key: JBAS-6807 URL: https://jira.jboss.org/jira/browse/JBAS-6807 Project: JBoss Application Server Issue Type: Bug Security Level: Public(Everyone can see) Components: Clustering Environment: any, perf lab, brno lab Reporter: Radoslav Husar Assignee: Radoslav Husar Priority: Blocker Fix For: JBossAS-5.1.0.CR1 Attachments: graph_image.gif, ipth.png A cluster refuses to add more members. Problem detected on Friday 17th/4. http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... Coordinator fails to flush: 12:27:33,209 WARN [GMS] GMS flush by coordinator at 10.34.32.226:55576 failed Joining node logs: 12:33:41,025 WARN [GMS] join(10.34.32.232:39410) sent to 10.34.32.226:55576 timed out (after 3000 ms), retrying Command used to startup: java -server -Xms256m -Xmx1024m -XX:MaxPermSize=256m -Dxb.builder.useUnorderedSequence=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djgroups.udp.ip_ttl=1 -Djava.net.preferIPv4Stack=true -Djboss.messaging.ServerPeerID=`hostname | cut -c "6"` -jar /tmp/jboss-5.1.0.CR1/bin/run.jar -c all -b `hostname` -Djboss.partition.name=SoakTestPartition -Dcluster.route.name=`hostname` -u 234.1.79.154

Radoslav Husar (JIRA)

4:25 p.m.

New subject: [JBoss JIRA] Commented: (JBAS-6807) Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed)

...

Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed) ------------------------------------------------------------------------ Key: JBAS-6807 URL: https://jira.jboss.org/jira/browse/JBAS-6807 Project: JBoss Application Server Issue Type: Bug Security Level: Public(Everyone can see) Components: Clustering Environment: any, perf lab, brno lab Reporter: Radoslav Husar Assignee: Radoslav Husar Priority: Blocker Fix For: JBossAS-5.1.0.CR1 Attachments: graph_image.gif, ipth.png A cluster refuses to add more members. Problem detected on Friday 17th/4. http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... Coordinator fails to flush: 12:27:33,209 WARN [GMS] GMS flush by coordinator at 10.34.32.226:55576 failed Joining node logs: 12:33:41,025 WARN [GMS] join(10.34.32.232:39410) sent to 10.34.32.226:55576 timed out (after 3000 ms), retrying Command used to startup: java -server -Xms256m -Xmx1024m -XX:MaxPermSize=256m -Dxb.builder.useUnorderedSequence=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djgroups.udp.ip_ttl=1 -Djava.net.preferIPv4Stack=true -Djboss.messaging.ServerPeerID=`hostname | cut -c "6"` -jar /tmp/jboss-5.1.0.CR1/bin/run.jar -c all -b `hostname` -Djboss.partition.name=SoakTestPartition -Dcluster.route.name=`hostname` -u 234.1.79.154

Radoslav Husar (JIRA)

5:40 p.m.

New subject: [JBoss JIRA] Closed: (JBAS-6807) Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed)

[ https://jira.jboss.org/jira/browse/JBAS-6807?page=com.atlassian.jira.plug... ] Radoslav Husar closed JBAS-6807. -------------------------------- Resolution: Done fixed (bug was introduced with fix for JBAS-6792)

...

Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed) ------------------------------------------------------------------------ Key: JBAS-6807 URL: https://jira.jboss.org/jira/browse/JBAS-6807 Project: JBoss Application Server Issue Type: Bug Security Level: Public(Everyone can see) Components: Clustering Environment: any, perf lab, brno lab Reporter: Radoslav Husar Assignee: Radoslav Husar Priority: Blocker Fix For: JBossAS-5.1.0.CR1 Attachments: graph_image.gif, ipth.png A cluster refuses to add more members. Problem detected on Friday 17th/4. http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... http://hudson.qa.jboss.com/hudson/job/as5-soak-ejb3-total-async-failover/... Coordinator fails to flush: 12:27:33,209 WARN [GMS] GMS flush by coordinator at 10.34.32.226:55576 failed Joining node logs: 12:33:41,025 WARN [GMS] join(10.34.32.232:39410) sent to 10.34.32.226:55576 timed out (after 3000 ms), retrying Command used to startup: java -server -Xms256m -Xmx1024m -XX:MaxPermSize=256m -Dxb.builder.useUnorderedSequence=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djgroups.udp.ip_ttl=1 -Djava.net.preferIPv4Stack=true -Djboss.messaging.ServerPeerID=`hostname | cut -c "6"` -jar /tmp/jboss-5.1.0.CR1/bin/run.jar -c all -b `hostname` -Djboss.partition.name=SoakTestPartition -Dcluster.route.name=`hostname` -u 234.1.79.154

5628

days inactive

5631

days old

jboss-jira@lists.jboss.org

Manage subscription

15 comments

4 participants

tags (0)

participants (4)

Bela Ban (JIRA)
Brian Stansberry (JIRA)
Radoslav Husar (JIRA)
Vladimir Blagojevic (JIRA)

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

[JBoss JIRA] Created: (JBAS-6807) Unable to form a cluster Branch_5_x (flush by coordinator at xyz failed)