4:21 a.m.
FC: all threads are blocked at credits_available.await
------------------------------------------------------
Key: JGRP-1272
URL: https://issues.jboss.org/browse/JGRP-1272
Project: JGroups
Issue Type: Bug
Affects Versions: 2.11
Environment: ubuntu 10.04.1
Reporter: Victor N
Assignee: Bela Ban
Attachments: jgroups-tcp.xml
After several days of working one node can not rejoin because of FC protocol issue. The
situation is reproduced once or several times per week.
The problematic node is called "gate9", its view is outdated:
[gate10.mydomain|1175] [gate10.mydomain, gate7.mydomain, gate8.mydomain, gate9.mydomain,
gate11.mydomain, gate14.mydomain, gate5.mydomain, gate12.mydomain, gate2.mydomain,
gate4.mydomain, gate3.mydomain, gate6.mydomain]
Actual view (seen by all other nodes) is: [gate10.mydomain|1176] [gate10.mydomain,
gate7.mydomain, gate8.mydomain, gate11.mydomain, gate14.mydomain, gate5.mydomain,
gate12.mydomain, gate2.mydomain, gate4.mydomain, gate3.mydomain, gate6.mydomain]
In log file at gate9 I can see that it sends CREDIT_REQUEST constantly:
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate10.mydomain, src=gate9.mydomain, headers are FC: CREDIT_REQUEST, UNICAST: DATA,
seqno=4308, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.BasicTCP.sendUnicast():
dest=192.168.1.10:40001 (94 bytes)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.UNICAST.retransmit():
gate9.mydomain --> XMIT(gate10.mydomain: #2014)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate10.mydomain, src=gate9.mydomain, headers are FC: CREDIT_REQUEST, UNICAST: DATA,
seqno=2014, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.BasicTCP.sendUnicast():
dest=192.168.1.10:40001 (94 bytes)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.UNICAST.retransmit():
gate9.mydomain --> XMIT(gate10.mydomain: #1124)
...
At gate10 (the coordinator) I see that it receives the requests and responds:
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.passMessageUp(): received [dst:
gate10.mydomain, src: gate9.mydomain (3 headers), size=9 bytes], headers are FC:
CREDIT_REQUEST, UNICAST: DATA, seqno=2397, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.UNICAST.handleDataReceived():
gate10.mydomain <-- DATA(gate9.mydomain: #2397, conn_id=14)
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.UNICAST.sendRequestForFirstSeqno():
gate10.mydomain --> SEND_FIRST_SEQNO(gate9.mydomain)
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate9.mydomain, src=gate10.mydomain, headers are UNICAST: SEND_FIRST_SEQNO, seqno=0, TCP:
[channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.passMessageUp(): received [dst:
gate10.mydomain, src: gate9.mydomain (3 headers), size=9 bytes], headers are FC:
CREDIT_REQUEST, UNICAST: DATA, seqno=1202, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.UNICAST.handleDataReceived():
gate10.mydomain <-- DATA(gate9.mydomain: #1202, conn_id=14)
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.UNICAST.sendRequestForFirstSeqno():
gate10.mydomain --> SEND_FIRST_SEQNO(gate9.mydomain)
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate9.mydomain, src=gate10.mydomain, headers are UNICAST: SEND_FIRST_SEQNO, seqno=0, TCP:
[channel_name=GateCluster]
...
but I do not see any answer to this at gate9.
My config and stack traces are attached below. I do not know how to reproduce the problem
in tests. But it occurs, so I can provide you with any details, just let me know.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:32 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-1272) FC: all threads are blocked at credits_available.await
[
https://issues.jboss.org/browse/JGRP-1272?page=com.atlassian.jira.plugin....
]
Victor N updated JGRP-1272:
---------------------------
Attachment: s1.txt
stack traces at the problematic gate9
FC: all threads are blocked at credits_available.await
------------------------------------------------------
Key: JGRP-1272
URL: https://issues.jboss.org/browse/JGRP-1272
Project: JGroups
Issue Type: Bug
Affects Versions: 2.11
Environment: ubuntu 10.04.1
Reporter: Victor N
Assignee: Bela Ban
Attachments: jgroups-tcp.xml, s1.txt
After several days of working one node can not rejoin because of FC protocol issue. The
situation is reproduced once or several times per week.
The problematic node is called "gate9", its view is outdated:
[gate10.mydomain|1175] [gate10.mydomain, gate7.mydomain, gate8.mydomain, gate9.mydomain,
gate11.mydomain, gate14.mydomain, gate5.mydomain, gate12.mydomain, gate2.mydomain,
gate4.mydomain, gate3.mydomain, gate6.mydomain]
Actual view (seen by all other nodes) is: [gate10.mydomain|1176] [gate10.mydomain,
gate7.mydomain, gate8.mydomain, gate11.mydomain, gate14.mydomain, gate5.mydomain,
gate12.mydomain, gate2.mydomain, gate4.mydomain, gate3.mydomain, gate6.mydomain]
In log file at gate9 I can see that it sends CREDIT_REQUEST constantly:
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate10.mydomain, src=gate9.mydomain, headers are FC: CREDIT_REQUEST, UNICAST: DATA,
seqno=4308, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.BasicTCP.sendUnicast():
dest=192.168.1.10:40001 (94 bytes)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.UNICAST.retransmit():
gate9.mydomain --> XMIT(gate10.mydomain: #2014)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate10.mydomain, src=gate9.mydomain, headers are FC: CREDIT_REQUEST, UNICAST: DATA,
seqno=2014, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.BasicTCP.sendUnicast():
dest=192.168.1.10:40001 (94 bytes)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.UNICAST.retransmit():
gate9.mydomain --> XMIT(gate10.mydomain: #1124)
...
At gate10 (the coordinator) I see that it receives the requests and responds:
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.passMessageUp(): received [dst:
gate10.mydomain, src: gate9.mydomain (3 headers), size=9 bytes], headers are FC:
CREDIT_REQUEST, UNICAST: DATA, seqno=2397, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.UNICAST.handleDataReceived():
gate10.mydomain <-- DATA(gate9.mydomain: #2397, conn_id=14)
17/01/2011 09:55:55 UTC| TRACE |
org.jgroups.protocols.UNICAST.sendRequestForFirstSeqno(): gate10.mydomain -->
SEND_FIRST_SEQNO(gate9.mydomain)
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate9.mydomain, src=gate10.mydomain, headers are UNICAST: SEND_FIRST_SEQNO, seqno=0, TCP:
[channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.passMessageUp(): received [dst:
gate10.mydomain, src: gate9.mydomain (3 headers), size=9 bytes], headers are FC:
CREDIT_REQUEST, UNICAST: DATA, seqno=1202, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.UNICAST.handleDataReceived():
gate10.mydomain <-- DATA(gate9.mydomain: #1202, conn_id=14)
17/01/2011 09:55:55 UTC| TRACE |
org.jgroups.protocols.UNICAST.sendRequestForFirstSeqno(): gate10.mydomain -->
SEND_FIRST_SEQNO(gate9.mydomain)
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate9.mydomain, src=gate10.mydomain, headers are UNICAST: SEND_FIRST_SEQNO, seqno=0, TCP:
[channel_name=GateCluster]
...
but I do not see any answer to this at gate9.
My config and stack traces are attached below. I do not know how to reproduce the problem
in tests. But it occurs, so I can provide you with any details, just let me know.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:42 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1272) FC: all threads are blocked at credits_available.await
[
https://issues.jboss.org/browse/JGRP-1272?page=com.atlassian.jira.plugin....
]
Victor N commented on JGRP-1272:
--------------------------------
As it was with JGRP-1265, the problem is reproduced after a full garbage collection. Rigth
after that I see the following at gate9:
17/01/2011 02:46:28 UTC| WARN | org.jgroups.protocols.FD.up(): I was suspected by
gate8.mydomain; ignoring the SUSPECT message and sending back a HEARTBEAT_ACK
17/01/2011 02:46:34 UTC| WARN | org.jgroups.protocols.pbcast.GMS.installView():
gate9.mydomain: not member of view [gate10.mydomain|1176] [gate10.mydomain,
gate7.mydomain, gate8.mydomain, gate11.mydomain, gate14.mydomain, gate5.mydomain,
gate12.mydomain, gate2.mydomain, gate4.mydomain, gate3.mydomain, gate6.mydomain];
discarding it
17/01/2011 07:49:22 UTC| ERROR | org.jgroups.protocols.TP.down(): failed sending message
to gate10.mydomain (91 bytes): java.lang.IllegalStateException: Queue full
17/01/2011 07:49:27 UTC| ERROR | org.jgroups.protocols.TP.down(): failed sending message
to gate10.mydomain (91 bytes): java.lang.IllegalStateException: Queue full
...
FC: all threads are blocked at credits_available.await
------------------------------------------------------
Key: JGRP-1272
URL: https://issues.jboss.org/browse/JGRP-1272
Project: JGroups
Issue Type: Bug
Affects Versions: 2.11
Environment: ubuntu 10.04.1
Reporter: Victor N
Assignee: Bela Ban
Attachments: jgroups-tcp.xml, s1.txt
After several days of working one node can not rejoin because of FC protocol issue. The
situation is reproduced once or several times per week.
The problematic node is called "gate9", its view is outdated:
[gate10.mydomain|1175] [gate10.mydomain, gate7.mydomain, gate8.mydomain, gate9.mydomain,
gate11.mydomain, gate14.mydomain, gate5.mydomain, gate12.mydomain, gate2.mydomain,
gate4.mydomain, gate3.mydomain, gate6.mydomain]
Actual view (seen by all other nodes) is: [gate10.mydomain|1176] [gate10.mydomain,
gate7.mydomain, gate8.mydomain, gate11.mydomain, gate14.mydomain, gate5.mydomain,
gate12.mydomain, gate2.mydomain, gate4.mydomain, gate3.mydomain, gate6.mydomain]
In log file at gate9 I can see that it sends CREDIT_REQUEST constantly:
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate10.mydomain, src=gate9.mydomain, headers are FC: CREDIT_REQUEST, UNICAST: DATA,
seqno=4308, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.BasicTCP.sendUnicast():
dest=192.168.1.10:40001 (94 bytes)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.UNICAST.retransmit():
gate9.mydomain --> XMIT(gate10.mydomain: #2014)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate10.mydomain, src=gate9.mydomain, headers are FC: CREDIT_REQUEST, UNICAST: DATA,
seqno=2014, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.BasicTCP.sendUnicast():
dest=192.168.1.10:40001 (94 bytes)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.UNICAST.retransmit():
gate9.mydomain --> XMIT(gate10.mydomain: #1124)
...
At gate10 (the coordinator) I see that it receives the requests and responds:
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.passMessageUp(): received [dst:
gate10.mydomain, src: gate9.mydomain (3 headers), size=9 bytes], headers are FC:
CREDIT_REQUEST, UNICAST: DATA, seqno=2397, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.UNICAST.handleDataReceived():
gate10.mydomain <-- DATA(gate9.mydomain: #2397, conn_id=14)
17/01/2011 09:55:55 UTC| TRACE |
org.jgroups.protocols.UNICAST.sendRequestForFirstSeqno(): gate10.mydomain -->
SEND_FIRST_SEQNO(gate9.mydomain)
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate9.mydomain, src=gate10.mydomain, headers are UNICAST: SEND_FIRST_SEQNO, seqno=0, TCP:
[channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.passMessageUp(): received [dst:
gate10.mydomain, src: gate9.mydomain (3 headers), size=9 bytes], headers are FC:
CREDIT_REQUEST, UNICAST: DATA, seqno=1202, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.UNICAST.handleDataReceived():
gate10.mydomain <-- DATA(gate9.mydomain: #1202, conn_id=14)
17/01/2011 09:55:55 UTC| TRACE |
org.jgroups.protocols.UNICAST.sendRequestForFirstSeqno(): gate10.mydomain -->
SEND_FIRST_SEQNO(gate9.mydomain)
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate9.mydomain, src=gate10.mydomain, headers are UNICAST: SEND_FIRST_SEQNO, seqno=0, TCP:
[channel_name=GateCluster]
...
but I do not see any answer to this at gate9.
My config and stack traces are attached below. I do not know how to reproduce the problem
in tests. But it occurs, so I can provide you with any details, just let me know.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:22 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1272) FC: all threads are blocked at credits_available.await
[
https://issues.jboss.org/browse/JGRP-1272?page=com.atlassian.jira.plugin....
]
Bela Ban commented on JGRP-1272:
--------------------------------
Rather than creating a JIRA issue right away, we should discuss this on the mailing list..
the way it looks I'm going to close this one too, as it's not reproduceable, like
your other case.
Having said that, there are a few things you could try out (again, we should discuss that
on jg-users):
- FILE_PING.num_initial_members should be more than 1, I suggest set it to 10 for now
- Set TCP.logical_addr_max_size to 500
- Set TCP.logical_addr_cache_expiration to 600000
- Try the latest JGroups (2.12) from GitHub github: github.com/belaban/JGroups
FC: all threads are blocked at credits_available.await
------------------------------------------------------
Key: JGRP-1272
URL: https://issues.jboss.org/browse/JGRP-1272
Project: JGroups
Issue Type: Bug
Affects Versions: 2.11
Environment: ubuntu 10.04.1
Reporter: Victor N
Assignee: Bela Ban
Attachments: jgroups-tcp.xml, s1.txt
After several days of working one node can not rejoin because of FC protocol issue. The
situation is reproduced once or several times per week.
The problematic node is called "gate9", its view is outdated:
[gate10.mydomain|1175] [gate10.mydomain, gate7.mydomain, gate8.mydomain, gate9.mydomain,
gate11.mydomain, gate14.mydomain, gate5.mydomain, gate12.mydomain, gate2.mydomain,
gate4.mydomain, gate3.mydomain, gate6.mydomain]
Actual view (seen by all other nodes) is: [gate10.mydomain|1176] [gate10.mydomain,
gate7.mydomain, gate8.mydomain, gate11.mydomain, gate14.mydomain, gate5.mydomain,
gate12.mydomain, gate2.mydomain, gate4.mydomain, gate3.mydomain, gate6.mydomain]
In log file at gate9 I can see that it sends CREDIT_REQUEST constantly:
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate10.mydomain, src=gate9.mydomain, headers are FC: CREDIT_REQUEST, UNICAST: DATA,
seqno=4308, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.BasicTCP.sendUnicast():
dest=192.168.1.10:40001 (94 bytes)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.UNICAST.retransmit():
gate9.mydomain --> XMIT(gate10.mydomain: #2014)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate10.mydomain, src=gate9.mydomain, headers are FC: CREDIT_REQUEST, UNICAST: DATA,
seqno=2014, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.BasicTCP.sendUnicast():
dest=192.168.1.10:40001 (94 bytes)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.UNICAST.retransmit():
gate9.mydomain --> XMIT(gate10.mydomain: #1124)
...
At gate10 (the coordinator) I see that it receives the requests and responds:
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.passMessageUp(): received [dst:
gate10.mydomain, src: gate9.mydomain (3 headers), size=9 bytes], headers are FC:
CREDIT_REQUEST, UNICAST: DATA, seqno=2397, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.UNICAST.handleDataReceived():
gate10.mydomain <-- DATA(gate9.mydomain: #2397, conn_id=14)
17/01/2011 09:55:55 UTC| TRACE |
org.jgroups.protocols.UNICAST.sendRequestForFirstSeqno(): gate10.mydomain -->
SEND_FIRST_SEQNO(gate9.mydomain)
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate9.mydomain, src=gate10.mydomain, headers are UNICAST: SEND_FIRST_SEQNO, seqno=0, TCP:
[channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.passMessageUp(): received [dst:
gate10.mydomain, src: gate9.mydomain (3 headers), size=9 bytes], headers are FC:
CREDIT_REQUEST, UNICAST: DATA, seqno=1202, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.UNICAST.handleDataReceived():
gate10.mydomain <-- DATA(gate9.mydomain: #1202, conn_id=14)
17/01/2011 09:55:55 UTC| TRACE |
org.jgroups.protocols.UNICAST.sendRequestForFirstSeqno(): gate10.mydomain -->
SEND_FIRST_SEQNO(gate9.mydomain)
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate9.mydomain, src=gate10.mydomain, headers are UNICAST: SEND_FIRST_SEQNO, seqno=0, TCP:
[channel_name=GateCluster]
...
but I do not see any answer to this at gate9.
My config and stack traces are attached below. I do not know how to reproduce the problem
in tests. But it occurs, so I can provide you with any details, just let me know.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:01 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1272) FC: all threads are blocked at credits_available.await
[
https://issues.jboss.org/browse/JGRP-1272?page=com.atlassian.jira.plugin....
]
Bela Ban commented on JGRP-1272:
--------------------------------
There was a comment by Adrian Tarau on jg-users [1], saying that the indefinite blocking
could possibly be caused by a bug in the JDK.
The original mailing thread is on concurrency-interest [2]
[1]
http://sourceforge.net/mailarchive/forum.php?thread_name=4D344F95.9070408...
[2] http://cs.oswego.edu/pipermail/concurrency-interest/2011-January/007665.html
FC: all threads are blocked at credits_available.await
------------------------------------------------------
Key: JGRP-1272
URL: https://issues.jboss.org/browse/JGRP-1272
Project: JGroups
Issue Type: Bug
Affects Versions: 2.11
Environment: ubuntu 10.04.1
Reporter: Victor N
Assignee: Bela Ban
Attachments: jgroups-tcp.xml, s1.txt
After several days of working one node can not rejoin because of FC protocol issue. The
situation is reproduced once or several times per week.
The problematic node is called "gate9", its view is outdated:
[gate10.mydomain|1175] [gate10.mydomain, gate7.mydomain, gate8.mydomain, gate9.mydomain,
gate11.mydomain, gate14.mydomain, gate5.mydomain, gate12.mydomain, gate2.mydomain,
gate4.mydomain, gate3.mydomain, gate6.mydomain]
Actual view (seen by all other nodes) is: [gate10.mydomain|1176] [gate10.mydomain,
gate7.mydomain, gate8.mydomain, gate11.mydomain, gate14.mydomain, gate5.mydomain,
gate12.mydomain, gate2.mydomain, gate4.mydomain, gate3.mydomain, gate6.mydomain]
In log file at gate9 I can see that it sends CREDIT_REQUEST constantly:
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate10.mydomain, src=gate9.mydomain, headers are FC: CREDIT_REQUEST, UNICAST: DATA,
seqno=4308, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.BasicTCP.sendUnicast():
dest=192.168.1.10:40001 (94 bytes)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.UNICAST.retransmit():
gate9.mydomain --> XMIT(gate10.mydomain: #2014)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate10.mydomain, src=gate9.mydomain, headers are FC: CREDIT_REQUEST, UNICAST: DATA,
seqno=2014, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.BasicTCP.sendUnicast():
dest=192.168.1.10:40001 (94 bytes)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.UNICAST.retransmit():
gate9.mydomain --> XMIT(gate10.mydomain: #1124)
...
At gate10 (the coordinator) I see that it receives the requests and responds:
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.passMessageUp(): received [dst:
gate10.mydomain, src: gate9.mydomain (3 headers), size=9 bytes], headers are FC:
CREDIT_REQUEST, UNICAST: DATA, seqno=2397, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.UNICAST.handleDataReceived():
gate10.mydomain <-- DATA(gate9.mydomain: #2397, conn_id=14)
17/01/2011 09:55:55 UTC| TRACE |
org.jgroups.protocols.UNICAST.sendRequestForFirstSeqno(): gate10.mydomain -->
SEND_FIRST_SEQNO(gate9.mydomain)
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate9.mydomain, src=gate10.mydomain, headers are UNICAST: SEND_FIRST_SEQNO, seqno=0, TCP:
[channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.passMessageUp(): received [dst:
gate10.mydomain, src: gate9.mydomain (3 headers), size=9 bytes], headers are FC:
CREDIT_REQUEST, UNICAST: DATA, seqno=1202, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.UNICAST.handleDataReceived():
gate10.mydomain <-- DATA(gate9.mydomain: #1202, conn_id=14)
17/01/2011 09:55:55 UTC| TRACE |
org.jgroups.protocols.UNICAST.sendRequestForFirstSeqno(): gate10.mydomain -->
SEND_FIRST_SEQNO(gate9.mydomain)
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate9.mydomain, src=gate10.mydomain, headers are UNICAST: SEND_FIRST_SEQNO, seqno=0, TCP:
[channel_name=GateCluster]
...
but I do not see any answer to this at gate9.
My config and stack traces are attached below. I do not know how to reproduce the problem
in tests. But it occurs, so I can provide you with any details, just let me know.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:19 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-1272) FC: all threads are blocked at credits_available.await
[
https://issues.jboss.org/browse/JGRP-1272?page=com.atlassian.jira.plugin....
]
Victor N updated JGRP-1272:
---------------------------
Environment:
CentOS release 5.2 (Final)
kernel 2.6.27.24rootserver-20090525a
JDK 1.6.0_22 64-bit
was:ubuntu 10.04.1
FC: all threads are blocked at credits_available.await
------------------------------------------------------
Key: JGRP-1272
URL: https://issues.jboss.org/browse/JGRP-1272
Project: JGroups
Issue Type: Bug
Affects Versions: 2.11
Environment: CentOS release 5.2 (Final)
kernel 2.6.27.24rootserver-20090525a
JDK 1.6.0_22 64-bit
Reporter: Victor N
Assignee: Bela Ban
Attachments: jgroups-tcp.xml, s1.txt
After several days of working one node can not rejoin because of FC protocol issue. The
situation is reproduced once or several times per week.
The problematic node is called "gate9", its view is outdated:
[gate10.mydomain|1175] [gate10.mydomain, gate7.mydomain, gate8.mydomain, gate9.mydomain,
gate11.mydomain, gate14.mydomain, gate5.mydomain, gate12.mydomain, gate2.mydomain,
gate4.mydomain, gate3.mydomain, gate6.mydomain]
Actual view (seen by all other nodes) is: [gate10.mydomain|1176] [gate10.mydomain,
gate7.mydomain, gate8.mydomain, gate11.mydomain, gate14.mydomain, gate5.mydomain,
gate12.mydomain, gate2.mydomain, gate4.mydomain, gate3.mydomain, gate6.mydomain]
In log file at gate9 I can see that it sends CREDIT_REQUEST constantly:
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate10.mydomain, src=gate9.mydomain, headers are FC: CREDIT_REQUEST, UNICAST: DATA,
seqno=4308, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.BasicTCP.sendUnicast():
dest=192.168.1.10:40001 (94 bytes)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.UNICAST.retransmit():
gate9.mydomain --> XMIT(gate10.mydomain: #2014)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate10.mydomain, src=gate9.mydomain, headers are FC: CREDIT_REQUEST, UNICAST: DATA,
seqno=2014, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.BasicTCP.sendUnicast():
dest=192.168.1.10:40001 (94 bytes)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.UNICAST.retransmit():
gate9.mydomain --> XMIT(gate10.mydomain: #1124)
...
At gate10 (the coordinator) I see that it receives the requests and responds:
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.passMessageUp(): received [dst:
gate10.mydomain, src: gate9.mydomain (3 headers), size=9 bytes], headers are FC:
CREDIT_REQUEST, UNICAST: DATA, seqno=2397, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.UNICAST.handleDataReceived():
gate10.mydomain <-- DATA(gate9.mydomain: #2397, conn_id=14)
17/01/2011 09:55:55 UTC| TRACE |
org.jgroups.protocols.UNICAST.sendRequestForFirstSeqno(): gate10.mydomain -->
SEND_FIRST_SEQNO(gate9.mydomain)
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate9.mydomain, src=gate10.mydomain, headers are UNICAST: SEND_FIRST_SEQNO, seqno=0, TCP:
[channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.passMessageUp(): received [dst:
gate10.mydomain, src: gate9.mydomain (3 headers), size=9 bytes], headers are FC:
CREDIT_REQUEST, UNICAST: DATA, seqno=1202, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.UNICAST.handleDataReceived():
gate10.mydomain <-- DATA(gate9.mydomain: #1202, conn_id=14)
17/01/2011 09:55:55 UTC| TRACE |
org.jgroups.protocols.UNICAST.sendRequestForFirstSeqno(): gate10.mydomain -->
SEND_FIRST_SEQNO(gate9.mydomain)
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate9.mydomain, src=gate10.mydomain, headers are UNICAST: SEND_FIRST_SEQNO, seqno=0, TCP:
[channel_name=GateCluster]
...
but I do not see any answer to this at gate9.
My config and stack traces are attached below. I do not know how to reproduce the problem
in tests. But it occurs, so I can provide you with any details, just let me know.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:37 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1272) FC: all threads are blocked at credits_available.await
[
https://issues.jboss.org/browse/JGRP-1272?page=com.atlassian.jira.plugin....
]
Bela Ban commented on JGRP-1272:
--------------------------------
I don't think the bug in the JVM causes this; it looks rather that the blocked senders
don't get any credits from a node which has network issues. Possibly a
logical_addr_cache issue ? Trace on TCP will show...
FC: all threads are blocked at credits_available.await
------------------------------------------------------
Key: JGRP-1272
URL: https://issues.jboss.org/browse/JGRP-1272
Project: JGroups
Issue Type: Bug
Affects Versions: 2.11
Environment: CentOS release 5.2 (Final)
kernel 2.6.27.24rootserver-20090525a
JDK 1.6.0_22 64-bit
Reporter: Victor N
Assignee: Bela Ban
Attachments: jgroups-tcp.xml, s1.txt
After several days of working one node can not rejoin because of FC protocol issue. The
situation is reproduced once or several times per week.
The problematic node is called "gate9", its view is outdated:
[gate10.mydomain|1175] [gate10.mydomain, gate7.mydomain, gate8.mydomain, gate9.mydomain,
gate11.mydomain, gate14.mydomain, gate5.mydomain, gate12.mydomain, gate2.mydomain,
gate4.mydomain, gate3.mydomain, gate6.mydomain]
Actual view (seen by all other nodes) is: [gate10.mydomain|1176] [gate10.mydomain,
gate7.mydomain, gate8.mydomain, gate11.mydomain, gate14.mydomain, gate5.mydomain,
gate12.mydomain, gate2.mydomain, gate4.mydomain, gate3.mydomain, gate6.mydomain]
In log file at gate9 I can see that it sends CREDIT_REQUEST constantly:
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate10.mydomain, src=gate9.mydomain, headers are FC: CREDIT_REQUEST, UNICAST: DATA,
seqno=4308, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.BasicTCP.sendUnicast():
dest=192.168.1.10:40001 (94 bytes)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.UNICAST.retransmit():
gate9.mydomain --> XMIT(gate10.mydomain: #2014)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate10.mydomain, src=gate9.mydomain, headers are FC: CREDIT_REQUEST, UNICAST: DATA,
seqno=2014, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.BasicTCP.sendUnicast():
dest=192.168.1.10:40001 (94 bytes)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.UNICAST.retransmit():
gate9.mydomain --> XMIT(gate10.mydomain: #1124)
...
At gate10 (the coordinator) I see that it receives the requests and responds:
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.passMessageUp(): received [dst:
gate10.mydomain, src: gate9.mydomain (3 headers), size=9 bytes], headers are FC:
CREDIT_REQUEST, UNICAST: DATA, seqno=2397, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.UNICAST.handleDataReceived():
gate10.mydomain <-- DATA(gate9.mydomain: #2397, conn_id=14)
17/01/2011 09:55:55 UTC| TRACE |
org.jgroups.protocols.UNICAST.sendRequestForFirstSeqno(): gate10.mydomain -->
SEND_FIRST_SEQNO(gate9.mydomain)
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate9.mydomain, src=gate10.mydomain, headers are UNICAST: SEND_FIRST_SEQNO, seqno=0, TCP:
[channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.passMessageUp(): received [dst:
gate10.mydomain, src: gate9.mydomain (3 headers), size=9 bytes], headers are FC:
CREDIT_REQUEST, UNICAST: DATA, seqno=1202, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.UNICAST.handleDataReceived():
gate10.mydomain <-- DATA(gate9.mydomain: #1202, conn_id=14)
17/01/2011 09:55:55 UTC| TRACE |
org.jgroups.protocols.UNICAST.sendRequestForFirstSeqno(): gate10.mydomain -->
SEND_FIRST_SEQNO(gate9.mydomain)
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate9.mydomain, src=gate10.mydomain, headers are UNICAST: SEND_FIRST_SEQNO, seqno=0, TCP:
[channel_name=GateCluster]
...
but I do not see any answer to this at gate9.
My config and stack traces are attached below. I do not know how to reproduce the problem
in tests. But it occurs, so I can provide you with any details, just let me know.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:31 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1272) FC: all threads are blocked at credits_available.await
[
https://issues.jboss.org/browse/JGRP-1272?page=com.atlassian.jira.plugin....
]
Victor N commented on JGRP-1272:
--------------------------------
Bela, the problem is reproduced with the following increased params:
FILE_PING.num_initial_members=10
TCP.logical_addr_cache_max_size=200
TCP.logical_addr_cache_expiration=600000
Yes, it was after a network failure.
I can catch a tcpdump on that server right now and send it to you.
FC: all threads are blocked at credits_available.await
------------------------------------------------------
Key: JGRP-1272
URL: https://issues.jboss.org/browse/JGRP-1272
Project: JGroups
Issue Type: Bug
Affects Versions: 2.11
Environment: CentOS release 5.2 (Final)
kernel 2.6.27.24rootserver-20090525a
JDK 1.6.0_22 64-bit
Reporter: Victor N
Assignee: Bela Ban
Attachments: jgroups-tcp.xml, s1.txt
After several days of working one node can not rejoin because of FC protocol issue. The
situation is reproduced once or several times per week.
The problematic node is called "gate9", its view is outdated:
[gate10.mydomain|1175] [gate10.mydomain, gate7.mydomain, gate8.mydomain, gate9.mydomain,
gate11.mydomain, gate14.mydomain, gate5.mydomain, gate12.mydomain, gate2.mydomain,
gate4.mydomain, gate3.mydomain, gate6.mydomain]
Actual view (seen by all other nodes) is: [gate10.mydomain|1176] [gate10.mydomain,
gate7.mydomain, gate8.mydomain, gate11.mydomain, gate14.mydomain, gate5.mydomain,
gate12.mydomain, gate2.mydomain, gate4.mydomain, gate3.mydomain, gate6.mydomain]
In log file at gate9 I can see that it sends CREDIT_REQUEST constantly:
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate10.mydomain, src=gate9.mydomain, headers are FC: CREDIT_REQUEST, UNICAST: DATA,
seqno=4308, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.BasicTCP.sendUnicast():
dest=192.168.1.10:40001 (94 bytes)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.UNICAST.retransmit():
gate9.mydomain --> XMIT(gate10.mydomain: #2014)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate10.mydomain, src=gate9.mydomain, headers are FC: CREDIT_REQUEST, UNICAST: DATA,
seqno=2014, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.BasicTCP.sendUnicast():
dest=192.168.1.10:40001 (94 bytes)
17/01/2011 09:55:52 UTC| TRACE | org.jgroups.protocols.UNICAST.retransmit():
gate9.mydomain --> XMIT(gate10.mydomain: #1124)
...
At gate10 (the coordinator) I see that it receives the requests and responds:
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.passMessageUp(): received [dst:
gate10.mydomain, src: gate9.mydomain (3 headers), size=9 bytes], headers are FC:
CREDIT_REQUEST, UNICAST: DATA, seqno=2397, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.UNICAST.handleDataReceived():
gate10.mydomain <-- DATA(gate9.mydomain: #2397, conn_id=14)
17/01/2011 09:55:55 UTC| TRACE |
org.jgroups.protocols.UNICAST.sendRequestForFirstSeqno(): gate10.mydomain -->
SEND_FIRST_SEQNO(gate9.mydomain)
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate9.mydomain, src=gate10.mydomain, headers are UNICAST: SEND_FIRST_SEQNO, seqno=0, TCP:
[channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.passMessageUp(): received [dst:
gate10.mydomain, src: gate9.mydomain (3 headers), size=9 bytes], headers are FC:
CREDIT_REQUEST, UNICAST: DATA, seqno=1202, conn_id=14, TCP: [channel_name=GateCluster]
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.UNICAST.handleDataReceived():
gate10.mydomain <-- DATA(gate9.mydomain: #1202, conn_id=14)
17/01/2011 09:55:55 UTC| TRACE |
org.jgroups.protocols.UNICAST.sendRequestForFirstSeqno(): gate10.mydomain -->
SEND_FIRST_SEQNO(gate9.mydomain)
17/01/2011 09:55:55 UTC| TRACE | org.jgroups.protocols.TP.down(): sending msg to
gate9.mydomain, src=gate10.mydomain, headers are UNICAST: SEND_FIRST_SEQNO, seqno=0, TCP:
[channel_name=GateCluster]
...
but I do not see any answer to this at gate9.
My config and stack traces are attached below. I do not know how to reproduce the problem
in tests. But it occurs, so I can provide you with any details, just let me know.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4913
days inactive
4921
days old
7 comments
2 participants
participants (2)
-
Bela Ban (JIRA) -
Victor N (JIRA)