2:45 a.m.
Problems with large return value and FC
---------------------------------------
Key: JGRP-610
URL: http://jira.jboss.com/jira/browse/JGRP-610
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.6
When sending a return value whose size is larger than FC.max_credits, the call hangs. Unit
test RpcDispatcher.testLargeReturnValueUnicastCall() reproduces the issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:38 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-610) Problems with large return value and FC
[ http://jira.jboss.com/jira/browse/JGRP-610?page=comments#action_12384500 ]
Bela Ban commented on JGRP-610:
-------------------------------
Might be a UNICAST retransmission issue in connection with FRAG(2). If FRAG2 mangles the
(unicast) headers, then we have an issue in unicast.
Problems with large return value and FC
---------------------------------------
Key: JGRP-610
URL: http://jira.jboss.com/jira/browse/JGRP-610
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.6
When sending a return value whose size is larger than FC.max_credits, the call hangs.
Unit test RpcDispatcher.testLargeReturnValueUnicastCall() reproduces the issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:40 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-610) Problems with large return value and FC
[ http://jira.jboss.com/jira/browse/JGRP-610?page=comments#action_12384501 ]
Bela Ban commented on JGRP-610:
-------------------------------
Not possible: FRAG2 is on top of UNICAST ! But possibly UNICAST's retransmission
mangles the fragmentation IDs (FragHeader) ?
Problems with large return value and FC
---------------------------------------
Key: JGRP-610
URL: http://jira.jboss.com/jira/browse/JGRP-610
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.6
When sending a return value whose size is larger than FC.max_credits, the call hangs.
Unit test RpcDispatcher.testLargeReturnValueUnicastCall() reproduces the issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:19 p.m.
New subject: [JBoss JIRA] Commented: (JGRP-610) Problems with large return value and FC
[ http://jira.jboss.com/jira/browse/JGRP-610?page=comments#action_12384639 ]
vivek v commented on JGRP-610:
------------------------------
Could this be related to - JGRP-590?
We have also seen this problem on our system, when a large return value (when using
RpcDispatcher) never gets to the caller and the caller always times out (with
TimeoutException).
(http://sourceforge.net/mailarchive/message.php?msg_id=a206fb7e0710042312g...)
We use Gossip Router with UDP.
Problems with large return value and FC
---------------------------------------
Key: JGRP-610
URL: http://jira.jboss.com/jira/browse/JGRP-610
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.6
When sending a return value whose size is larger than FC.max_credits, the call hangs.
Unit test RpcDispatcher.testLargeReturnValueUnicastCall() reproduces the issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:42 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-610) Problems with large return value and FC
[ http://jira.jboss.com/jira/browse/JGRP-610?page=comments#action_12385051 ]
Bela Ban commented on JGRP-610:
-------------------------------
The problem is that, when a synchronous call is received via UNICAST, the thread acquires
a caller-based lock in UNICAST and then dispatches the request up the stack.
When the same thread is used to send a large return value, FRAG2 splits the response
message into many smaller messages, which then get sent to the sender (self in
RpcDispatcher.testLargeReturnValue()), thus blocking FC.
The thread pool for regular threads is blocked and all other messages (even OOB ones !)
are sitting in the IP buffers and don't even make it to the transport. This means,
credits (sent via OOB messages) are not received and we just block.
Solution (pick one or more):
- In the regular thread pool:
- Use a queue
- Increase the max thread
- Set rejection policy to "discard" rather than "run"
- Increase FC.max_credits to a value greater than the largest return value. Dangerous, as
this increases time without blocks
I'll keep looking for a better solution
Problems with large return value and FC
---------------------------------------
Key: JGRP-610
URL: http://jira.jboss.com/jira/browse/JGRP-610
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.6
When sending a return value whose size is larger than FC.max_credits, the call hangs.
Unit test RpcDispatcher.testLargeReturnValueUnicastCall() reproduces the issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:55 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-610) Problems with large return value and FC
[ http://jira.jboss.com/jira/browse/JGRP-610?page=comments#action_12385326 ]
Bela Ban commented on JGRP-610:
-------------------------------
Detailed explanation:
- A unicast is received. This acquires a (sender-based) lockin UNICAST on its way up
- When the message is received, the RpcDispatcher sends the large return value back *on
the same thread* (the lock in UNICAST is not released)
- FRAG2 splits the return message into many smaller messages
- All messages are sent down, but when max_credits is exceeded, FC.down() blocks. This
means the lock in UNICAST is *still* not released
- On the receiver side (sender == receiver), incoming calls cannot reach FC due to the
lock in UNICAST (because FC is somewhere above
UNICAST), therefore FC will not send new credits (via OOB) to unblock the sender
- The only way credits will be received is for the sender to periodically (max_block_time
ms) send an OOB credit request and unblock when it
receives the credit response
- However, even OOB messages will not get processed when the default thread pool in TP has
the queue *disabled* and all threads are *busy* !
Problems with large return value and FC
---------------------------------------
Key: JGRP-610
URL: http://jira.jboss.com/jira/browse/JGRP-610
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.6
When sending a return value whose size is larger than FC.max_credits, the call hangs.
Unit test RpcDispatcher.testLargeReturnValueUnicastCall() reproduces the issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:15 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-610) Problems with large return value and FC
[ http://jira.jboss.com/jira/browse/JGRP-610?page=comments#action_12385713 ]
Bela Ban commented on JGRP-610:
-------------------------------
Okay, here's a better solution:
- UNICAST acquires a lock (changed from synchronized to Lock)
- UNICAST adds the (Thread,Lock) pair into the map in ProtocolStack. (All locks for
matching threads in ProtocolStack.down() will be released)
- UNICAST passes the message up. When it returns, UNICAST removes the (Thread,Lock) pair
from the map again (if existing) and releases the
lock (if still held)
- On ProtocolStack.down(): if there is a lock for the current thread in the map: remove
the (Thread,Lock) pair and release the lock
The effect of this change is that the sender-based lock held by UNICAST.up() is held only
until it is processed by the application. If the application uses the same thread to send
another message, ProtocolStack will release the lock, so that we don't have to wait
(and potentially block on FC.down()) until the message has been sent.
Problems with large return value and FC
---------------------------------------
Key: JGRP-610
URL: http://jira.jboss.com/jira/browse/JGRP-610
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.6
When sending a return value whose size is larger than FC.max_credits, the call hangs.
Unit test RpcDispatcher.testLargeReturnValueUnicastCall() reproduces the issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:11 a.m.
New subject: [JBoss JIRA] Resolved: (JGRP-610) Problems with large return value and FC
[ http://jira.jboss.com/jira/browse/JGRP-610?page=all ]
Bela Ban resolved JGRP-610.
---------------------------
Resolution: Done
Fixed according to solution. RpcDispatcherTest now passes
Problems with large return value and FC
---------------------------------------
Key: JGRP-610
URL: http://jira.jboss.com/jira/browse/JGRP-610
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.6
When sending a return value whose size is larger than FC.max_credits, the call hangs.
Unit test RpcDispatcher.testLargeReturnValueUnicastCall() reproduces the issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:26 a.m.
New subject: [JBoss JIRA] Reopened: (JGRP-610) Problems with large return value and FC
[ http://jira.jboss.com/jira/browse/JGRP-610?page=all ]
Bela Ban reopened JGRP-610:
---------------------------
Need to backport this to the 2.5 branch
Problems with large return value and FC
---------------------------------------
Key: JGRP-610
URL: http://jira.jboss.com/jira/browse/JGRP-610
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.6
When sending a return value whose size is larger than FC.max_credits, the call hangs.
Unit test RpcDispatcher.testLargeReturnValueUnicastCall() reproduces the issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:41 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-610) Problems with large return value and FC
[ http://jira.jboss.com/jira/browse/JGRP-610?page=all ]
Bela Ban updated JGRP-610:
--------------------------
Fix Version/s: 2.5.2
Problems with large return value and FC
---------------------------------------
Key: JGRP-610
URL: http://jira.jboss.com/jira/browse/JGRP-610
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.6, 2.5.2
When sending a return value whose size is larger than FC.max_credits, the call hangs.
Unit test RpcDispatcher.testLargeReturnValueUnicastCall() reproduces the issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:43 a.m.
New subject: [JBoss JIRA] Resolved: (JGRP-610) Problems with large return value and FC
[ http://jira.jboss.com/jira/browse/JGRP-610?page=all ]
Bela Ban resolved JGRP-610.
---------------------------
Resolution: Done
backported from 2.6 to 2.5
Problems with large return value and FC
---------------------------------------
Key: JGRP-610
URL: http://jira.jboss.com/jira/browse/JGRP-610
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assigned To: Bela Ban
Fix For: 2.5.2, 2.6
When sending a return value whose size is larger than FC.max_credits, the call hangs.
Unit test RpcDispatcher.testLargeReturnValueUnicastCall() reproduces the issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6266
days inactive
6273
days old
10 comments
2 participants
participants (2)
-
Bela Ban (JIRA) -
vivek v (JIRA)