6:42 a.m.
Many retransmissions with message bundling and OOB messages
-----------------------------------------------------------
Key: JGRP-1090
URL: https://jira.jboss.org/jira/browse/JGRP-1090
Project: JGroups
Issue Type: Task
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.9
When message bundling is enabled, we queue *regular* messages until max_bundle_size has
been reached or max_bundle_timeout has exceeded.
The problem is that - when sending OOB (as opposed to regular) messages - they're
*not* queued. This means the OOB message is possibly received *before* the regular
messages and thus causes retransmission !
E.g. we send regular message 1-10, then OOB message 11.
Say 1-3 are sent immediately (because the bundle timeout kicked in or the size was
exceeded). Messages 4-10 are queued and sent later. Message 11 is sent immediately.
On the receiver side, message reception could be for example: 1-3, 11, 4-10
When OOB message 11 is received, the receiver asks for retransmission of messages 4-10 !
This is not incorrect, but causes a lot of unneeded retransmissions !
public static void main(String[] args) throws Exception {
JChannel c1=new JChannel("/home/bela/fast.xml");
c1.connect("bla3");
JChannel c2=new JChannel("/home/bela/fast.xml");
c2.connect("bla3");
for(int i=1; i <= 20; i++) {
c1.send(null, null, "hello-" + i);
}
Message oob_msg=new Message(null, null, "oob msg");
oob_msg.setFlag(Message.OOB);
c1.send(oob_msg);
Util.sleep(20000);
Util.close(c2, c1);
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:44 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1090) Many retransmissions with message bundling and OOB messages
[
https://jira.jboss.org/jira/browse/JGRP-1090?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1090:
--------------------------------
A workaround is to make sure the max_bundle_timeout is smaller than the smallest
retransmission timeout in NAKACK (timeout).
Many retransmissions with message bundling and OOB messages
-----------------------------------------------------------
Key: JGRP-1090
URL: https://jira.jboss.org/jira/browse/JGRP-1090
Project: JGroups
Issue Type: Task
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.9
When message bundling is enabled, we queue *regular* messages until max_bundle_size has
been reached or max_bundle_timeout has exceeded.
The problem is that - when sending OOB (as opposed to regular) messages - they're
*not* queued. This means the OOB message is possibly received *before* the regular
messages and thus causes retransmission !
E.g. we send regular message 1-10, then OOB message 11.
Say 1-3 are sent immediately (because the bundle timeout kicked in or the size was
exceeded). Messages 4-10 are queued and sent later. Message 11 is sent immediately.
On the receiver side, message reception could be for example: 1-3, 11, 4-10
When OOB message 11 is received, the receiver asks for retransmission of messages 4-10 !
This is not incorrect, but causes a lot of unneeded retransmissions !
public static void main(String[] args) throws Exception {
JChannel c1=new JChannel("/home/bela/fast.xml");
c1.connect("bla3");
JChannel c2=new JChannel("/home/bela/fast.xml");
c2.connect("bla3");
for(int i=1; i <= 20; i++) {
c1.send(null, null, "hello-" + i);
}
Message oob_msg=new Message(null, null, "oob msg");
oob_msg.setFlag(Message.OOB);
c1.send(oob_msg);
Util.sleep(20000);
Util.close(c2, c1);
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:46 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1090) Many retransmissions with message bundling and OOB messages
[
https://jira.jboss.org/jira/browse/JGRP-1090?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1090:
--------------------------------
Analyze which protocols above NAKACK send multicast OOB messages (STABLE for sure, maybe
others ?)
Many retransmissions with message bundling and OOB messages
-----------------------------------------------------------
Key: JGRP-1090
URL: https://jira.jboss.org/jira/browse/JGRP-1090
Project: JGroups
Issue Type: Task
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.9
When message bundling is enabled, we queue *regular* messages until max_bundle_size has
been reached or max_bundle_timeout has exceeded.
The problem is that - when sending OOB (as opposed to regular) messages - they're
*not* queued. This means the OOB message is possibly received *before* the regular
messages and thus causes retransmission !
E.g. we send regular message 1-10, then OOB message 11.
Say 1-3 are sent immediately (because the bundle timeout kicked in or the size was
exceeded). Messages 4-10 are queued and sent later. Message 11 is sent immediately.
On the receiver side, message reception could be for example: 1-3, 11, 4-10
When OOB message 11 is received, the receiver asks for retransmission of messages 4-10 !
This is not incorrect, but causes a lot of unneeded retransmissions !
public static void main(String[] args) throws Exception {
JChannel c1=new JChannel("/home/bela/fast.xml");
c1.connect("bla3");
JChannel c2=new JChannel("/home/bela/fast.xml");
c2.connect("bla3");
for(int i=1; i <= 20; i++) {
c1.send(null, null, "hello-" + i);
}
Message oob_msg=new Message(null, null, "oob msg");
oob_msg.setFlag(Message.OOB);
c1.send(oob_msg);
Util.sleep(20000);
Util.close(c2, c1);
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:13 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1090) Many retransmissions with message bundling and OOB messages
[
https://jira.jboss.org/jira/browse/JGRP-1090?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1090:
--------------------------------
Maybe we should bundle OOB messages, too !
Many retransmissions with message bundling and OOB messages
-----------------------------------------------------------
Key: JGRP-1090
URL: https://jira.jboss.org/jira/browse/JGRP-1090
Project: JGroups
Issue Type: Task
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.9
When message bundling is enabled, we queue *regular* messages until max_bundle_size has
been reached or max_bundle_timeout has exceeded.
The problem is that - when sending OOB (as opposed to regular) messages - they're
*not* queued. This means the OOB message is possibly received *before* the regular
messages and thus causes retransmission !
E.g. we send regular message 1-10, then OOB message 11.
Say 1-3 are sent immediately (because the bundle timeout kicked in or the size was
exceeded). Messages 4-10 are queued and sent later. Message 11 is sent immediately.
On the receiver side, message reception could be for example: 1-3, 11, 4-10
When OOB message 11 is received, the receiver asks for retransmission of messages 4-10 !
This is not incorrect, but causes a lot of unneeded retransmissions !
public static void main(String[] args) throws Exception {
JChannel c1=new JChannel("/home/bela/fast.xml");
c1.connect("bla3");
JChannel c2=new JChannel("/home/bela/fast.xml");
c2.connect("bla3");
for(int i=1; i <= 20; i++) {
c1.send(null, null, "hello-" + i);
}
Message oob_msg=new Message(null, null, "oob msg");
oob_msg.setFlag(Message.OOB);
c1.send(oob_msg);
Util.sleep(20000);
Util.close(c2, c1);
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:33 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1090) Many retransmissions with message bundling and OOB messages
[
https://jira.jboss.org/jira/browse/JGRP-1090?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1090:
--------------------------------
same with messages with the DONT_BUNDLE flag set
Many retransmissions with message bundling and OOB messages
-----------------------------------------------------------
Key: JGRP-1090
URL: https://jira.jboss.org/jira/browse/JGRP-1090
Project: JGroups
Issue Type: Task
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.9
When message bundling is enabled, we queue *regular* messages until max_bundle_size has
been reached or max_bundle_timeout has exceeded.
The problem is that - when sending OOB (as opposed to regular) messages - they're
*not* queued. This means the OOB message is possibly received *before* the regular
messages and thus causes retransmission !
E.g. we send regular message 1-10, then OOB message 11.
Say 1-3 are sent immediately (because the bundle timeout kicked in or the size was
exceeded). Messages 4-10 are queued and sent later. Message 11 is sent immediately.
On the receiver side, message reception could be for example: 1-3, 11, 4-10
When OOB message 11 is received, the receiver asks for retransmission of messages 4-10 !
This is not incorrect, but causes a lot of unneeded retransmissions !
public static void main(String[] args) throws Exception {
JChannel c1=new JChannel("/home/bela/fast.xml");
c1.connect("bla3");
JChannel c2=new JChannel("/home/bela/fast.xml");
c2.connect("bla3");
for(int i=1; i <= 20; i++) {
c1.send(null, null, "hello-" + i);
}
Message oob_msg=new Message(null, null, "oob msg");
oob_msg.setFlag(Message.OOB);
c1.send(oob_msg);
Util.sleep(20000);
Util.close(c2, c1);
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:31 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1090) Many retransmissions with message bundling and OOB messages
[
https://jira.jboss.org/jira/browse/JGRP-1090?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1090:
--------------------------------
[Email from Brian]
I saw JGRP-1090 today an it reminded me of something I was thinking
about back in Sept.
On the receiver side, when a bundled message comes in, a single pool
thread unpacks it and then loops carrying each individual message up
the stack.
I wonder if you can get some strange effects from that under load.
E.g. sender A sends bundle 1 and then bundle 2. On receiver B, thread
1 is processing bundle 1 when bundle 2 comes in and gets handled by
thread 2. As a result, bundle 2 messages start arriving in NAKACK
before a bunch of bundle 1 messages, leading to extra retransmission
tasks.
Problem is worse if thread 1 happens to block a bit above NAKACK.
A possibility is to unpack the bundled message and then resubmit each
individual message to the pool. But I suspect that would cost more
than it gains.
TBH, I doubt this is a big issue, but wanted to throw it out there in
case it sparks any thoughts on your end.
Many retransmissions with message bundling and OOB messages
-----------------------------------------------------------
Key: JGRP-1090
URL: https://jira.jboss.org/jira/browse/JGRP-1090
Project: JGroups
Issue Type: Task
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.9
When message bundling is enabled, we queue *regular* messages until max_bundle_size has
been reached or max_bundle_timeout has exceeded.
The problem is that - when sending OOB (as opposed to regular) messages - they're
*not* queued. This means the OOB message is possibly received *before* the regular
messages and thus causes retransmission !
E.g. we send regular message 1-10, then OOB message 11.
Say 1-3 are sent immediately (because the bundle timeout kicked in or the size was
exceeded). Messages 4-10 are queued and sent later. Message 11 is sent immediately.
On the receiver side, message reception could be for example: 1-3, 11, 4-10
When OOB message 11 is received, the receiver asks for retransmission of messages 4-10 !
This is not incorrect, but causes a lot of unneeded retransmissions !
public static void main(String[] args) throws Exception {
JChannel c1=new JChannel("/home/bela/fast.xml");
c1.connect("bla3");
JChannel c2=new JChannel("/home/bela/fast.xml");
c2.connect("bla3");
for(int i=1; i <= 20; i++) {
c1.send(null, null, "hello-" + i);
}
Message oob_msg=new Message(null, null, "oob msg");
oob_msg.setFlag(Message.OOB);
c1.send(oob_msg);
Util.sleep(20000);
Util.close(c2, c1);
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:45 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1090) Many retransmissions with message bundling and OOB messages
[
https://jira.jboss.org/jira/browse/JGRP-1090?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1090:
--------------------------------
Idea to think about: submit each message in a bundle individually to the thread pool:
#1 A thread sequentially passes all messages of a bundle up the stack
- If a bundle 1 [100-120] is processed, thread A passes 100 up first, then 101 and so on.
Thread A currently removes and processes 100
- Meanwhile thread B processes bundle 2 [121-130], adds all messages into the NRW. This
causes retransmission requests for [101-120]
- When thread A returns from processing 100, it adds 101 to the NRW, cancelling the
retransmission of 101, then 102 etc
- The issue is that processing 101-120 takes time, during which retransmission for 101-120
will be going on
A better solution would be if thread A added *all* messages [100-120] into the NRW
*before* it started removing them. This would work with submitting each message of a
bundle to the thread pool, e.g.
if(is_message_list) { // used if message bundling is enabled
List<Message> msgs=readMessageList(dis);
for(Message msg: msgs) {
if(msg.isFlagSet(Message.OOB)) {
log.warn("bundled message should not be marked as OOB");
}
pool.execute(new Runnable() {public void run() {handleMyMessage(msg, multicast);}}
}
However, this taxes the thread pool, creates more threads and thus more context switching,
which is also overhead. As a matter of fact, I tried this out and got a 5-10% performance
drop.
Many retransmissions with message bundling and OOB messages
-----------------------------------------------------------
Key: JGRP-1090
URL: https://jira.jboss.org/jira/browse/JGRP-1090
Project: JGroups
Issue Type: Task
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.9
When message bundling is enabled, we queue *regular* messages until max_bundle_size has
been reached or max_bundle_timeout has exceeded.
The problem is that - when sending OOB (as opposed to regular) messages - they're
*not* queued. This means the OOB message is possibly received *before* the regular
messages and thus causes retransmission !
E.g. we send regular message 1-10, then OOB message 11.
Say 1-3 are sent immediately (because the bundle timeout kicked in or the size was
exceeded). Messages 4-10 are queued and sent later. Message 11 is sent immediately.
On the receiver side, message reception could be for example: 1-3, 11, 4-10
When OOB message 11 is received, the receiver asks for retransmission of messages 4-10 !
This is not incorrect, but causes a lot of unneeded retransmissions !
public static void main(String[] args) throws Exception {
JChannel c1=new JChannel("/home/bela/fast.xml");
c1.connect("bla3");
JChannel c2=new JChannel("/home/bela/fast.xml");
c2.connect("bla3");
for(int i=1; i <= 20; i++) {
c1.send(null, null, "hello-" + i);
}
Message oob_msg=new Message(null, null, "oob msg");
oob_msg.setFlag(Message.OOB);
c1.send(oob_msg);
Util.sleep(20000);
Util.close(c2, c1);
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1090) Many retransmissions with message bundling and OOB messages
[
https://jira.jboss.org/jira/browse/JGRP-1090?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1090:
--------------------------------
Correction: *add* DONT_BUNDLE to OOB, if OOB is used for non-bundling purposes. If not,
replace OOB with DONT_BUNDLE
Many retransmissions with message bundling and OOB messages
-----------------------------------------------------------
Key: JGRP-1090
URL: https://jira.jboss.org/jira/browse/JGRP-1090
Project: JGroups
Issue Type: Task
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.9
When message bundling is enabled, we queue *regular* messages until max_bundle_size has
been reached or max_bundle_timeout has exceeded.
The problem is that - when sending OOB (as opposed to regular) messages - they're
*not* queued. This means the OOB message is possibly received *before* the regular
messages and thus causes retransmission !
E.g. we send regular message 1-10, then OOB message 11.
Say 1-3 are sent immediately (because the bundle timeout kicked in or the size was
exceeded). Messages 4-10 are queued and sent later. Message 11 is sent immediately.
On the receiver side, message reception could be for example: 1-3, 11, 4-10
When OOB message 11 is received, the receiver asks for retransmission of messages 4-10 !
This is not incorrect, but causes a lot of unneeded retransmissions !
public static void main(String[] args) throws Exception {
JChannel c1=new JChannel("/home/bela/fast.xml");
c1.connect("bla3");
JChannel c2=new JChannel("/home/bela/fast.xml");
c2.connect("bla3");
for(int i=1; i <= 20; i++) {
c1.send(null, null, "hello-" + i);
}
Message oob_msg=new Message(null, null, "oob msg");
oob_msg.setFlag(Message.OOB);
c1.send(oob_msg);
Util.sleep(20000);
Util.close(c2, c1);
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1090) Many retransmissions with message bundling and OOB messages
[
https://jira.jboss.org/jira/browse/JGRP-1090?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1090:
--------------------------------
Replace OOB with DONT_BUNDLE in JGroups internal code
Many retransmissions with message bundling and OOB messages
-----------------------------------------------------------
Key: JGRP-1090
URL: https://jira.jboss.org/jira/browse/JGRP-1090
Project: JGroups
Issue Type: Task
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.9
When message bundling is enabled, we queue *regular* messages until max_bundle_size has
been reached or max_bundle_timeout has exceeded.
The problem is that - when sending OOB (as opposed to regular) messages - they're
*not* queued. This means the OOB message is possibly received *before* the regular
messages and thus causes retransmission !
E.g. we send regular message 1-10, then OOB message 11.
Say 1-3 are sent immediately (because the bundle timeout kicked in or the size was
exceeded). Messages 4-10 are queued and sent later. Message 11 is sent immediately.
On the receiver side, message reception could be for example: 1-3, 11, 4-10
When OOB message 11 is received, the receiver asks for retransmission of messages 4-10 !
This is not incorrect, but causes a lot of unneeded retransmissions !
public static void main(String[] args) throws Exception {
JChannel c1=new JChannel("/home/bela/fast.xml");
c1.connect("bla3");
JChannel c2=new JChannel("/home/bela/fast.xml");
c2.connect("bla3");
for(int i=1; i <= 20; i++) {
c1.send(null, null, "hello-" + i);
}
Message oob_msg=new Message(null, null, "oob msg");
oob_msg.setFlag(Message.OOB);
c1.send(oob_msg);
Util.sleep(20000);
Util.close(c2, c1);
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:31 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-1090) Bundle OOB messages
[
https://jira.jboss.org/jira/browse/JGRP-1090?page=com.atlassian.jira.plug...
]
Bela Ban updated JGRP-1090:
---------------------------
Summary: Bundle OOB messages (was: Many retransmissions with message bundling and OOB
messages)
Bundle OOB messages
-------------------
Key: JGRP-1090
URL: https://jira.jboss.org/jira/browse/JGRP-1090
Project: JGroups
Issue Type: Task
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.9
When message bundling is enabled, we queue *regular* messages until max_bundle_size has
been reached or max_bundle_timeout has exceeded.
The problem is that - when sending OOB (as opposed to regular) messages - they're
*not* queued. This means the OOB message is possibly received *before* the regular
messages and thus causes retransmission !
E.g. we send regular message 1-10, then OOB message 11.
Say 1-3 are sent immediately (because the bundle timeout kicked in or the size was
exceeded). Messages 4-10 are queued and sent later. Message 11 is sent immediately.
On the receiver side, message reception could be for example: 1-3, 11, 4-10
When OOB message 11 is received, the receiver asks for retransmission of messages 4-10 !
This is not incorrect, but causes a lot of unneeded retransmissions !
public static void main(String[] args) throws Exception {
JChannel c1=new JChannel("/home/bela/fast.xml");
c1.connect("bla3");
JChannel c2=new JChannel("/home/bela/fast.xml");
c2.connect("bla3");
for(int i=1; i <= 20; i++) {
c1.send(null, null, "hello-" + i);
}
Message oob_msg=new Message(null, null, "oob msg");
oob_msg.setFlag(Message.OOB);
c1.send(oob_msg);
Util.sleep(20000);
Util.close(c2, c1);
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:44 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-1090) Bundle OOB messages
[
https://jira.jboss.org/jira/browse/JGRP-1090?page=com.atlassian.jira.plug...
]
Bela Ban updated JGRP-1090:
---------------------------
Fix Version/s: 2.10
(was: 2.9)
Bundle OOB messages
-------------------
Key: JGRP-1090
URL: https://jira.jboss.org/jira/browse/JGRP-1090
Project: JGroups
Issue Type: Task
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.10
When message bundling is enabled, we queue *regular* messages until max_bundle_size has
been reached or max_bundle_timeout has exceeded.
The problem is that - when sending OOB (as opposed to regular) messages - they're
*not* queued. This means the OOB message is possibly received *before* the regular
messages and thus causes retransmission !
E.g. we send regular message 1-10, then OOB message 11.
Say 1-3 are sent immediately (because the bundle timeout kicked in or the size was
exceeded). Messages 4-10 are queued and sent later. Message 11 is sent immediately.
On the receiver side, message reception could be for example: 1-3, 11, 4-10
When OOB message 11 is received, the receiver asks for retransmission of messages 4-10 !
This is not incorrect, but causes a lot of unneeded retransmissions !
public static void main(String[] args) throws Exception {
JChannel c1=new JChannel("/home/bela/fast.xml");
c1.connect("bla3");
JChannel c2=new JChannel("/home/bela/fast.xml");
c2.connect("bla3");
for(int i=1; i <= 20; i++) {
c1.send(null, null, "hello-" + i);
}
Message oob_msg=new Message(null, null, "oob msg");
oob_msg.setFlag(Message.OOB);
c1.send(oob_msg);
Util.sleep(20000);
Util.close(c2, c1);
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:21 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1090) Bundle OOB messages
[
https://jira.jboss.org/jira/browse/JGRP-1090?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1090:
--------------------------------
The current implementation is not very efficient: removeMany() in NAKACK and UNICAST will
usually only remove 1 message for a given bundle:
The processing in UNICAST and NAKACK is as follows:
- Receive a bundle of 10 messages (in TP)
- For each message M:
- Add the message to the window
- Remove as many messages as possible and pass them up to the application
(Usually this will be just 1 message !)
- Continue with M+1 etc
==> Every message is added to and removed from the window *separately*; this defies the
purpose of removeMany() !
Bundle OOB messages
-------------------
Key: JGRP-1090
URL: https://jira.jboss.org/jira/browse/JGRP-1090
Project: JGroups
Issue Type: Task
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.10
When message bundling is enabled, we queue *regular* messages until max_bundle_size has
been reached or max_bundle_timeout has exceeded.
The problem is that - when sending OOB (as opposed to regular) messages - they're
*not* queued. This means the OOB message is possibly received *before* the regular
messages and thus causes retransmission !
E.g. we send regular message 1-10, then OOB message 11.
Say 1-3 are sent immediately (because the bundle timeout kicked in or the size was
exceeded). Messages 4-10 are queued and sent later. Message 11 is sent immediately.
On the receiver side, message reception could be for example: 1-3, 11, 4-10
When OOB message 11 is received, the receiver asks for retransmission of messages 4-10 !
This is not incorrect, but causes a lot of unneeded retransmissions !
public static void main(String[] args) throws Exception {
JChannel c1=new JChannel("/home/bela/fast.xml");
c1.connect("bla3");
JChannel c2=new JChannel("/home/bela/fast.xml");
c2.connect("bla3");
for(int i=1; i <= 20; i++) {
c1.send(null, null, "hello-" + i);
}
Message oob_msg=new Message(null, null, "oob msg");
oob_msg.setFlag(Message.OOB);
c1.send(oob_msg);
Util.sleep(20000);
Util.close(c2, c1);
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:12 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1090) Bundle OOB messages
[
https://jira.jboss.org/jira/browse/JGRP-1090?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1090:
--------------------------------
Maybe an Event.MSGLIST type is needed, which can pass multiple incoming messages up the
stack. This requires changes to several protocols though, especially if we want to extend
this to all protocols, and to sending of messages as well.
Bundle OOB messages
-------------------
Key: JGRP-1090
URL: https://jira.jboss.org/jira/browse/JGRP-1090
Project: JGroups
Issue Type: Task
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.10
When message bundling is enabled, we queue *regular* messages until max_bundle_size has
been reached or max_bundle_timeout has exceeded.
The problem is that - when sending OOB (as opposed to regular) messages - they're
*not* queued. This means the OOB message is possibly received *before* the regular
messages and thus causes retransmission !
E.g. we send regular message 1-10, then OOB message 11.
Say 1-3 are sent immediately (because the bundle timeout kicked in or the size was
exceeded). Messages 4-10 are queued and sent later. Message 11 is sent immediately.
On the receiver side, message reception could be for example: 1-3, 11, 4-10
When OOB message 11 is received, the receiver asks for retransmission of messages 4-10 !
This is not incorrect, but causes a lot of unneeded retransmissions !
public static void main(String[] args) throws Exception {
JChannel c1=new JChannel("/home/bela/fast.xml");
c1.connect("bla3");
JChannel c2=new JChannel("/home/bela/fast.xml");
c2.connect("bla3");
for(int i=1; i <= 20; i++) {
c1.send(null, null, "hello-" + i);
}
Message oob_msg=new Message(null, null, "oob msg");
oob_msg.setFlag(Message.OOB);
c1.send(oob_msg);
Util.sleep(20000);
Util.close(c2, c1);
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:55 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1090) Bundle OOB messages
[
https://jira.jboss.org/jira/browse/JGRP-1090?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1090:
--------------------------------
I tried assigning execution of individual messages of a bundle to a thread (OOB or
regular) - performance dropped ca 20-30% !
I have to investigate further, but I believe this is due to increased context switching
(vmstat should show that).
I also tried MSG_LIST, and here I got a new (albeit small) perf improvement ! Small
probably because the receiver isn't doing anything with the messages, but just
increments a counter. So the above scenario of messages ahead of the queue being processed
by the same thread blocking doesn't happen.
If we added a sleep in the receiver, we should see bigger differences...
Bundle OOB messages
-------------------
Key: JGRP-1090
URL: https://jira.jboss.org/jira/browse/JGRP-1090
Project: JGroups
Issue Type: Task
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.10
When message bundling is enabled, we queue *regular* messages until max_bundle_size has
been reached or max_bundle_timeout has exceeded.
The problem is that - when sending OOB (as opposed to regular) messages - they're
*not* queued. This means the OOB message is possibly received *before* the regular
messages and thus causes retransmission !
E.g. we send regular message 1-10, then OOB message 11.
Say 1-3 are sent immediately (because the bundle timeout kicked in or the size was
exceeded). Messages 4-10 are queued and sent later. Message 11 is sent immediately.
On the receiver side, message reception could be for example: 1-3, 11, 4-10
When OOB message 11 is received, the receiver asks for retransmission of messages 4-10 !
This is not incorrect, but causes a lot of unneeded retransmissions !
public static void main(String[] args) throws Exception {
JChannel c1=new JChannel("/home/bela/fast.xml");
c1.connect("bla3");
JChannel c2=new JChannel("/home/bela/fast.xml");
c2.connect("bla3");
for(int i=1; i <= 20; i++) {
c1.send(null, null, "hello-" + i);
}
Message oob_msg=new Message(null, null, "oob msg");
oob_msg.setFlag(Message.OOB);
c1.send(oob_msg);
Util.sleep(20000);
Util.close(c2, c1);
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:50 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1090) Bundle OOB messages
[
https://jira.jboss.org/jira/browse/JGRP-1090?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1090:
--------------------------------
A new event type MSG_LIST is not good, either: this requires too many changes in protocol
code, ie. all protocols which implement up(MSG), will now also have to handle up(MSG_LIST)
!
Bundle OOB messages
-------------------
Key: JGRP-1090
URL: https://jira.jboss.org/jira/browse/JGRP-1090
Project: JGroups
Issue Type: Task
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.10
When message bundling is enabled, we queue *regular* messages until max_bundle_size has
been reached or max_bundle_timeout has exceeded.
The problem is that - when sending OOB (as opposed to regular) messages - they're
*not* queued. This means the OOB message is possibly received *before* the regular
messages and thus causes retransmission !
E.g. we send regular message 1-10, then OOB message 11.
Say 1-3 are sent immediately (because the bundle timeout kicked in or the size was
exceeded). Messages 4-10 are queued and sent later. Message 11 is sent immediately.
On the receiver side, message reception could be for example: 1-3, 11, 4-10
When OOB message 11 is received, the receiver asks for retransmission of messages 4-10 !
This is not incorrect, but causes a lot of unneeded retransmissions !
public static void main(String[] args) throws Exception {
JChannel c1=new JChannel("/home/bela/fast.xml");
c1.connect("bla3");
JChannel c2=new JChannel("/home/bela/fast.xml");
c2.connect("bla3");
for(int i=1; i <= 20; i++) {
c1.send(null, null, "hello-" + i);
}
Message oob_msg=new Message(null, null, "oob msg");
oob_msg.setFlag(Message.OOB);
c1.send(oob_msg);
Util.sleep(20000);
Util.close(c2, c1);
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:55 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1090) Bundle OOB messages
[
https://jira.jboss.org/jira/browse/JGRP-1090?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1090:
--------------------------------
For now, the bundlers were changed to store messages per *cluster and destination
address*, e.g. messages with dest=null for "cluster1" are stored separately from
messages to "cluster2".
This way, messages sent on a shared transport to different clusters won't get handled
by the same thread at the receiver(s). Also, multicast messages won't interfere with
unicast messages.
The old solution stored messages under only the destination as key, and didn't take
the cluster name into account.
So the only messages that will be delivered by the same thread are messages from the same
sender to the same destination and cluster.
To be investigated: should we neverthless bundle OOB messages as well ? This would entail
having a separate hashmap in the bundlers, and handle every OOB message of a message
bundle with a separate thread at the receiver(s).
Bundle OOB messages
-------------------
Key: JGRP-1090
URL: https://jira.jboss.org/jira/browse/JGRP-1090
Project: JGroups
Issue Type: Task
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.10
When message bundling is enabled, we queue *regular* messages until max_bundle_size has
been reached or max_bundle_timeout has exceeded.
The problem is that - when sending OOB (as opposed to regular) messages - they're
*not* queued. This means the OOB message is possibly received *before* the regular
messages and thus causes retransmission !
E.g. we send regular message 1-10, then OOB message 11.
Say 1-3 are sent immediately (because the bundle timeout kicked in or the size was
exceeded). Messages 4-10 are queued and sent later. Message 11 is sent immediately.
On the receiver side, message reception could be for example: 1-3, 11, 4-10
When OOB message 11 is received, the receiver asks for retransmission of messages 4-10 !
This is not incorrect, but causes a lot of unneeded retransmissions !
public static void main(String[] args) throws Exception {
JChannel c1=new JChannel("/home/bela/fast.xml");
c1.connect("bla3");
JChannel c2=new JChannel("/home/bela/fast.xml");
c2.connect("bla3");
for(int i=1; i <= 20; i++) {
c1.send(null, null, "hello-" + i);
}
Message oob_msg=new Message(null, null, "oob msg");
oob_msg.setFlag(Message.OOB);
c1.send(oob_msg);
Util.sleep(20000);
Util.close(c2, c1);
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:57 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1090) Bundle OOB messages
[
https://jira.jboss.org/jira/browse/JGRP-1090?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1090:
--------------------------------
Creating a separate hashmap for OOB messages, and bundling them, yielded very bad
performance !
I guess the solution as it stands now is tolerable, and we won't bundle OOB message,
at least for the time being.
Bundle OOB messages
-------------------
Key: JGRP-1090
URL: https://jira.jboss.org/jira/browse/JGRP-1090
Project: JGroups
Issue Type: Task
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.10
When message bundling is enabled, we queue *regular* messages until max_bundle_size has
been reached or max_bundle_timeout has exceeded.
The problem is that - when sending OOB (as opposed to regular) messages - they're
*not* queued. This means the OOB message is possibly received *before* the regular
messages and thus causes retransmission !
E.g. we send regular message 1-10, then OOB message 11.
Say 1-3 are sent immediately (because the bundle timeout kicked in or the size was
exceeded). Messages 4-10 are queued and sent later. Message 11 is sent immediately.
On the receiver side, message reception could be for example: 1-3, 11, 4-10
When OOB message 11 is received, the receiver asks for retransmission of messages 4-10 !
This is not incorrect, but causes a lot of unneeded retransmissions !
public static void main(String[] args) throws Exception {
JChannel c1=new JChannel("/home/bela/fast.xml");
c1.connect("bla3");
JChannel c2=new JChannel("/home/bela/fast.xml");
c2.connect("bla3");
for(int i=1; i <= 20; i++) {
c1.send(null, null, "hello-" + i);
}
Message oob_msg=new Message(null, null, "oob msg");
oob_msg.setFlag(Message.OOB);
c1.send(oob_msg);
Util.sleep(20000);
Util.close(c2, c1);
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:59 a.m.
New subject: [JBoss JIRA] Resolved: (JGRP-1090) Bundle OOB messages
[
https://jira.jboss.org/jira/browse/JGRP-1090?page=com.atlassian.jira.plug...
]
Bela Ban resolved JGRP-1090.
----------------------------
Resolution: Done
Resolved for now, might revisit in a later release
Bundle OOB messages
-------------------
Key: JGRP-1090
URL: https://jira.jboss.org/jira/browse/JGRP-1090
Project: JGroups
Issue Type: Task
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.10
When message bundling is enabled, we queue *regular* messages until max_bundle_size has
been reached or max_bundle_timeout has exceeded.
The problem is that - when sending OOB (as opposed to regular) messages - they're
*not* queued. This means the OOB message is possibly received *before* the regular
messages and thus causes retransmission !
E.g. we send regular message 1-10, then OOB message 11.
Say 1-3 are sent immediately (because the bundle timeout kicked in or the size was
exceeded). Messages 4-10 are queued and sent later. Message 11 is sent immediately.
On the receiver side, message reception could be for example: 1-3, 11, 4-10
When OOB message 11 is received, the receiver asks for retransmission of messages 4-10 !
This is not incorrect, but causes a lot of unneeded retransmissions !
public static void main(String[] args) throws Exception {
JChannel c1=new JChannel("/home/bela/fast.xml");
c1.connect("bla3");
JChannel c2=new JChannel("/home/bela/fast.xml");
c2.connect("bla3");
for(int i=1; i <= 20; i++) {
c1.send(null, null, "hello-" + i);
}
Message oob_msg=new Message(null, null, "oob msg");
oob_msg.setFlag(Message.OOB);
c1.send(oob_msg);
Util.sleep(20000);
Util.close(c2, c1);
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5946
days inactive
6101
days old
17 comments
1 participants
participants (1)
-
Bela Ban (JIRA)