9:53 a.m.
STREAMING_STATE_TRANSFER abysmally slow
---------------------------------------
Key: JGRP-1223
URL: https://jira.jboss.org/browse/JGRP-1223
Project: JGroups
Issue Type: Bug
Affects Versions: 2.8
Environment: Windows XP
Reporter: Kornelius Elstner
Assignee: Bela Ban
With a relatively large shared state (anything in excess of a few MBytes)
STREAMING_STATE_TRANSFER is very slow, it can take minutes for the state transfer to
complete. The underlying issue is that
STREAMING_STATE_TRANSFER.StreamingOutputStreamWrapper subclasses
java.io.FilterOutputStream which always calls write(int b), even if buffers are to be
written to the underlying socket.
As a consequence each byte to be transferred results in a call to write() to the
underlying socket.
The fix is simple, the byte array variants of StreamingOutputStreamWrapper.write() should
delegate to the wrapped socket directly.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:57 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-1223) STREAMING_STATE_TRANSFER abysmally slow
[
https://jira.jboss.org/browse/JGRP-1223?page=com.atlassian.jira.plugin.sy...
]
Kornelius Elstner updated JGRP-1223:
------------------------------------
Attachment: JGRP-1223.zip
STREAMING_STATE_TRANSFER abysmally slow
---------------------------------------
Key: JGRP-1223
URL: https://jira.jboss.org/browse/JGRP-1223
Project: JGroups
Issue Type: Bug
Affects Versions: 2.8
Environment: Windows XP
Reporter: Kornelius Elstner
Assignee: Bela Ban
Attachments: JGRP-1223.zip
With a relatively large shared state (anything in excess of a few MBytes)
STREAMING_STATE_TRANSFER is very slow, it can take minutes for the state transfer to
complete. The underlying issue is that
STREAMING_STATE_TRANSFER.StreamingOutputStreamWrapper subclasses
java.io.FilterOutputStream which always calls write(int b), even if buffers are to be
written to the underlying socket.
As a consequence each byte to be transferred results in a call to write() to the
underlying socket.
The fix is simple, the byte array variants of StreamingOutputStreamWrapper.write() should
delegate to the wrapped socket directly.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:59 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-1223) STREAMING_STATE_TRANSFER abysmally slow
[
https://jira.jboss.org/browse/JGRP-1223?page=com.atlassian.jira.plugin.sy...
]
Kornelius Elstner updated JGRP-1223:
------------------------------------
Steps to Reproduce:
Run two instances of StateTransferTest.java (see attached file) with the included config.
After one minute the second instance will (typically) end up with an empty map:
INFO JChannel - JGroups version: 2.8.0.GA
-------------------------------------------------------------------
GMS: address=WZUR4269297-5387, cluster=state-transfer-test, physical
address=165.222.17.200:2599
-------------------------------------------------------------------
INFO ReplicatedHashMap - state could not be retrieved (first member)
Loading state took: 60135ms
Shared map has 0 entries
With the included patch the state is loaded successfully:
INFO JChannel - JGroups version: 2.8.0.GA
-------------------------------------------------------------------
GMS: address=WZUR4269297-19455, cluster=state-transfer-test, physical
address=165.222.17.200:2654
-------------------------------------------------------------------
INFO ReplicatedHashMap - state was retrieved successfully, waiting for setState()
INFO ReplicatedHashMap - setState() was called
Loading state took: 1484ms
Shared map has 10000 entries
was:
run two instances of:
======================= StateTransferTest.java: ===============================
import java.io.File;
import java.io.IOException;
import java.util.Random;
import org.jgroups.Channel;
import org.jgroups.ChannelException;
import org.jgroups.JChannel;
import org.jgroups.blocks.ReplicatedHashMap;
import org.jgroups.util.UUID;
public class StateTransferTest {
private final ReplicatedHashMap<String, byte[]> sharedCache;
private final Channel channel;
private final Random random = new Random();
private final boolean isCoordinator;
private byte[] makeRandom(final int size) {
final byte[] ret = new byte[size];
random.nextBytes(ret);
return ret;
}
public StateTransferTest() throws ChannelException {
channel = new JChannel(new File("jgroups-config.xml"));
channel.connect("state-transfer-test");
isCoordinator = channel.getView().getMembers().get(0).equals(channel.getAddress());
sharedCache = new ReplicatedHashMap<String, byte[]>(channel);
sharedCache.setBlockingUpdates(true);
final long startTime = System.currentTimeMillis();
sharedCache.start(60000);
final long endTime = System.currentTimeMillis();
System.out.println("Loading state took: " + (endTime - startTime) +
"ms");
if (isCoordinator) {
for (int i = 0; i < 10000; i++) {
final byte[] sub = makeRandom(5000);
sharedCache.put(UUID.randomUUID().toString(), sub);
}
System.out.println("Created shared data");
}
System.out.println("Shared map has " + sharedCache.size() + "
entries");
}
private void stop() {
channel.disconnect();
}
public static void main(final String[] args) throws IOException, ChannelException {
final StateTransferTest stateTransferTest = new StateTransferTest();
if (!stateTransferTest.isCoordinator) {
stateTransferTest.stop();
}
}
}
===============================================================================
with the following config:
============================= jgroups-config.xml ==============================
<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups
http://www.jgroups.org/schema/JGroups-2.8.xsd">
<UDP
mcast_port="45000"
tos="8"
ucast_recv_buf_size="20M"
ucast_send_buf_size="640K"
mcast_recv_buf_size="25M"
mcast_send_buf_size="640K"
loopback="false"
discard_incompatible_packets="true"
max_bundle_size="64K"
max_bundle_timeout="30"
ip_ttl="2"
enable_bundling="true"
enable_diagnostics="true"
thread_naming_pattern="cl"
timer.num_threads="4"
thread_pool.enabled="true"
thread_pool.min_threads="2"
thread_pool.max_threads="8"
thread_pool.keep_alive_time="5000"
thread_pool.queue_enabled="true"
thread_pool.queue_max_size="10000"
thread_pool.rejection_policy="discard"
oob_thread_pool.enabled="true"
oob_thread_pool.min_threads="1"
oob_thread_pool.max_threads="8"
oob_thread_pool.keep_alive_time="5000"
oob_thread_pool.queue_enabled="false"
oob_thread_pool.queue_max_size="100"
oob_thread_pool.rejection_policy="Run"/>
<PING timeout="2000"
num_initial_members="3"/>
<MERGE2 max_interval="30000" min_interval="10000"/>
<FD_SOCK/>
<FD timeout="10000" max_tries="5"/>
<VERIFY_SUSPECT timeout="1500"/>
<BARRIER/>
<pbcast.NAKACK use_mcast_xmit="false" gc_lag="0"
retransmit_timeout="300,600,1200,2400,4800"
discard_delivered_msgs="true" />
<UNICAST timeout="300,600,1200" />
<pbcast.STABLE stability_delay="1000"
desired_avg_gossip="50000" max_bytes="400000" />
<VIEW_SYNC avg_send_interval="60000"/>
<pbcast.GMS print_local_addr="true" join_timeout="3000"
view_bundling="true" />
<FC max_credits="20000000" min_threshold="0.10" />
<FRAG2 frag_size="60000" />
<pbcast.STREAMING_STATE_TRANSFER />
</config>
===============================================================================
After one minute the second instance will (typically) end up with an empty map:
INFO JChannel - JGroups version: 2.8.0.GA
-------------------------------------------------------------------
GMS: address=WZUR4269297-5387, cluster=state-transfer-test, physical
address=165.222.17.200:2599
-------------------------------------------------------------------
INFO ReplicatedHashMap - state could not be retrieved (first member)
Loading state took: 60135ms
Shared map has 0 entries
After applying the patch:
===============================================================================
---
JGroups-2.8.1.GA.src\src\org\jgroups\protocols\pbcast\STREAMING_STATE_TRANSFER.java 2010-04-30
13:52:36.000000000 +0200
+++ patch\src\org\jgroups\protocols\pbcast\STREAMING_STATE_TRANSFER.java 2010-06-29
16:43:45.753146800 +0200
@@ -724,17 +724,17 @@ public class STREAMING_STATE_TRANSFER ex
}
public void write(byte[] b, int off, int len) throws IOException {
- super.write(b, off, len);
+ out.write(b, off, len);
bytesWrittenCounter += len;
}
public void write(byte[] b) throws IOException {
- super.write(b);
+ out.write(b);
bytesWrittenCounter += b.length;
}
public void write(int b) throws IOException {
- super.write(b);
+ out.write(b);
bytesWrittenCounter += 1;
}
}
===============================================================================
The state is loaded successfully:
INFO JChannel - JGroups version: 2.8.0.GA
-------------------------------------------------------------------
GMS: address=WZUR4269297-19455, cluster=state-transfer-test, physical
address=165.222.17.200:2654
-------------------------------------------------------------------
INFO ReplicatedHashMap - state was retrieved successfully, waiting for setState()
INFO ReplicatedHashMap - setState() was called
Loading state took: 1484ms
Shared map has 10000 entries
STREAMING_STATE_TRANSFER abysmally slow
---------------------------------------
Key: JGRP-1223
URL: https://jira.jboss.org/browse/JGRP-1223
Project: JGroups
Issue Type: Bug
Affects Versions: 2.8
Environment: Windows XP
Reporter: Kornelius Elstner
Assignee: Bela Ban
Attachments: JGRP-1223.zip
With a relatively large shared state (anything in excess of a few MBytes)
STREAMING_STATE_TRANSFER is very slow, it can take minutes for the state transfer to
complete. The underlying issue is that
STREAMING_STATE_TRANSFER.StreamingOutputStreamWrapper subclasses
java.io.FilterOutputStream which always calls write(int b), even if buffers are to be
written to the underlying socket.
As a consequence each byte to be transferred results in a call to write() to the
underlying socket.
The fix is simple, the byte array variants of StreamingOutputStreamWrapper.write() should
delegate to the wrapped socket directly.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:23 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-1223) STREAMING_STATE_TRANSFER abysmally slow
[
https://jira.jboss.org/browse/JGRP-1223?page=com.atlassian.jira.plugin.sy...
]
Bela Ban updated JGRP-1223:
---------------------------
Assignee: Vladimir Blagojevic (was: Bela Ban)
Fix Version/s: 2.6.16
2.10
STREAMING_STATE_TRANSFER abysmally slow
---------------------------------------
Key: JGRP-1223
URL: https://jira.jboss.org/browse/JGRP-1223
Project: JGroups
Issue Type: Bug
Affects Versions: 2.8
Environment: Windows XP
Reporter: Kornelius Elstner
Assignee: Vladimir Blagojevic
Fix For: 2.6.16, 2.10
Attachments: JGRP-1223.zip
With a relatively large shared state (anything in excess of a few MBytes)
STREAMING_STATE_TRANSFER is very slow, it can take minutes for the state transfer to
complete. The underlying issue is that
STREAMING_STATE_TRANSFER.StreamingOutputStreamWrapper subclasses
java.io.FilterOutputStream which always calls write(int b), even if buffers are to be
written to the underlying socket.
As a consequence each byte to be transferred results in a call to write() to the
underlying socket.
The fix is simple, the byte array variants of StreamingOutputStreamWrapper.write() should
delegate to the wrapped socket directly.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:03 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1223) STREAMING_STATE_TRANSFER abysmally slow
[
https://jira.jboss.org/browse/JGRP-1223?page=com.atlassian.jira.plugin.sy...
]
Vladimir Blagojevic commented on JGRP-1223:
-------------------------------------------
Wow, interesting catch!!! I would have never guessed that FilterOutputStream does that.
Did you notice a significant change in performance with your patch?
STREAMING_STATE_TRANSFER abysmally slow
---------------------------------------
Key: JGRP-1223
URL: https://jira.jboss.org/browse/JGRP-1223
Project: JGroups
Issue Type: Bug
Affects Versions: 2.8
Environment: Windows XP
Reporter: Kornelius Elstner
Assignee: Vladimir Blagojevic
Fix For: 2.6.16, 2.10
Attachments: JGRP-1223.zip
With a relatively large shared state (anything in excess of a few MBytes)
STREAMING_STATE_TRANSFER is very slow, it can take minutes for the state transfer to
complete. The underlying issue is that
STREAMING_STATE_TRANSFER.StreamingOutputStreamWrapper subclasses
java.io.FilterOutputStream which always calls write(int b), even if buffers are to be
written to the underlying socket.
As a consequence each byte to be transferred results in a call to write() to the
underlying socket.
The fix is simple, the byte array variants of StreamingOutputStreamWrapper.write() should
delegate to the wrapped socket directly.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:15 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1223) STREAMING_STATE_TRANSFER abysmally slow
[
https://jira.jboss.org/browse/JGRP-1223?page=com.atlassian.jira.plugin.sy...
]
Vladimir Blagojevic commented on JGRP-1223:
-------------------------------------------
http://www.redirecttonull.com/?p=186
STREAMING_STATE_TRANSFER abysmally slow
---------------------------------------
Key: JGRP-1223
URL: https://jira.jboss.org/browse/JGRP-1223
Project: JGroups
Issue Type: Bug
Affects Versions: 2.8
Environment: Windows XP
Reporter: Kornelius Elstner
Assignee: Vladimir Blagojevic
Fix For: 2.6.16, 2.10
Attachments: JGRP-1223.zip
With a relatively large shared state (anything in excess of a few MBytes)
STREAMING_STATE_TRANSFER is very slow, it can take minutes for the state transfer to
complete. The underlying issue is that
STREAMING_STATE_TRANSFER.StreamingOutputStreamWrapper subclasses
java.io.FilterOutputStream which always calls write(int b), even if buffers are to be
written to the underlying socket.
As a consequence each byte to be transferred results in a call to write() to the
underlying socket.
The fix is simple, the byte array variants of StreamingOutputStreamWrapper.write() should
delegate to the wrapped socket directly.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:47 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1223) STREAMING_STATE_TRANSFER abysmally slow
[
https://jira.jboss.org/browse/JGRP-1223?page=com.atlassian.jira.plugin.sy...
]
Kornelius Elstner commented on JGRP-1223:
-----------------------------------------
It's exactly the issue described in the /dev/null post above. There is a marked
difference in performance with the patch. Unpatched, with a shared state of about
50MBytes, it took about 550 seconds to complete the state transfer (100MBit link). With
the patch it went down to just under 8secs.
STREAMING_STATE_TRANSFER abysmally slow
---------------------------------------
Key: JGRP-1223
URL: https://jira.jboss.org/browse/JGRP-1223
Project: JGroups
Issue Type: Bug
Affects Versions: 2.8
Environment: Windows XP
Reporter: Kornelius Elstner
Assignee: Vladimir Blagojevic
Fix For: 2.6.16, 2.10
Attachments: JGRP-1223.zip
With a relatively large shared state (anything in excess of a few MBytes)
STREAMING_STATE_TRANSFER is very slow, it can take minutes for the state transfer to
complete. The underlying issue is that
STREAMING_STATE_TRANSFER.StreamingOutputStreamWrapper subclasses
java.io.FilterOutputStream which always calls write(int b), even if buffers are to be
written to the underlying socket.
As a consequence each byte to be transferred results in a call to write() to the
underlying socket.
The fix is simple, the byte array variants of StreamingOutputStreamWrapper.write() should
delegate to the wrapped socket directly.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:22 p.m.
New subject: [JBoss JIRA] Resolved: (JGRP-1223) STREAMING_STATE_TRANSFER abysmally slow
[
https://jira.jboss.org/browse/JGRP-1223?page=com.atlassian.jira.plugin.sy...
]
Vladimir Blagojevic resolved JGRP-1223.
---------------------------------------
Fix Version/s: (was: 2.6.16)
Resolution: Done
STREAMING_STATE_TRANSFER abysmally slow
---------------------------------------
Key: JGRP-1223
URL: https://jira.jboss.org/browse/JGRP-1223
Project: JGroups
Issue Type: Bug
Affects Versions: 2.8
Environment: Windows XP
Reporter: Kornelius Elstner
Assignee: Vladimir Blagojevic
Fix For: 2.10
Attachments: JGRP-1223.zip
With a relatively large shared state (anything in excess of a few MBytes)
STREAMING_STATE_TRANSFER is very slow, it can take minutes for the state transfer to
complete. The underlying issue is that
STREAMING_STATE_TRANSFER.StreamingOutputStreamWrapper subclasses
java.io.FilterOutputStream which always calls write(int b), even if buffers are to be
written to the underlying socket.
As a consequence each byte to be transferred results in a call to write() to the
underlying socket.
The fix is simple, the byte array variants of StreamingOutputStreamWrapper.write() should
delegate to the wrapped socket directly.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:22 p.m.
New subject: [JBoss JIRA] Commented: (JGRP-1223) STREAMING_STATE_TRANSFER abysmally slow
[
https://jira.jboss.org/browse/JGRP-1223?page=com.atlassian.jira.plugin.sy...
]
Vladimir Blagojevic commented on JGRP-1223:
-------------------------------------------
Branch 2.6 is not affected. The fix I applied to HEAD overrides the
FilterOutputStream.write(byte, int, int) method. Other write methods use this overridden
method.
STREAMING_STATE_TRANSFER abysmally slow
---------------------------------------
Key: JGRP-1223
URL: https://jira.jboss.org/browse/JGRP-1223
Project: JGroups
Issue Type: Bug
Affects Versions: 2.8
Environment: Windows XP
Reporter: Kornelius Elstner
Assignee: Vladimir Blagojevic
Fix For: 2.10
Attachments: JGRP-1223.zip
With a relatively large shared state (anything in excess of a few MBytes)
STREAMING_STATE_TRANSFER is very slow, it can take minutes for the state transfer to
complete. The underlying issue is that
STREAMING_STATE_TRANSFER.StreamingOutputStreamWrapper subclasses
java.io.FilterOutputStream which always calls write(int b), even if buffers are to be
written to the underlying socket.
As a consequence each byte to be transferred results in a call to write() to the
underlying socket.
The fix is simple, the byte array variants of StreamingOutputStreamWrapper.write() should
delegate to the wrapped socket directly.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5214
days inactive
5214
days old
8 comments
3 participants
participants (3)
-
Bela Ban (JIRA) -
Kornelius Elstner (JIRA)
-
Vladimir Blagojevic (JIRA)