2:37 a.m.
OK, I committed a first version of udp-largecluster.xml [1]. I recommend
use it instead of the config you posted below.
Comments regarding your config below:
- UDP.max_bundle_size of 4MB won't work ! Datagram packets can only be
65535 bytes max, so any packet larger than that would get dropped !
- UDP: the thread pools' max sizes are too small, I increased them. Note
that idle threads will be reaped
- PING.timeout= 360 secs ? A node will potentially block for 360 secs ?
- PING.break_on_coord_rsp is false ? Don't you want to return when the
coordinator has sent you the discovery response ?
- I changed UNICAST to UNICAST2; the latter has no acks for every message
- Note that STABLE.max_bytes is automatically adjusted if
ergonomics=true, based on the cluster size
- GMS.join_timeout = 60 secs ?
- I increased GMS.max_bundling_time (from 50ms) for view bundling; this
handles concurrent connects much better
- FRAG2.frag_size == 2MB ? This has to be smaller than
UDP.max_bundle_size. I suggest 60K (see [1])
- Remove FLUSH; FLUSH can be used in clusters up to 20-30 nodes, but
won't scale to larger clusters
[1] https://github.com/belaban/JGroups/blob/master/conf/udp-largecluster.xml
On 3/22/11 2:05 AM, Dave wrote:
I switched back to UDP today based on your feedback. Our config
resembles
the config below. Like I said we just increased sizes and timeouts. If you
ask me why I tweaked a certain parameter my response would be that it seemed
like a good idea based on the JGroups documentation. UDP seemed a little
more problematic than TCP, not sure why though.
<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups file:schema/JGroups-2.8.xsd">
<UDP
mcast_addr="${jgroups.udp.mcast_addr:228.6.7.8}"
mcast_port="${jgroups.udp.mcast_port:46655}"
tos="8"
ucast_recv_buf_size="20000000"
ucast_send_buf_size="640000"
mcast_recv_buf_size="25000000"
mcast_send_buf_size="640000"
loopback="true"
discard_incompatible_packets="true"
max_bundle_size="4000000"
max_bundle_timeout="30"
ip_ttl="${jgroups.udp.ip_ttl:2}"
enable_bundling="true"
enable_diagnostics="false"
thread_naming_pattern="pl"
thread_pool.enabled="true"
thread_pool.min_threads="2"
thread_pool.max_threads="30"
thread_pool.keep_alive_time="5000"
thread_pool.queue_enabled="true"
thread_pool.queue_max_size="1000"
thread_pool.rejection_policy="Discard"
oob_thread_pool.enabled="true"
oob_thread_pool.min_threads="2"
oob_thread_pool.max_threads="30"
oob_thread_pool.keep_alive_time="5000"
oob_thread_pool.queue_enabled="true"
oob_thread_pool.queue_max_size="1000"
oob_thread_pool.rejection_policy="Discard"
/>
<PING timeout="360000" num_initial_members="400"
break_on_coord_rsp="false"/>
<MERGE2 max_interval="30000" min_interval="10000"/>
<FD_SOCK/>
<FD_ALL/>
<BARRIER />
<pbcast.NAKACK use_stats_for_retransmission="false"
exponential_backoff="0"
use_mcast_xmit="true" gc_lag="0"
retransmit_timeout="300,600,1200,2400,3600,4800"
discard_delivered_msgs="true"/>
<UNICAST timeout="300,600,1200,2400,3600,4800"/>
<pbcast.STABLE stability_delay="1000"
desired_avg_gossip="50000"
max_bytes="1000000"/>
<pbcast.GMS print_local_addr="false" join_timeout="60000"
view_bundling="true" use_flush_if_present="false"/>
<UFC max_credits="2FRAG000" min_threshold="0.20"/>
<MFC max_credits="2000000" min_threshold="0.20"/>
<FRAG2 frag_size="2000000" />
<pbcast.STREAMING_STATE_TRANSFER/>
<!--<pbcast.STATE_TRANSFER/> -->
<pbcast.FLUSH timeout="0"/>
</config>
--
Bela Ban
Lead JGroups / Clustering Team
JBoss