[jboss-cvs] JBossAS SVN: r82582 - in projects/docs/community/5/Clustering_Guide/en-US: images and 1 other directory.
jboss-cvs-commits at lists.jboss.org
jboss-cvs-commits at lists.jboss.org
Thu Jan 1 15:55:34 EST 2009
Author: bstansberry at jboss.com
Date: 2009-01-01 15:55:34 -0500 (Thu, 01 Jan 2009)
New Revision: 82582
Added:
projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_JBoss_Cache.xml
projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_JGroups.xml
projects/docs/community/5/Clustering_Guide/en-US/images/JGroupsStack.png
Removed:
projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_JBoss_Cache_JGroups.xml
projects/docs/community/5/Clustering_Guide/en-US/images/jbosscache-JGroupsStack.png
Modified:
projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide.xml
projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_EJBs.xml
projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_HTTP.xml
projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_Introduction.xml
Log:
[JBAS-6350] Break JBoss Cache refs out of the JGroups chapter.
Modified: projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide.xml
===================================================================
--- projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide.xml 2009-01-01 20:45:49 UTC (rev 82581)
+++ projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide.xml 2009-01-01 20:55:34 UTC (rev 82582)
@@ -10,5 +10,6 @@
<xi:include href="Clustering_Guide_HTTP.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
<xi:include href="Clustering_Guide_HASingleton.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
<xi:include href="Clustering_Guide_JMS.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
-<xi:include href="Clustering_Guide_JBoss_Cache_JGroups.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
+<xi:include href="Clustering_Guide_JGroups.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
+<xi:include href="Clustering_Guide_JBoss_Cache.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
</book>
Modified: projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_EJBs.xml
===================================================================
--- projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_EJBs.xml 2009-01-01 20:45:49 UTC (rev 82581)
+++ projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_EJBs.xml 2009-01-01 20:55:34 UTC (rev 82582)
@@ -481,7 +481,7 @@
<para>The configuration attributes in this MBean are essentially the same as the attributes in the standard JBoss Cache <literal>TreeCache</literal> MBean discussed
in <xref linkend="jbosscache.chapt"/>. Again, we omitted the JGroups configurations in the
- <literal>ClusterConfig</literal> attribute (see more in <xref linkend="jbosscache-jgroups"/>). Two noteworthy items:</para>
+ <literal>ClusterConfig</literal> attribute (see more in <xref linkend="jgroups-configuration"/>). Two noteworthy items:</para>
<itemizedlist>
<listitem>
<para>The cache is configured to support eviction. The EJB3 SFSB container uses the JBoss Cache eviction mechanism to manage SFSB passivation. When beans are deployed, the EJB container will programatically add eviction regions to the cache, one region per bean type.
Modified: projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_HTTP.xml
===================================================================
--- projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_HTTP.xml 2009-01-01 20:45:49 UTC (rev 82581)
+++ projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_HTTP.xml 2009-01-01 20:55:34 UTC (rev 82582)
@@ -316,7 +316,7 @@
(see later). Otherwise, they are default to <literal>false</literal>.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">ClusterConfig</emphasis> configures the underlying JGroups stack. Please refer to <xref linkend="jbosscache-jgroups"/> for more information.</para>
+ <para><emphasis role="bold">ClusterConfig</emphasis> configures the underlying JGroups stack. Please refer to <xref linkend="jgroups-configuration"/> for more information.</para>
</listitem>
<listitem>
<para><emphasis role="bold">LockAcquisitionTimeout</emphasis> sets the maximum number of
Modified: projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_Introduction.xml
===================================================================
--- projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_Introduction.xml 2009-01-01 20:45:49 UTC (rev 82581)
+++ projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_Introduction.xml 2009-01-01 20:55:34 UTC (rev 82582)
@@ -2,7 +2,7 @@
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.3//EN" "http://www.oasis-open.org/docbook/xml/4.3/docbookx.dtd">
<chapter id="cluster.chapt">
- <title>Clustering</title>
+ <title>Clustering Concepts</title>
<subtitle>High Availability Enterprise Services via JBoss Clusters</subtitle>
<section id="clustering-intro">
@@ -85,7 +85,7 @@
</mbean>
</programlisting>
<para>Here, we omitted the detailed JGroups protocol configuration for this channel. JGroups handles the
- underlying peer-to-peer communication between nodes, and its configuration is discussed in <xref linkend="jbosscache-jgroups"/>. The following list shows the available configuration attributes
+ underlying peer-to-peer communication between nodes, and its configuration is discussed in <xref linkend="jgroups.chapt"/>. The following list shows the available configuration attributes
in the <literal>HAPartition</literal> MBean.</para>
<itemizedlist>
<listitem>
@@ -105,7 +105,7 @@
</listitem>
<listitem>
<para><emphasis role="bold">PartitionConfig</emphasis> is an element to specify JGroup
- configuration options for this cluster (see <xref linkend="jbosscache-jgroups"/>).</para>
+ configuration options for this cluster (see <xref linkend="jgroups-configuration"/>).</para>
</listitem>
</itemizedlist>
<para>In order for nodes to form a cluster, they must have the exact same <literal>PartitionName</literal> and the <literal>ParitionConfig</literal> elements. Changes in either element on some but not all nodes would cause the cluster to split.
Added: projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_JBoss_Cache.xml
===================================================================
--- projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_JBoss_Cache.xml (rev 0)
+++ projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_JBoss_Cache.xml 2009-01-01 20:55:34 UTC (rev 82582)
@@ -0,0 +1,27 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.3//EN" "http://www.oasis-open.org/docbook/xml/4.3/docbookx.dtd">
+
+<chapter id="jbosscache.chapt">
+ <title>JBoss Cache Services</title>
+ <para>JBoss Cache provides the underlying distributed caching support used
+ by many of the standard clustered services in a JBoss AS cluster, including:
+ <itemizedlist>
+ <listitem><para>Replication of clustered webapp sessions.</para></listitem>
+ <listitem><para>Replication of clustered EJB3 Stateful Session beans.</para></listitem>
+ <listitem><para>Clustered caching of JPA and Hibernate entities.</para></listitem>
+ <listitem><para>Clustered Single Sign-On.</para></listitem>
+ <listitem><para>The HA-JNDI replicated tree.</para></listitem>
+ <listitem><para>DistributedStateService</para></listitem>
+ </itemizedlist>
+ </para>
+ <para>JBoss AS ships with a reasonable set of default JBoss Cache configurations.
+ Most applications just work out of the box with the default configurations.
+ You only need to tweak them when you are deploying an application that
+ has special network or performance requirements.</para>
+
+ <section id="jbosscache-cachemanager">
+ <title>CacheManager Service</title>
+ <para></para>
+ </section>
+
+ </chapter>
Deleted: projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_JBoss_Cache_JGroups.xml
===================================================================
--- projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_JBoss_Cache_JGroups.xml 2009-01-01 20:45:49 UTC (rev 82581)
+++ projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_JBoss_Cache_JGroups.xml 2009-01-01 20:55:34 UTC (rev 82582)
@@ -1,1231 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.3//EN" "http://www.oasis-open.org/docbook/xml/4.3/docbookx.dtd">
-
-<chapter id="jbosscache.chapt">
- <title>JBossCache and JGroups Services</title>
- <para>JGroups and JBossCache provide the underlying communication, node replication and caching services, for
- JBoss AS clusters. Those services are configured as MBeans. There is a set of JBossCache and JGroups MBeans
- for each type of clustering applications (e.g., the Stateful Session EJBs, HTTP session replication etc.).
- </para>
- <para>The JBoss AS ships with a reasonable set of default JGroups and JBossCache MBean configurations. Most
- applications just work out of the box with the default MBean configurations. You only need to tweak them
- when you are deploying an application that has special network or performance requirements.</para>
-
- <section id="jbosscache-jgroups">
- <title>JGroups Configuration</title>
- <para>The JGroups framework provides services to enable peer-to-peer communications between nodes in a
- cluster. It is built on top a stack of network communication protocols that provide transport,
- discovery, reliability and failure detection, and cluster membership management services. <xref linkend="jbosscache-JGroupsStack.fig"/> shows the protocol stack in JGroups.</para>
- <figure id="jbosscache-JGroupsStack.fig">
- <title>Protocol stack in JGroups</title>
- <mediaobject>
- <imageobject>
- <imagedata align="center" fileref="images/jbosscache-JGroupsStack.png"/>
- </imageobject>
- </mediaobject>
- </figure>
- <para>JGroups configurations often appear as a nested attribute in cluster related MBean services, such as
- the <literal>PartitionConfig</literal> attribute in the <literal>ClusterPartition</literal> MBean or the
- <literal>ClusterConfig</literal> attribute in the <literal>TreeCache</literal> MBean. You can
- configure the behavior and properties of each protocol in JGroups via those MBean attributes. Below is
- an example JGroups configuration in the <literal>ClusterPartition</literal> MBean.</para>
-<programlisting><![CDATA[
-<mbean code="org.jboss.ha.framework.server.ClusterPartition"
- name="jboss:service=${jboss.partition.name:DefaultPartition}">
-
- ... ...
-
- <attribute name="PartitionConfig">
- <Config>
-
- <UDP mcast_addr="${jboss.partition.udpGroup:228.1.2.3}"
- mcast_port="${jboss.hapartition.mcast_port:45566}"
- tos="8"
- ucast_recv_buf_size="20000000"
- ucast_send_buf_size="640000"
- mcast_recv_buf_size="25000000"
- mcast_send_buf_size="640000"
- loopback="false"
- discard_incompatible_packets="true"
- enable_bundling="false"
- max_bundle_size="64000"
- max_bundle_timeout="30"
- use_incoming_packet_handler="true"
- use_outgoing_packet_handler="false"
- ip_ttl="${jgroups.udp.ip_ttl:2}"
- down_thread="false" up_thread="false"/>
-
- <PING timeout="2000"
- down_thread="false" up_thread="false" num_initial_members="3"/>
-
- <MERGE2 max_interval="100000"
- down_thread="false" up_thread="false" min_interval="20000"/>
- <FD_SOCK down_thread="false" up_thread="false"/>
-
- <FD timeout="10000" max_tries="5"
- down_thread="false" up_thread="false" shun="true"/>
- <VERIFY_SUSPECT timeout="1500" down_thread="false" up_thread="false"/>
- <pbcast.NAKACK max_xmit_size="60000"
- use_mcast_xmit="false" gc_lag="0"
- retransmit_timeout="300,600,1200,2400,4800"
- down_thread="false" up_thread="false"
- discard_delivered_msgs="true"/>
- <UNICAST timeout="300,600,1200,2400,3600"
- down_thread="false" up_thread="false"/>
- <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
- down_thread="false" up_thread="false"
- max_bytes="400000"/>
- <pbcast.GMS print_local_addr="true" join_timeout="3000"
- down_thread="false" up_thread="false"
- join_retry_timeout="2000" shun="true"
- view_bundling="true"/>
- <FRAG2 frag_size="60000" down_thread="false" up_thread="false"/>
- <pbcast.STATE_TRANSFER down_thread="false"
- up_thread="false" use_flush="false"/>
- </Config>
- </attribute>
-</mbean> ]]>
-</programlisting>
-<para>
- All the JGroups configuration data is contained in the <Config> element under the JGroups config MBean attribute. This information is used to configure a JGroups Channel; the Channel is conceptually similar to a socket, and manages communication between peers in a cluster. Each element inside the <Config> element defines a particular JGroups Protocol; each Protocol performs one function, and the combination of those functions is what defines the characteristics of the overall Channel. In the next several sections, we will dig into the commonly used protocols and their options and explain exactly what they mean.
-</para>
-</section>
-
-<section><title>Common Configuration Properties</title>
- <para>The following common properties are exposed by all of the JGroups protocols discussed below:
-</para>
-<itemizedlist>
- <listitem>
- <para><literal>down_thread</literal> whether the protocol should create an internal queue and a queue processing thread (aka the down_thread) for messages passed down from higher layers. The higher layer could be another protocol higher in the stack, or the application itself, if the protocol is the top one on the stack. If true (the default), when a message is passed down from a higher layer, the calling thread places the message in the protocol's queue, and then returns immediately. The protocol's down_thread is responsible for reading messages off the queue, doing whatever protocol-specific processing is required, and passing the message on to the next protocol in the stack.
- </para>
- </listitem>
- <listitem>
- <para><literal>up_thread</literal> is conceptually similar to down_thread, but here the queue and thread are for messages received from lower layers in the protocol stack.
- </para>
- </listitem>
-</itemizedlist>
-<para>Generally speaking, <literal>up_thread</literal> and <literal>down_thread</literal> should be set to false.
-</para>
-
-</section>
-
-<section id="jbosscache-jgroups-transport">
- <title>Transport Protocols</title>
- <para>The transport protocols send messages from one cluster node to another (unicast) or from cluster
- node to all other nodes in the cluster (mcast). JGroups supports UDP, TCP, and TUNNEL as transport
- protocols.</para>
- <note>
- <para>The <literal>UDP</literal>, <literal>TCP</literal>, and <literal>TUNNEL</literal> elements are
- mutually exclusive. You can only have one transport protocol in each JGroups
- <literal>Config</literal> element</para>
- </note>
- <section id="jbosscache-jgroups-transport-udp">
- <title>UDP configuration</title>
- <para>UDP is the preferred protocol for JGroups. UDP uses multicast or multiple unicasts to send and
- receive messages. If you choose UDP as the transport protocol for your cluster service, you need
- to configure it in the <literal>UDP</literal> sub-element in the JGroups
- <literal>Config</literal> element. Here is an example.</para>
-<programlisting><![CDATA[
-<UDP mcast_addr="${jboss.partition.udpGroup:228.1.2.3}"
- mcast_port="${jboss.hapartition.mcast_port:45566}"
- tos="8"
- ucast_recv_buf_size="20000000"
- ucast_send_buf_size="640000"
- mcast_recv_buf_size="25000000"
- mcast_send_buf_size="640000"
- loopback="false"
- discard_incompatible_packets="true"
- enable_bundling="false"
- max_bundle_size="64000"
- max_bundle_timeout="30"
- use_incoming_packet_handler="true"
- use_outgoing_packet_handler="false"
- ip_ttl="${jgroups.udp.ip_ttl:2}"
- down_thread="false" up_thread="false"/>]]>
-</programlisting>
-
-
- <para>The available attributes in the above JGroups configuration are listed below.</para>
- <itemizedlist>
- <listitem>
- <para><emphasis role="bold">ip_mcast</emphasis> specifies whether or not to use IP
- multicasting. The default is <literal>true</literal>. If set to false, it will send n unicast packets rather than 1 multicast packet. Either way, packets are UDP datagrams.
- </para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">mcast_addr</emphasis> specifies the multicast address (class D) for joining a group (i.e., the cluster). If omitted, the default is <literal>228.8.8.8
- </literal>.
- </para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">mcast_port</emphasis> specifies the multicast port number. If omitted, the
- default is <literal>45566</literal>.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">bind_addr</emphasis> specifies the interface on which to receive and send multicasts (uses the <literal>-Djgroups.bind_address</literal> system property, if present). If you have a multihomed machine, set the <literal>bind_addr</literal> attribute or system property to the appropriate NIC IP address. By default, system property setting takes priority over XML attribute unless -Djgroups.ignore.bind_addr system property is set.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">receive_on_all_interfaces </emphasis> specifies whether this node
- should listen on all interfaces for multicasts. The default is <literal>false</literal>.
- It overrides the <literal>bind_addr</literal> property for receiving multicasts.
- However, <literal>bind_addr</literal> (if set) is still used to send multicasts.</para>
- </listitem>
- <listitem><para><emphasis role="bold">send_on_all_interfaces</emphasis> specifies whether this node send UDP packets via all the NICs if you have a multi NIC machine. This means that the same multicast message is sent N times, so use with care.
- </para>
- </listitem>
-
- <listitem>
- <para><emphasis role="bold">receive_interfaces</emphasis> specifies a list of of interfaces to receive multicasts on. The multicast receive socket will listen on all of these interfaces. This is a comma-separated list of IP addresses or interface names. E.g. "<literal>192.168.5.1,eth1,127.0.0.1</literal>".
- </para>
- </listitem>
-
-
- <listitem>
- <para><emphasis role="bold">ip_ttl</emphasis> specifies time-to-live for IP Multicast packets. TTL is the commonly used term in multicast networking, but is actually something of a misnomer, since the value here refers to how many network hops a packet will be allowed to travel before networking equipment will drop it.
- </para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">use_incoming_packet_handler</emphasis> specifies whether to use a separate thread to process incoming messages. Sometimes receivers are overloaded (they have to handle de-serialization etc). Packet handler is a separate thread taking care of de-serialization, receiver thread(s) simply put packet in queue and return immediately. Setting this to true adds one more thread. The default is <literal>true</literal>.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">use_outgoing_packet_handler</emphasis> specifies whether to use a separate thread to process outgoing messages. The default is false.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">enable_bundling</emphasis> specifies whether to enable message bundling.
- If it is <literal>true</literal>, the node would queue outgoing messages until
- <literal>max_bundle_size</literal> bytes have accumulated, or
- <literal>max_bundle_time</literal> milliseconds have elapsed, whichever occurs
- first. Then bundle queued messages into a large message and send it. The messages are
- unbundled at the receiver. The default is <literal>false</literal>.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">loopback</emphasis> specifies whether to loop outgoing message
- back up the stack. In <literal>unicast</literal> mode, the messages are sent to self. In
- <literal>mcast</literal> mode, a copy of the mcast message is sent. The default is <literal>false</literal></para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">discard_incompatibe_packets</emphasis> specifies whether to
- discard packets from different JGroups versions. Each message in the cluster is tagged
- with a JGroups version. When a message from a different version of JGroups is received,
- it will be discarded if set to true, otherwise a warning will be logged. The default is <literal>false</literal></para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">mcast_send_buf_size, mcast_recv_buf_size, ucast_send_buf_size,
- ucast_recv_buf_size</emphasis> define receive and send buffer sizes. It is good to
- have a large receiver buffer size, so packets are less likely to get dropped due to
- buffer overflow.</para>
- </listitem>
- <listitem>
- <para><literal>tos</literal> specifies traffic class for sending unicast and multicast datagrams.
- </para>
- </listitem>
-
- </itemizedlist>
- <note>
- <para>On Windows 2000 machines, because of the media sense feature being broken with multicast
- (even after disabling media sense), you need to set the UDP protocol's
- <literal>loopback</literal> attribute to <literal>true</literal>.</para>
- </note>
- </section>
-
-
- <section id="jbosscache-jgroups-transport-tcp">
- <title>TCP configuration</title>
- <para>Alternatively, a JGroups-based cluster can also work over TCP connections. Compared with UDP,
- TCP generates more network traffic when the cluster size increases. TCP
- is fundamentally a unicast protocol. To send multicast messages, JGroups uses multiple TCP
- unicasts. To use TCP as a transport protocol, you should define a <literal>TCP</literal> element
- in the JGroups <literal>Config</literal> element. Here is an example of the
- <literal>TCP</literal> element.</para>
- <programlisting>
-<TCP start_port="7800"
- bind_addr="192.168.5.1"
- loopback="true"
- down_thread="false" up_thread="false"/>
- </programlisting>
- <para>Below are the attributes available in the <literal>TCP</literal> element.</para>
- <itemizedlist>
- <listitem>
- <para><emphasis role="bold">bind_addr</emphasis> specifies the binding address. It can also
- be set with the <literal>-Djgroups.bind_address</literal> command line option at server
- startup.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">start_port, end_port</emphasis> define the range of TCP ports
- the server should bind to. The server socket is bound to the first available port from
- <literal>start_port</literal>. If no available port is found (e.g., because of a
- firewall) before the <literal>end_port</literal>, the server throws an exception. If no <literal>end_port</literal> is provided or <literal>end_port < start_port</literal> then there is no upper limit on the port range. If <literal>start_port == end_port</literal>, then we force JGroups to use the given port (start fails if port is not available). The default is 7800. If set to 0, then the operating system will pick a port. Please, bear in mind that setting it to 0 will work only if we use MPING or TCPGOSSIP as discovery protocol because <literal>TCCPING</literal> requires listing the nodes and their corresponding ports.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">loopback</emphasis> specifies whether to loop outgoing message
- back up the stack. In <literal>unicast</literal> mode, the messages are sent to self. In
- <literal>mcast</literal> mode, a copy of the mcast message is sent. The default is false.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">recv_buf_size, send_buf_size</emphasis> define receive and send buffer sizes. It is good to have a large receiver buffer size, so packets are less likely to get dropped due to buffer overflow.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">conn_expire_time</emphasis> specifies the time (in milliseconds)
- after which a connection can be closed by the reaper if no traffic has been
- received.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">reaper_interval</emphasis> specifies interval (in milliseconds)
- to run the reaper. If both values are 0, no reaping will be done. If either value is
- > 0, reaping will be enabled. By default, reaper_interval is 0, which means no reaper.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">sock_conn_timeout</emphasis> specifies max time in millis for a socket creation. When doing the initial discovery, and a peer hangs, don't wait forever but go on after the timeout to ping other members. Reduces chances of *not* finding any members at all. The default is 2000.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">use_send_queues</emphasis> specifies whether to use separate send queues for each connection. This prevents blocking on write if the peer hangs. The default is true.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">external_addr</emphasis> specifies external IP address to broadcast to other group members (if different to local address). This is useful when you have use (Network Address Translation) NAT, e.g. a node on a private network, behind a firewall, but you can only route to it via an externally visible address, which is different from the local address it is bound to. Therefore, the node can be configured to broadcast its external address, while still able to bind to the local one. This avoids having to use the TUNNEL protocol, (and hence a requirement for a central gossip router) because nodes outside the firewall can still route to the node inside the firewall, but only on its external address. Without setting the external_addr, the node behind the firewall will broadcast its private address to the other nodes which will not be able to route to it.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">skip_suspected_members</emphasis> specifies whether unicast messages should not be sent to suspected members. The default is true.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">tcp_nodelay</emphasis> specifies TCP_NODELAY. TCP by default nagles messages, that is, conceptually, smaller messages are bundled into larger ones. If we want to invoke synchronous cluster method calls, then we need to disable nagling in addition to disabling message bundling (by setting <literal>enable_bundling</literal> to false). Nagling is disabled by setting <literal>tcp_nodelay</literal> to true. The default is false.
- </para>
- </listitem>
-
- </itemizedlist>
- </section>
-
- <section id="jbosscache-jgroups-transport-tunnel">
- <title>TUNNEL configuration</title>
- <para>The TUNNEL protocol uses an external router to send messages. The external router is known as
- a <literal>GossipRouter</literal>. Each node has to register with the router. All messages are sent to the router and forwarded on to their destinations. The TUNNEL approach can be used to setup communication with nodes behind firewalls. A node can establish a TCP connection to the GossipRouter through the firewall (you can use port 80). The same connection is used by the router to send messages to nodes behind the firewall as most firewalls do not permit outside hosts to initiate a TCP connection to a host inside the firewall. The TUNNEL configuration is defined in the TUNNEL element in the JGroups Config element. Here is an example..
- </para>
-
-
- <programlisting>
-<TUNNEL router_port="12001"
- router_host="192.168.5.1"
- down_thread="false" up_thread="false/>
- </programlisting>
-
-
-<para>The available attributes in the <literal>TUNNEL</literal> element are listed below.</para>
- <itemizedlist>
- <listitem>
- <para><emphasis role="bold">router_host</emphasis> specifies the host on which the
- GossipRouter is running.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">router_port</emphasis> specifies the port on which the
- GossipRouter is listening.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">loopback</emphasis> specifies whether to loop messages back up
- the stack. The default is <literal>true</literal>.</para>
- </listitem>
- </itemizedlist>
- </section>
- </section>
-
-
-
-
- <section id="jbosscache-jgroups-discovery">
- <title>Discovery Protocols</title>
- <para>
- The cluster needs to maintain a list of current member nodes at all times so that the load balancer and client interceptor know how to route their requests. Discovery protocols are used to discover active nodes in the cluster and detect the oldest member of the cluster, which is the coordinator. All initial nodes are discovered when the cluster starts up.
- When a new node joins the cluster later, it is only discovered after the group membership protocol
- (GMS, see <xref linkend="jbosscache-jgroups-other-gms"/>) admits it into the group.</para>
- <para>Since the discovery protocols sit on top of the transport protocol, you can choose to use different discovery protocols based on your transport protocol. These are also configured as sub-elements in the JGroups MBean <literal>Config</literal> element.</para>
-
-
-
-
- <section id="jbosscache-jgroups-discovery-ping">
- <title>PING</title>
- <para>
- PING is a discovery protocol that works by either multicasting PING requests to an IP multicast address or connecting to a gossip router. As such, PING normally sits on top of the UDP or TUNNEL transport protocols. Each node responds with a packet {C, A}, where C=coordinator's address and A=own address. After timeout milliseconds or num_initial_members replies, the joiner determines the coordinator from the responses, and sends a JOIN request to it (handled by). If nobody responds, we assume we are the first member of a group.
- </para>
- <para>Here is an example PING configuration for IP multicast.
- </para>
-
-
- <programlisting>
-<PING timeout="2000"
- num_initial_members="2"
- down_thread="false" up_thread="false"/>
- </programlisting>
-<para>
- Here is another example PING configuration for contacting a Gossip Router.
-<programlisting><![CDATA[
-<PING gossip_host="localhost"
- gossip_port="1234"
- timeout="3000"
- num_initial_members="3"
- down_thread="false" up_thread="false"/>]]>
-</programlisting>
-
- </para>
-
-
- <para>The available attributes in the <literal>PING</literal> element are listed below.</para>
- <itemizedlist>
- <listitem>
- <para><emphasis role="bold">timeout</emphasis> specifies the maximum number of milliseconds
- to wait for any responses. The default is 3000.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">num_initial_members</emphasis> specifies the maximum number of
- responses to wait for unless timeout has expired. The default is 2.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">gossip_host</emphasis> specifies the host on which the
- GossipRouter is running.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">gossip_port</emphasis> specifies the port on which the
- GossipRouter is listening on.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">gossip_refresh</emphasis> specifies the interval (in
- milliseconds) for the lease from the GossipRouter. The default is 20000.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">initial_hosts</emphasis> is a comma-seperated list of addresses
- (e.g., <literal>host1[12345],host2[23456]</literal>), which are pinged for
- discovery.</para>
- </listitem>
- </itemizedlist>
- <para>If both <literal>gossip_host</literal> and <literal>gossip_port</literal> are defined, the
- cluster uses the GossipRouter for the initial discovery. If the <literal>initial_hosts</literal>
- is specified, the cluster pings that static list of addresses for discovery. Otherwise, the
- cluster uses IP multicasting for discovery.</para>
- <note>
- <para>The discovery phase returns when the <literal>timeout</literal> ms have elapsed or the
- <literal>num_initial_members</literal> responses have been received.</para>
- </note>
- </section>
-
-
-
- <section id="jbosscache-jgroups-discovery-tcpgossip">
- <title>TCPGOSSIP</title>
- <para>The TCPGOSSIP protocol only works with a GossipRouter. It works essentially the same way as
- the PING protocol configuration with valid <literal>gossip_host</literal> and
- <literal>gossip_port</literal> attributes. It works on top of both UDP and TCP transport protocols. Here is an example.</para>
-<programlisting><![CDATA[
-<TCPGOSSIP timeout="2000"
- initial_hosts="192.168.5.1[12000],192.168.0.2[12000]"
- num_initial_members="3"
- down_thread="false" up_thread="false"/>]]>
-</programlisting>
-
-
- <para>The available attributes in the <literal>TCPGOSSIP</literal> element are listed below.</para>
- <itemizedlist>
- <listitem>
- <para><emphasis role="bold">timeout</emphasis> specifies the maximum number of milliseconds
- to wait for any responses. The default is 3000.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">num_initial_members</emphasis> specifies the maximum number of
- responses to wait for unless timeout has expired. The default is 2.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">initial_hosts</emphasis> is a comma-seperated list of addresses
- (e.g., <literal>host1[12345],host2[23456]</literal>) for GossipRouters to register
- with.</para>
- </listitem>
- </itemizedlist>
- </section>
-
-
-
- <section id="jbosscache-jgroups-discovery-tcpping">
- <title>TCPPING</title>
- <para>The TCPPING protocol takes a set of known members and ping them for discovery. This is
- essentially a static configuration. It works on top of TCP. Here is an example of the
- <literal>TCPPING</literal> configuration element in the JGroups <literal>Config</literal>
- element.</para>
- <programlisting>
-<TCPPING timeout="2000"
- initial_hosts="hosta[2300],hostb[3400],hostc[4500]"
- port_range="3"
- num_initial_members="3"
- down_thread="false" up_thread="false"/>
-</programlisting>
-
-
- <para>The available attributes in the <literal>TCPPING</literal> element are listed below.</para>
- <itemizedlist>
- <listitem>
- <para><emphasis role="bold">timeout</emphasis> specifies the maximum number of milliseconds
- to wait for any responses. The default is 3000.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">num_initial_members</emphasis> specifies the maximum number of
- responses to wait for unless timeout has expired. The default is 2.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">initial_hosts</emphasis> is a comma-seperated list of addresses
- (e.g., <literal>host1[12345],host2[23456]</literal>) for pinging.</para>
- </listitem>
- <listitem>
- <para>
- <emphasis role="bold">port_range</emphasis> specifies the number of consecutive ports to be probed when getting the initial membership, starting with the port specified in the initial_hosts parameter. Given the current values of port_range and initial_hosts above, the TCPPING layer will try to connect to hosta:2300, hosta:2301, hosta:2302, hostb:3400, hostb:3401, hostb:3402, hostc:4500, hostc:4501, hostc:4502. The configuration options allows for multiple nodes on the same host to be pinged.
- </para>
- </listitem>
- </itemizedlist>
- </section>
-
-
-
-
- <section id="jbosscache-jgroups-discovery-mping">
- <title>MPING</title>
- <para>
- MPING uses IP multicast to discover the initial membership. It can be used with all transports, but usually this is used in combination with TCP. TCP usually requires TCPPING, which has to list all group members explicitly, but MPING doesn't have this requirement. The typical use case for this is when we want TCP as transport, but multicasting for discovery so we don't have to define a static list of initial hosts in TCPPING or require external Gossip Router.
- </para>
-
-<programlisting>
-<MPING timeout="2000"
- bind_to_all_interfaces="true"
- mcast_addr="228.8.8.8"
- mcast_port="7500"
- ip_ttl="8"
- num_initial_members="3"
- down_thread="false" up_thread="false"/>
-</programlisting>
-
-
- <para>The available attributes in the <literal>MPING</literal> element are listed below.</para>
- <itemizedlist>
- <listitem>
- <para><emphasis role="bold">timeout</emphasis> specifies the maximum number of milliseconds
- to wait for any responses. The default is 3000.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">num_initial_members</emphasis> specifies the maximum number of
- responses to wait for unless timeout has expired. The default is 2..</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">bind_addr</emphasis> specifies the interface on which to send
- and receive multicast packets.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">bind_to_all_interfaces</emphasis> overrides the
- <literal>bind_addr</literal> and uses all interfaces in multihome nodes.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">mcast_addr, mcast_port, ip_ttl</emphasis> attributes are the
- same as related attributes in the UDP protocol configuration.</para>
- </listitem>
- </itemizedlist>
- </section>
- </section>
-
-
-
- <section id="jbosscache-jgroups-fd">
- <title>Failure Detection Protocols</title>
- <para>The failure detection protocols are used to detect failed nodes. Once a failed node is detected, a suspect verification phase can occur after which, if the node is still considered dead, the cluster updates its view so that the load balancer and client interceptors know to avoid the dead node. The failure detection protocols are configured as sub-elements in the JGroups MBean
- <literal>Config</literal> element.</para>
-
-
-
-<section id="jbosscache-jgroups-fd-fd">
- <title>FD</title>
- <para>
- FD is a failure detection protocol based on heartbeat messages. This protocol requires each node to periodically send are-you-alive messages to its neighbour. If the neighbour fails to respond, the calling node sends a SUSPECT message to the cluster. The current group coordinator can optionally double check whether the suspected node is indeed dead after which, if the node is still considered dead, updates the cluster's view. Here is an example FD configuration.
- </para>
-
-<programlisting>
-<FD timeout="2000"
- max_tries="3"
- shun="true"
- down_thread="false" up_thread="false"/>
-</programlisting>
-
-
- <para>The available attributes in the <literal>FD</literal> element are listed below.</para>
- <itemizedlist>
- <listitem>
- <para><emphasis role="bold">timeout</emphasis> specifies the maximum number of milliseconds
- to wait for the responses to the are-you-alive messages. The default is 3000.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">max_tries</emphasis> specifies the number of missed
- are-you-alive messages from a node before the node is suspected. The default is 2.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">shun</emphasis> specifies whether a failed node will be shunned.
- Once shunned, the node will be expelled from the cluster even if it comes back later.
- The shunned node would have to re-join the cluster through the discovery process. JGroups allows to configure itself such that shunning leads to automatic rejoins and state transfer, which is the default behaivour within JBoss Application Server.</para>
- </listitem>
- </itemizedlist>
- <note>
- <para>Regular traffic from a node counts as if it is a live. So, the are-you-alive messages are
- only sent when there is no regular traffic to the node for sometime.</para>
- </note>
- </section>
-
-
-
-
- <section id="jbosscache-jgroups-fd-fdsock">
- <title>FD_SOCK</title>
- <para>
-FD_SOCK is a failure detection protocol based on a ring of TCP sockets created between group members. Each member in a group connects to its neighbor (last member connects to first) thus forming a ring. Member B is suspected when its neighbor A detects abnormally closed TCP socket (presumably due to a node B crash). However, if a member B is about to leave gracefully, it lets its neighbor A know, so that it does not become suspected. The simplest FD_SOCK configuration does not take any attribute. You can just declare an empty <literal>FD_SOCK</literal> element in JGroups's <literal>Config</literal> element.</para>
-
-
-<programlisting>
-<FD_SOCK_down_thread="false" up_thread="false"/>
-</programlisting>
-
-<para>There available attributes in the <literal>FD_SOCK</literal> element are listed below.</para>
- <itemizedlist>
- <listitem>
- <para><emphasis role="bold">bind_addr</emphasis> specifies the interface to which the server socket should bind to. If -Djgroups.bind_address system property is defined, XML value will be ignore. This behaivour can be reversed setting -Djgroups.ignore.bind_addr=true system property.</para>
- </listitem>
- </itemizedlist>
-
- </section>
-
-
-
- <section><title>VERIFY_SUSPECT</title>
- <para>
- This protocol verifies whether a suspected member is really dead by pinging that member once again. This verification is performed by the coordinator of the cluster. The suspected member is dropped from the cluster group if confirmed to be dead. The aim of this protocol is to minimize false suspicions. Here's an example.
- </para>
-
-<programlisting><![CDATA[
-<VERIFY_SUSPECT timeout="1500"
- down_thread="false" up_thread="false"/>]]>
-</programlisting>
-
- <para>
- The available attributes in the FD_SOCK element are listed below.
- </para>
- <itemizedlist>
- <listitem>
- <para>
- timeout specifies how long to wait for a response from the suspected member before considering it dead.
- </para>
-</listitem>
-</itemizedlist>
-
- </section>
-
-
-
-
- <section><title>FD versus FD_SOCK</title>
- <para>
- FD and FD_SOCK, each taken individually, do not provide a solid failure detection layer. Let's look at the the differences between these failure detection protocols to understand how they complement each other:
- </para>
- <itemizedlist>
- <listitem><para><emphasis>FD</emphasis></para>
- </listitem>
- </itemizedlist>
- <itemizedlist>
- <listitem>
- <para>
- An overloaded machine might be slow in sending are-you-alive responses.
- </para>
- </listitem>
- <listitem>
- <para>
- A member will be suspected when suspended in a debugger/profiler.
- </para>
- </listitem>
- <listitem>
- <para>
- Low timeouts lead to higher probability of false suspicions and higher network traffic.
- </para>
- </listitem>
- <listitem>
- <para>
- High timeouts will not detect and remove crashed members for some time.
- </para>
- </listitem>
-</itemizedlist>
-
-<itemizedlist>
-<listitem><para><emphasis>FD_SOCK</emphasis>:</para>
-</listitem>
-</itemizedlist>
-
-<itemizedlist>
- <listitem>
- <para>
- Suspended in a debugger is no problem because the TCP connection is still open.
- </para>
- </listitem>
- <listitem>
- <para>
- High load no problem either for the same reason.
- </para>
- </listitem>
- <listitem>
- <para>
- Members will only be suspected when TCP connection breaks
- </para>
- </listitem>
-</itemizedlist>
-
-
- <itemizedlist>
- <listitem>
- <para>
- So hung members will not be detected.
- </para>
- </listitem>
- <listitem>
- <para>
-
- Also, a crashed switch will not be detected until the connection runs into the TCP timeout (between 2-20 minutes, depending on TCP/IP stack implementation).
- </para>
- </listitem>
-</itemizedlist>
-
-<para>
- The aim of a failure detection layer is to report real failures and therefore avoid false suspicions. There are two solutions:
- </para>
- <orderedlist>
- <listitem>
- <para>
- By default, JGroups configures the FD_SOCK socket with KEEP_ALIVE, which means that TCP sends a heartbeat on socket on which no traffic has been received in 2 hours. If a host crashed (or an intermediate switch or router crashed) without closing the TCP connection properly, we would detect this after 2 hours (plus a few minutes). This is of course better than never closing the connection (if KEEP_ALIVE is off), but may not be of much help. So, the first solution would be to lower the timeout value for KEEP_ALIVE. This can only be done for the entire kernel in most operating systems, so if this is lowered to 15 minutes, this will affect all TCP sockets.
- </para>
- </listitem>
- <listitem>
- <para>
- The second solution is to combine FD_SOCK and FD; the timeout in FD can be set such that it is much lower than the TCP timeout, and this can be configured individually per process. FD_SOCK will already generate a suspect message if the socket was closed abnormally. However, in the case of a crashed switch or host, FD will make sure the socket is eventually closed and the suspect message generated. Example:
- </para>
- </listitem>
-</orderedlist>
-<programlisting><![CDATA[
-<FD_SOCK down_thread="false" up_thread="false"/>
-<FD timeout="10000" max_tries="5" shun="true"
-down_thread="false" up_thread="false" /> ]]>
-</programlisting>
-
-<para>
- This suspects a member when the socket to the neighbor has been closed abonormally (e.g. process crash, because the OS closes all sockets). However, f a host or switch crashes, then the sockets won't be closed, therefore, as a seond line of defense, FD will suspect the neighbor after 50 seconds. Note that with this example, if you have your system stopped in a breakpoint in the debugger, the node you're debugging will be suspected after ca 50 seconds.
- </para>
-<para>
- A combination of FD and FD_SOCK provides a solid failure detection layer and for this reason, such technique is used accross JGroups configurations included within JBoss Application Server.
- </para>
- </section>
- </section>
-
-
-
- <section id="jbosscache-jgroups-reliable">
- <title>Reliable Delivery Protocols</title>
- <para>
- Reliable delivery protocols within the JGroups stack ensure that data pockets are actually delivered in the right order (FIFO) to the destination node. The basis for reliable message delivery is positive and negative delivery acknowledgments (ACK and NAK). In the ACK mode, the sender resends the message until the acknowledgment is received from the receiver. In the NAK mode, the receiver requests retransmission when it discovers a gap.
- </para>
-
-
-
- <section id="jbosscache-jgroups-reliable-unicast">
- <title>UNICAST</title>
- <para>
- The UNICAST protocol is used for unicast messages. It uses ACK. It is configured as a sub-element under the JGroups Config element. If the JGroups stack is configured with TCP transport protocol, UNICAST is not necessary because TCP itself guarantees FIFO delivery of unicast messages. Here is an example configuration for the <literal>UNICAST</literal> protocol.</para>
-
-<programlisting>
-<UNICAST timeout="100,200,400,800"
-down_thread="false" up_thread="false"/>
-</programlisting>
-
-<para>There is only one configurable attribute in the <literal>UNICAST</literal> element.</para>
- <itemizedlist>
- <listitem>
- <para><emphasis role="bold">timeout</emphasis> specifies the retransmission timeout (in
- milliseconds). For instance, if the timeout is "100,200,400,800", the sender resends the
- message if it hasn't received an ACK after 100 ms the first time, and the second time it
- waits for 200 ms before resending, and so on.</para>
- </listitem>
- </itemizedlist>
- </section>
-
-
- <section id="jbosscache-jgroups-reliable-nakack">
- <title>NAKACK</title>
- <para>The NAKACK protocol is used for multicast messages. It uses NAK. Under this protocol, each
- message is tagged with a sequence number. The receiver keeps track of the sequence numbers and
- deliver the messages in order. When a gap in the sequence number is detected, the receiver asks
- the sender to retransmit the missing message. The NAKACK protocol is configured as the
- <literal>pbcast.NAKACK</literal> sub-element under the JGroups <literal>Config</literal>
- element. Here is an example configuration.</para>
-
-<programlisting>
-<pbcast.NAKACK max_xmit_size="60000" use_mcast_xmit="false"
-
- retransmit_timeout="300,600,1200,2400,4800" gc_lag="0"
- discard_delivered_msgs="true"
- down_thread="false" up_thread="false"/>
-</programlisting>
-
-
-<para>The configurable attributes in the <literal>pbcast.NAKACK</literal> element are as follows.</para>
- <itemizedlist>
- <listitem>
- <para><emphasis role="bold">retransmit_timeout</emphasis> specifies the retransmission
- timeout (in milliseconds). It is the same as the <literal>timeout</literal> attribute in
- the UNICAST protocol.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">use_mcast_xmit</emphasis> determines whether the sender should
- send the retransmission to the entire cluster rather than just the node requesting it.
- This is useful when the sender drops the pocket -- so we do not need to retransmit for
- each node.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">max_xmit_size</emphasis> specifies maximum size for a bundled
- retransmission, if multiple packets are reported missing.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">discard_delivered_msgs</emphasis> specifies whether to discard
- delivery messages on the receiver nodes. By default, we save all delivered messages.
- However, if we only ask the sender to resend their messages, we can enable this option
- and discard delivered messages.</para>
- </listitem>
-
- <listitem>
- <para><emphasis role="bold">gc_lag specifies</emphasis> the number of messages garbage collection lags behind.
- </para>
- </listitem>
-
- </itemizedlist>
- </section>
- </section>
-
-
-
- <section id="jbosscache-jgroups-other">
- <title>Other Configuration Options</title>
- <para>In addition to the protocol stacks, you can also configure JGroups network services in the <literal>Config</literal> element.</para>
-
-
- <section id="jbosscache-jgroups-other-gms">
- <title>Group Membership</title>
- <para>The group membership service in the JGroups stack maintains a list of active nodes. It handles
- the requests to join and leave the cluster. It also handles the SUSPECT messages sent by failure
- detection protocols. All nodes in the cluster, as well as the load balancer and client side
- interceptors, are notified if the group membership changes. The group membership service is
- configured in the <literal>pbcast.GMS</literal> sub-element under the JGroups
- <literal>Config</literal> element. Here is an example configuration.</para>
-<programlisting>
-<pbcast.GMS print_local_addr="true"
- join_timeout="3000"
- down_thread="false" up_thread="false"
- join_retry_timeout="2000"
- shun="true"
- view_bundling="true"/>
-</programlisting>
-
-
-
-<para>The configurable attributes in the <literal>pbcast.GMS</literal> element are as follows.</para>
- <itemizedlist>
- <listitem>
- <para><emphasis role="bold">join_timeout</emphasis> specifies the maximum number of
- milliseconds to wait for a new node JOIN request to succeed. Retry afterwards.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">join_retry_timeout</emphasis> specifies the maximum number of
- milliseconds to wait after a failed JOIN to re-submit it.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">print_local_addr</emphasis> specifies whether to dump the node's
- own address to the output when started.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">shun</emphasis> specifies whether a node should shun itself if
- it receives a cluster view that it is not a member node.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">disable_initial_coord</emphasis> specifies whether to prevent
- this node as the cluster coordinator.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">view_bundling</emphasis> specifies whether multiple JOIN or LEAVE request arriving at the same time are bundled and handled together at the same time, only sending out 1 new view / bundle. This is is more efficient than handling each request separately.
- </para>
- </listitem>
-
- </itemizedlist>
- </section>
-
-
- <section id="jbosscache-jgroups-other-fc">
- <title>Flow Control</title>
- <para>The flow control service tries to adapt the sending data rate and the receiving data among
- nodes. If a sender node is too fast, it might overwhelm the receiver node and result in dropped
- packets that have to be retransmitted. In JGroups, the flow control is implemented via a
- credit-based system. The sender and receiver nodes have the same number of credits (bytes) to
- start with. The sender subtracts credits by the number of bytes in messages it sends. The
- receiver accumulates credits for the bytes in the messages it receives. When the sender's credit
- drops to a threshold, the receivers sends some credit to the sender. If the sender's credit is
- used up, the sender blocks until it receives credits from the receiver. The flow control service
- is configured in the <literal>FC</literal> sub-element under the JGroups
- <literal>Config</literal> element. Here is an example configuration.</para>
-
-<programlisting>
-<FC max_credits="1000000"
-down_thread="false" up_thread="false"
- min_threshold="0.10"/>
-</programlisting>
-
-
-<para>The configurable attributes in the <literal>FC</literal> element are as follows.</para>
- <itemizedlist>
- <listitem>
- <para><emphasis role="bold">max_credits</emphasis> specifies the maximum number of credits
- (in bytes). This value should be smaller than the JVM heap size.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">min_credits</emphasis> specifies the threshold credit on the
- sender, below which the receiver should send in more credits.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">min_threshold</emphasis> specifies percentage value of the
- threshold. It overrides the <literal>min_credits</literal> attribute.</para>
- </listitem>
- </itemizedlist>
-
-<note><title>Note</title>
- <para>
- Applications that use synchronous group RPC calls primarily do not require FC protocol in their JGroups protocol stack because synchronous communication, where the hread that makes the call blocks waiting for responses from all the members of the group, already slows overall rate of calls. Even though TCP provides flow control by itself, FC is still required in TCP based JGroups stacks because of group communication, where we essentially have to send group messages at the highest speed the slowest receiver can keep up with. TCP flow control only takes into account individual node communications and has not a notion of who's the slowest in the group, which is why FC is required.
- </para>
-</note>
-
-<section>
- <title>Why is FC needed on top of TCP ? TCP has its own flow control !</title>
- <para>
-
- The reason is group communication, where we essentially have to send group messages at the highest speed the slowest receiver can keep up with. Let's say we have a cluster {A,B,C,D}. D is slow (maybe overloaded), the rest is fast. When A sends a group message, it establishes TCP connections A-A (conceptually), A-B, A-C and A-D (if they don't yet exist). So let's say A sends 100 million messages to the cluster. Because TCP's flow control only applies to A-B, A-C and A-D, but not to A-{B,C,D}, where {B,C,D} is the group, it is possible that A, B and C receive the 100M, but D only received 1M messages. (BTW: this is also the reason why we need NAKACK, although TCP does its own retransmission).
- </para>
- <para>
- Now JGroups has to buffer all messages in memory for the case when the original sender S dies and a node asks for retransmission of a message of S. Because all members buffer all messages they received, they need to purge stable messages (= messages seen by everyone) every now and then. This is done by the STABLE protocol, which can be configured to run the stability protocol round time based (e.g. every 50s) or size based (whenever 400K data has been received).
- </para>
- <para>
- In the above case, the slow node D will prevent the group from purging messages above 1M, so every member will buffer 99M messages ! This in most cases leads to OOM exceptions. Note that - although the sliding window protocol in TCP will cause writes to block if the window is full - we assume in the above case that this is still much faster for A-B and A-C than for A-D.
- </para>
- <para>
- So, in summary, we need to send messages at a rate the slowest receiver (D) can handle.
- </para>
-</section>
-
-<section>
- <title>So do I always need FC?</title>
- <para>
- This depends on how the application uses the JGroups channel. Referring to the example above, if there was something about the application that would naturally cause A to slow down its rate of sending because D wasn't keeping up, then FC would not be needed.
- </para>
- <para>
- A good example of such an application is one that makes synchronous group RPC calls (typically using a JGroups RpcDispatcher.) By synchronous, we mean the thread that makes the call blocks waiting for responses from all the members of the group. In that kind of application, the threads on A that are making calls would block waiting for responses from D, thus naturally slowing the overall rate of calls.
- </para>
- <para>
- A JBoss Cache cluster configured for REPL_SYNC is a good example of an application that makes synchronous group RPC calls. If a channel is only used for a cache configured for REPL_SYNC, we recommend you remove FC from its protocol stack.
- </para>
- <para>
- And, of course, if your cluster only consists of two nodes, including FC in a TCP-based protocol stack is unnecessary. There is no group beyond the single peer-to-peer relationship, and TCP's internal flow control will handle that just fine.
- </para>
- <para>
- Another case where FC may not be needed is for a channel used by a JBoss Cache configured for buddy replication and a single buddy. Such a channel will in many respects act like a two node cluster, where messages are only exchanged with one other node, the buddy. (There may be other messages related to data gravitation that go to all members, but in a properly engineered buddy replication use case these should be infrequent. But if you remove FC be sure to load test your application.)
- </para>
-</section>
-
-
-
- </section>
-
-
-<section><title>Fragmentation</title>
- <para>
- This protocol fragments messages larger than certain size. Unfragments at the receiver's side. It works for both unicast and multicast messages. It is configured in the FRAG2 sub-element under the JGroups Config element. Here is an example configuration.
- </para>
-<programlisting><![CDATA[
- <FRAG2 frag_size="60000" down_thread="false" up_thread="false"/>]]>
-</programlisting>
-
-<para>
-The configurable attributes in the FRAG2 element are as follows.
-</para>
-
-<itemizedlist>
- <listitem><para><emphasis role="bold">frag_size</emphasis> specifies the max frag size in bytes. Messages larger than that are fragmented.</para></listitem>
-</itemizedlist>
-
-<note><title>Note</title>
- <para>
- TCP protocol already provides fragmentation but a fragmentation JGroups protocol is still needed if FC is used. The reason for this is that if you send a message larger than FC.max_bytes, FC protocol would block. So, frag_size within FRAG2 needs to be set to always be less than FC.max_bytes.
- </para>
-</note>
-
-
-</section>
-
- <section id="jbosscache-jgroups-other-st">
- <title>State Transfer</title>
- <para>The state transfer service transfers the state from an existing node (i.e., the cluster
- coordinator) to a newly joining node. It is configured in the
- <literal>pbcast.STATE_TRANSFER</literal> sub-element under the JGroups <literal>Config</literal>
- element. It does not have any configurable attribute. Here is an example configuration.</para>
-<programlisting>
-<pbcast.STATE_TRANSFER down_thread="false" up_thread="false"/>
-</programlisting>
- </section>
-
- <section id="jbosscache-jgroups-other-gc">
- <title>Distributed Garbage Collection</title>
- <para>
- In a JGroups cluster, all nodes have to store all messages received for potential retransmission in case of a failure. However, if we store all messages forever, we will run out of memory. So, the distributed garbage collection service in JGroups periodically purges messages that have seen by all nodes from the memory in each node. The distributed garbage collection service is configured in the <literal>pbcast.STABLE</literal> sub-element under the JGroups <literal>Config</literal> element. Here is an example configuration.
- </para>
-
-<programlisting>
-<pbcast.STABLE stability_delay="1000"
- desired_avg_gossip="5000"
- down_thread="false" up_thread="false"
- max_bytes="400000"/>
-</programlisting>
-
-<para>The configurable attributes in the <literal>pbcast.STABLE</literal> element are as follows.</para>
- <itemizedlist>
- <listitem>
- <para><emphasis role="bold">desired_avg_gossip</emphasis> specifies intervals (in
- milliseconds) of garbage collection runs. Value <literal>0</literal> disables this
- service.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">max_bytes</emphasis> specifies the maximum number of bytes
- received before the cluster triggers a garbage collection run. Value
- <literal>0</literal> disables this service.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">stability_delay</emphasis> specifies delay before we send STABILITY msg (give others a change to send first). If used together with max_bytes, this attribute should be set to a small number.</para>
- </listitem>
- </itemizedlist>
- <note>
- <para>Set the <literal>max_bytes</literal> attribute when you have a high traffic
- cluster.</para>
- </note>
- </section>
- <section id="jbosscache-jgroups-other-merge">
- <title>Merging</title>
- <para>
- When a network error occurs, the cluster might be partitioned into several different partitions. JGroups has a MERGE service that allows the coordinators in partitions to communicate with each other and form a single cluster back again. The flow control service is configured in the <literal>MERGE2</literal> sub-element under the JGroups <literal>Config</literal> element. Here is an example configuration.
- </para>
-
-<programlisting>
-<MERGE2 max_interval="10000"
- min_interval="2000"
- down_thread="false" up_thread="false"/>
-</programlisting>
-
-
- <para>The configurable attributes in the <literal>FC</literal> element are as follows.</para>
- <itemizedlist>
- <listitem>
- <para><emphasis role="bold">max_interval</emphasis> specifies the maximum number of
- milliseconds to send out a MERGE message.</para>
- </listitem>
- <listitem>
- <para><emphasis role="bold">min_interval</emphasis> specifies the minimum number of
- milliseconds to send out a MERGE message.</para>
- </listitem>
- </itemizedlist>
- <para>JGroups chooses a random value between <literal>min_interval</literal> and
- <literal>max_interval</literal> to send out the MERGE message.</para>
- <note>
- <para>
- The cluster states are not merged in a merger. This has to be done by the application. If <literal>MERGE2</literal> is used in conjunction with TCPPING, the <literal>initial_hosts</literal> attribute must contain all the nodes that could potentially be merged back, in order for the merge process to work properly. Otherwise, the merge process would not merge all the nodes even though shunning is disabled. Alternatively use MPING, which is commonly used with TCP to provide multicast member discovery capabilities, instead of TCPPING to avoid having to specify all the nodes.
- </para>
- </note>
- </section>
-
- <section><title>Binding JGroups Channels to a particular interface</title>
- <para>
- In the Transport Protocols section above, we briefly touched on how the interface to which JGroups will bind sockets is configured. Let's get into this topic in more depth:
- </para>
- <para>
- First, it's important to understand that the value set in any bind_addr element in an XML configuration file will be ignored by JGroups if it finds that system property jgroups.bind_addr (or a deprecated earlier name for the same thing, <literal>bind.address</literal>) has been set. The system property trumps XML. If JBoss AS is started with the -b (a.k.a. --host) switch, the AS will set <literal>jgroups.bind_addr</literal> to the specified value.
- </para>
- <para>
- Beginning with AS 4.2.0, for security reasons the AS will bind most services to localhost if -b is not set. The effect of this is that in most cases users are going to be setting -b and thus jgroups.bind_addr is going to be set and any XML setting will be ignored.
- </para>
- <para>
- So, what are <emphasis>best practices</emphasis> for managing how JGroups binds to interfaces?
- </para>
- <itemizedlist>
- <listitem>
- <para>
- Binding JGroups to the same interface as other services. Simple, just use -b:
- <screen>./run.sh -b 192.168.1.100 -c all</screen>
- </para>
-</listitem>
-<listitem>
- <para>
- Binding services (e.g., JBoss Web) to one interface, but use a different one for JGroups:
- <screen>./run.sh -b 10.0.0.100 -Djgroups.bind_addr=192.168.1.100 -c all</screen>
-
- Specifically setting the system property overrides the -b value. This is a common usage pattern; put client traffic on one network, with intra-cluster traffic on another.
- </para>
- </listitem>
- <listitem>
- <para>
-
- Binding services (e.g., JBoss Web) to all interfaces. This can be done like this:
- <screen>./run.sh -b 0.0.0.0 -c all</screen>
- However, doing this will not cause JGroups to bind to all interfaces! Instead , JGroups will bind to the machine's default interface. See the Transport Protocols section for how to tell JGroups to receive or send on all interfaces, if that is what you really want.
- </para>
- </listitem>
- <listitem>
- <para>
- Binding services (e.g., JBoss Web) to all interfaces, but specify the JGroups interface:
- <screen>./run.sh -b 0.0.0.0 -Djgroups.bind_addr=192.168.1.100 -c all</screen>
-
- Again, specifically setting the system property overrides the -b value.
- </para>
- </listitem>
- <listitem>
- <para>
- Using different interfaces for different channels:
- <screen>./run.sh -b 10.0.0.100 -Djgroups.ignore.bind_addr=true -c all</screen>
- </para>
- </listitem>
-</itemizedlist>
-
-<para>
-This setting tells JGroups to ignore the <literal>jgroups.bind_addr</literal> system property, and instead use whatever is specfied in XML. You would need to edit the various XML configuration files to set the <literal>bind_addr</literal> to the desired interfaces.
- </para>
- </section>
-
- <section><title>Isolating JGroups Channels</title>
- <para>
- Within JBoss AS, there are a number of services that independently create JGroups channels -- 3 different JBoss Cache services (used for HttpSession replication, EJB3 SFSB replication and EJB3 entity replication) along with the general purpose clustering service called HAPartition that underlies most other JBossHA services.
- </para>
- <para>
- It is critical that these channels only communicate with their intended peers; not with the channels used by other services and not with channels for the same service opened on machines not meant to be part of the group. Nodes improperly communicating with each other is one of the most common issues users have with JBoss AS clustering.
- </para>
- <para>
- Whom a JGroups channel will communicate with is defined by its group name, multicast address, and multicast port, so isolating JGroups channels comes down to ensuring different channels use different values for the group name, multicast address and multicast port.
- </para>
- <para>
- To isolate JGroups channels for different services on the same set of AS instances from each other, you MUST change the group name and the multicast port. In other words, each channel must have its own set of values.
- </para>
- <para>
- For example, say we have a production cluster of 3 machines, each of which has an HAPartition deployed along with a JBoss Cache used for web session clustering. The HAPartition channels should not communicate with the JBoss Cache channels. They should use a different group name and multicast port. They can use the same multicast address, although they don't need to.
- </para>
- <para>
- To isolate JGroups channels for the same service from other instances of the service on the network, you MUST change ALL three values. Each channel must have its own group name, multicast address, and multicast port.
- </para>
- <para>
- For example, say we have a production cluster of 3 machines, each of which has an HAPartition deployed. On the same network there is also a QA cluster of 3 machines, which also has an HAPartition deployed. The HAPartition group name, multicast address, and multicast port for the production machines must be different from those used on the QA machines.
- </para>
- </section>
-
-
- <section><title>Changing the Group Name</title>
- <para>
- The group name for a JGroups channel is configured via the service that starts the channel. Unfortunately, different services use different attribute names for configuring this. For HAPartition and related services configured in the deploy/cluster-service.xml file, this is configured via a PartitionName attribute. For JBoss Cache services, the name of the attribute is ClusterName.
- </para>
- <para>
- Starting with JBoss AS 4.0.4, for the HAPartition and all the standard JBoss Cache services, we make it easy for you to create unique groups names simply by using the -g (a.k.a. –partition) switch when starting JBoss:
- <screen>./run.sh -g QAPartition -b 192.168.1.100 -c all</screen>
- This switch sets the jboss.partition.name system property, which is used as a component in the configuration of the group name in all the standard clustering configuration files. For example,
-<screen><![CDATA[<attribute name="ClusterName">Tomcat-${jboss.partition.name:Cluster}</attribute>]]></screen>
- </para>
- </section>
-
-
-
-<section><title>Changing the multicast address and port</title>
- <para>
- The -u (a.k.a. --udp) command line switch may be used to control the multicast address used by the JGroups channels opened by all standard AS services.
-<screen><![CDATA[/run.sh -u 230.1.2.3 -g QAPartition -b 192.168.1.100 -c all]]></screen>
- This switch sets the jboss.partition.udpGroup system property, which you can see referenced in all of the standard protocol stack configs in JBoss AS:
- </para>
-
-<programlisting><![CDATA[<Config>
-<UDP mcast_addr="${jboss.partition.udpGroup:228.1.2.3}"
- ....]]>
-</programlisting>
-<para>
- Unfortunately, setting the multicast ports is not so simple. As described above, by default there are four separate JGroups channels in the standard JBoss AS all configuration, and each should be given a unique port. There are no command line switches to set these, but the standard configuration files do use system properties to set them. So, they can be configured from the command line by using -D. For example,
- </para>
-<programlisting>
- /run.sh -u 230.1.2.3 -g QAPartition -Djboss.hapartition.mcast_port=12345 -Djboss.webpartition.mcast_port=23456 -Djboss.ejb3entitypartition.mcast_port=34567 -Djboss.ejb3sfsbpartition.mcast_port=45678 -b 192.168.1.100 -c all
-</programlisting>
-
-<para><emphasis>Why isn't it sufficient to change the group name?</emphasis></para>
-<para>
- If channels with different group names share the same multicast address and port, the lower level JGroups protocols in each channel will see, process and eventually discard messages intended for the other group. This will at a minimum hurt performance and can lead to anomalous behavior.
- </para>
-
- <para><emphasis>Why do I need to change the multicast port if I change the address?</emphasis></para>
- <para>
- It should be sufficient to just change the address, but there is a problem on several operating systems whereby packets addressed to a particular multicast port are delivered to all listeners on that port, regardless of the multicast address they are listening on. So the recommendation is to change both the address and the port.
- </para>
-</section>
-
-
-<section><title>JGroups Troubleshooting</title>
- <para><emphasis>Nodes do not form a cluster</emphasis></para>
-
- <para>
- Make sure your machine is set up correctly for IP multicast. There are 2 test programs that can be used to detect this: McastReceiverTest and McastSenderTest. Go to the <literal>$JBOSS_HOME/server/all/lib</literal> directory and start McastReceiverTest, for example:
-<screen>java -cp jgroups.jar org.jgroups.tests.McastReceiverTest -mcast_addr 224.10.10.10 -port 5555 </screen>
-</para>
-
-<para>
-Then in another window start <literal>McastSenderTest</literal>:
-<screen>java -cp jgroups.jar org.jgroups.tests.McastSenderTest -mcast_addr 224.10.10.10 -port 5555</screen>
-</para>
-
-<para>
- If you want to bind to a specific network interface card (NIC), use <literal>-bind_addr 192.168.0.2</literal>, where 192.168.0.2 is the IP address of the NIC to which you want to bind. Use this parameter in both the sender and the receiver.
-</para>
-<para>
- You should be able to type in the <literal>McastSenderTest</literal> window and see the output in the <literal>McastReceiverTest</literal> window. If not, try to use -ttl 32 in the sender. If this still fails, consult a system administrator to help you setup IP multicast correctly, and ask the admin to make sure that multicast will work on the interface you have chosen or, if the machines have multiple interfaces, ask to be told the correct interface.
-Once you know multicast is working properly on each machine in your cluster, you can repeat the above test to test the network, putting the sender on one machine and the receiver on another.
-
- </para>
-</section>
-
-<section><title>Causes of missing heartbeats in FD</title>
- <para>
- Sometimes a member is suspected by FD because a heartbeat ack has not been received for some time T (defined by timeout and max_tries). This can have multiple reasons, e.g. in a cluster of A,B,C,D; C can be suspected if (note that A pings B, B pings C, C pings D and D pings A):
- </para>
-
- <itemizedlist>
- <listitem>
- <para>
- B or C are running at 100% CPU for more than T seconds. So even if C sends a heartbeat ack to B, B may not be able to process it because it is at 100%
- </para>
- </listitem>
- <listitem>
- <para>
- B or C are garbage collecting, same as above.
- </para>
- </listitem>
- <listitem>
- <para>
- A combination of the 2 cases above
- </para>
- </listitem>
- <listitem>
- <para>
- The network loses packets. This usually happens when there is a lot of traffic on the network, and the switch starts dropping packets (usually broadcasts first, then IP multicasts, TCP packets last).
- </para>
- </listitem>
- <listitem>
- <para>
- B or C are processing a callback. Let's say C received a remote method call over its channel and takes T+1 seconds to process it. During this time, C will not process any other messages, including heartbeats, and therefore B will not receive the heartbeat ack and will suspect C.
- </para>
- </listitem>
-</itemizedlist>
-
-</section>
-</section>
-
- </chapter>
Copied: projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_JGroups.xml (from rev 82580, projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_JBoss_Cache_JGroups.xml)
===================================================================
--- projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_JGroups.xml (rev 0)
+++ projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_JGroups.xml 2009-01-01 20:55:34 UTC (rev 82582)
@@ -0,0 +1,1231 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.3//EN" "http://www.oasis-open.org/docbook/xml/4.3/docbookx.dtd">
+
+<chapter id="jgroups.chapt">
+ <title>JBossCache and JGroups Services</title>
+ <para>JGroups and JBossCache provide the underlying communication, node replication and caching services, for
+ JBoss AS clusters. Those services are configured as MBeans. There is a set of JBossCache and JGroups MBeans
+ for each type of clustering applications (e.g., the Stateful Session EJBs, HTTP session replication etc.).
+ </para>
+ <para>The JBoss AS ships with a reasonable set of default JGroups and JBossCache MBean configurations. Most
+ applications just work out of the box with the default MBean configurations. You only need to tweak them
+ when you are deploying an application that has special network or performance requirements.</para>
+
+ <section id="jgroups-configuration">
+ <title>JGroups Configuration</title>
+ <para>The JGroups framework provides services to enable peer-to-peer communications between nodes in a
+ cluster. It is built on top a stack of network communication protocols that provide transport,
+ discovery, reliability and failure detection, and cluster membership management services. <xref linkend="JGroupsStack.fig"/> shows the protocol stack in JGroups.</para>
+ <figure id="JGroupsStack.fig">
+ <title>Protocol stack in JGroups</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata align="center" fileref="images/JGroupsStack.png"/>
+ </imageobject>
+ </mediaobject>
+ </figure>
+ <para>JGroups configurations often appear as a nested attribute in cluster related MBean services, such as
+ the <literal>PartitionConfig</literal> attribute in the <literal>ClusterPartition</literal> MBean or the
+ <literal>ClusterConfig</literal> attribute in the <literal>TreeCache</literal> MBean. You can
+ configure the behavior and properties of each protocol in JGroups via those MBean attributes. Below is
+ an example JGroups configuration in the <literal>ClusterPartition</literal> MBean.</para>
+<programlisting><![CDATA[
+<mbean code="org.jboss.ha.framework.server.ClusterPartition"
+ name="jboss:service=${jboss.partition.name:DefaultPartition}">
+
+ ... ...
+
+ <attribute name="PartitionConfig">
+ <Config>
+
+ <UDP mcast_addr="${jboss.partition.udpGroup:228.1.2.3}"
+ mcast_port="${jboss.hapartition.mcast_port:45566}"
+ tos="8"
+ ucast_recv_buf_size="20000000"
+ ucast_send_buf_size="640000"
+ mcast_recv_buf_size="25000000"
+ mcast_send_buf_size="640000"
+ loopback="false"
+ discard_incompatible_packets="true"
+ enable_bundling="false"
+ max_bundle_size="64000"
+ max_bundle_timeout="30"
+ use_incoming_packet_handler="true"
+ use_outgoing_packet_handler="false"
+ ip_ttl="${jgroups.udp.ip_ttl:2}"
+ down_thread="false" up_thread="false"/>
+
+ <PING timeout="2000"
+ down_thread="false" up_thread="false" num_initial_members="3"/>
+
+ <MERGE2 max_interval="100000"
+ down_thread="false" up_thread="false" min_interval="20000"/>
+ <FD_SOCK down_thread="false" up_thread="false"/>
+
+ <FD timeout="10000" max_tries="5"
+ down_thread="false" up_thread="false" shun="true"/>
+ <VERIFY_SUSPECT timeout="1500" down_thread="false" up_thread="false"/>
+ <pbcast.NAKACK max_xmit_size="60000"
+ use_mcast_xmit="false" gc_lag="0"
+ retransmit_timeout="300,600,1200,2400,4800"
+ down_thread="false" up_thread="false"
+ discard_delivered_msgs="true"/>
+ <UNICAST timeout="300,600,1200,2400,3600"
+ down_thread="false" up_thread="false"/>
+ <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
+ down_thread="false" up_thread="false"
+ max_bytes="400000"/>
+ <pbcast.GMS print_local_addr="true" join_timeout="3000"
+ down_thread="false" up_thread="false"
+ join_retry_timeout="2000" shun="true"
+ view_bundling="true"/>
+ <FRAG2 frag_size="60000" down_thread="false" up_thread="false"/>
+ <pbcast.STATE_TRANSFER down_thread="false"
+ up_thread="false" use_flush="false"/>
+ </Config>
+ </attribute>
+</mbean> ]]>
+</programlisting>
+<para>
+ All the JGroups configuration data is contained in the <Config> element under the JGroups config MBean attribute. This information is used to configure a JGroups Channel; the Channel is conceptually similar to a socket, and manages communication between peers in a cluster. Each element inside the <Config> element defines a particular JGroups Protocol; each Protocol performs one function, and the combination of those functions is what defines the characteristics of the overall Channel. In the next several sections, we will dig into the commonly used protocols and their options and explain exactly what they mean.
+</para>
+</section>
+
+<section><title>Common Configuration Properties</title>
+ <para>The following common properties are exposed by all of the JGroups protocols discussed below:
+</para>
+<itemizedlist>
+ <listitem>
+ <para><literal>down_thread</literal> whether the protocol should create an internal queue and a queue processing thread (aka the down_thread) for messages passed down from higher layers. The higher layer could be another protocol higher in the stack, or the application itself, if the protocol is the top one on the stack. If true (the default), when a message is passed down from a higher layer, the calling thread places the message in the protocol's queue, and then returns immediately. The protocol's down_thread is responsible for reading messages off the queue, doing whatever protocol-specific processing is required, and passing the message on to the next protocol in the stack.
+ </para>
+ </listitem>
+ <listitem>
+ <para><literal>up_thread</literal> is conceptually similar to down_thread, but here the queue and thread are for messages received from lower layers in the protocol stack.
+ </para>
+ </listitem>
+</itemizedlist>
+<para>Generally speaking, <literal>up_thread</literal> and <literal>down_thread</literal> should be set to false.
+</para>
+
+</section>
+
+<section id="jgroups-transport">
+ <title>Transport Protocols</title>
+ <para>The transport protocols send messages from one cluster node to another (unicast) or from cluster
+ node to all other nodes in the cluster (mcast). JGroups supports UDP, TCP, and TUNNEL as transport
+ protocols.</para>
+ <note>
+ <para>The <literal>UDP</literal>, <literal>TCP</literal>, and <literal>TUNNEL</literal> elements are
+ mutually exclusive. You can only have one transport protocol in each JGroups
+ <literal>Config</literal> element</para>
+ </note>
+ <section id="jgroups-transport-udp">
+ <title>UDP configuration</title>
+ <para>UDP is the preferred protocol for JGroups. UDP uses multicast or multiple unicasts to send and
+ receive messages. If you choose UDP as the transport protocol for your cluster service, you need
+ to configure it in the <literal>UDP</literal> sub-element in the JGroups
+ <literal>Config</literal> element. Here is an example.</para>
+<programlisting><![CDATA[
+<UDP mcast_addr="${jboss.partition.udpGroup:228.1.2.3}"
+ mcast_port="${jboss.hapartition.mcast_port:45566}"
+ tos="8"
+ ucast_recv_buf_size="20000000"
+ ucast_send_buf_size="640000"
+ mcast_recv_buf_size="25000000"
+ mcast_send_buf_size="640000"
+ loopback="false"
+ discard_incompatible_packets="true"
+ enable_bundling="false"
+ max_bundle_size="64000"
+ max_bundle_timeout="30"
+ use_incoming_packet_handler="true"
+ use_outgoing_packet_handler="false"
+ ip_ttl="${jgroups.udp.ip_ttl:2}"
+ down_thread="false" up_thread="false"/>]]>
+</programlisting>
+
+
+ <para>The available attributes in the above JGroups configuration are listed below.</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">ip_mcast</emphasis> specifies whether or not to use IP
+ multicasting. The default is <literal>true</literal>. If set to false, it will send n unicast packets rather than 1 multicast packet. Either way, packets are UDP datagrams.
+ </para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">mcast_addr</emphasis> specifies the multicast address (class D) for joining a group (i.e., the cluster). If omitted, the default is <literal>228.8.8.8
+ </literal>.
+ </para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">mcast_port</emphasis> specifies the multicast port number. If omitted, the
+ default is <literal>45566</literal>.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">bind_addr</emphasis> specifies the interface on which to receive and send multicasts (uses the <literal>-Djgroups.bind_address</literal> system property, if present). If you have a multihomed machine, set the <literal>bind_addr</literal> attribute or system property to the appropriate NIC IP address. By default, system property setting takes priority over XML attribute unless -Djgroups.ignore.bind_addr system property is set.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">receive_on_all_interfaces </emphasis> specifies whether this node
+ should listen on all interfaces for multicasts. The default is <literal>false</literal>.
+ It overrides the <literal>bind_addr</literal> property for receiving multicasts.
+ However, <literal>bind_addr</literal> (if set) is still used to send multicasts.</para>
+ </listitem>
+ <listitem><para><emphasis role="bold">send_on_all_interfaces</emphasis> specifies whether this node send UDP packets via all the NICs if you have a multi NIC machine. This means that the same multicast message is sent N times, so use with care.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold">receive_interfaces</emphasis> specifies a list of of interfaces to receive multicasts on. The multicast receive socket will listen on all of these interfaces. This is a comma-separated list of IP addresses or interface names. E.g. "<literal>192.168.5.1,eth1,127.0.0.1</literal>".
+ </para>
+ </listitem>
+
+
+ <listitem>
+ <para><emphasis role="bold">ip_ttl</emphasis> specifies time-to-live for IP Multicast packets. TTL is the commonly used term in multicast networking, but is actually something of a misnomer, since the value here refers to how many network hops a packet will be allowed to travel before networking equipment will drop it.
+ </para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">use_incoming_packet_handler</emphasis> specifies whether to use a separate thread to process incoming messages. Sometimes receivers are overloaded (they have to handle de-serialization etc). Packet handler is a separate thread taking care of de-serialization, receiver thread(s) simply put packet in queue and return immediately. Setting this to true adds one more thread. The default is <literal>true</literal>.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">use_outgoing_packet_handler</emphasis> specifies whether to use a separate thread to process outgoing messages. The default is false.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">enable_bundling</emphasis> specifies whether to enable message bundling.
+ If it is <literal>true</literal>, the node would queue outgoing messages until
+ <literal>max_bundle_size</literal> bytes have accumulated, or
+ <literal>max_bundle_time</literal> milliseconds have elapsed, whichever occurs
+ first. Then bundle queued messages into a large message and send it. The messages are
+ unbundled at the receiver. The default is <literal>false</literal>.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">loopback</emphasis> specifies whether to loop outgoing message
+ back up the stack. In <literal>unicast</literal> mode, the messages are sent to self. In
+ <literal>mcast</literal> mode, a copy of the mcast message is sent. The default is <literal>false</literal></para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">discard_incompatibe_packets</emphasis> specifies whether to
+ discard packets from different JGroups versions. Each message in the cluster is tagged
+ with a JGroups version. When a message from a different version of JGroups is received,
+ it will be discarded if set to true, otherwise a warning will be logged. The default is <literal>false</literal></para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">mcast_send_buf_size, mcast_recv_buf_size, ucast_send_buf_size,
+ ucast_recv_buf_size</emphasis> define receive and send buffer sizes. It is good to
+ have a large receiver buffer size, so packets are less likely to get dropped due to
+ buffer overflow.</para>
+ </listitem>
+ <listitem>
+ <para><literal>tos</literal> specifies traffic class for sending unicast and multicast datagrams.
+ </para>
+ </listitem>
+
+ </itemizedlist>
+ <note>
+ <para>On Windows 2000 machines, because of the media sense feature being broken with multicast
+ (even after disabling media sense), you need to set the UDP protocol's
+ <literal>loopback</literal> attribute to <literal>true</literal>.</para>
+ </note>
+ </section>
+
+
+ <section id="jgroups-transport-tcp">
+ <title>TCP configuration</title>
+ <para>Alternatively, a JGroups-based cluster can also work over TCP connections. Compared with UDP,
+ TCP generates more network traffic when the cluster size increases. TCP
+ is fundamentally a unicast protocol. To send multicast messages, JGroups uses multiple TCP
+ unicasts. To use TCP as a transport protocol, you should define a <literal>TCP</literal> element
+ in the JGroups <literal>Config</literal> element. Here is an example of the
+ <literal>TCP</literal> element.</para>
+ <programlisting>
+<TCP start_port="7800"
+ bind_addr="192.168.5.1"
+ loopback="true"
+ down_thread="false" up_thread="false"/>
+ </programlisting>
+ <para>Below are the attributes available in the <literal>TCP</literal> element.</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">bind_addr</emphasis> specifies the binding address. It can also
+ be set with the <literal>-Djgroups.bind_address</literal> command line option at server
+ startup.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">start_port, end_port</emphasis> define the range of TCP ports
+ the server should bind to. The server socket is bound to the first available port from
+ <literal>start_port</literal>. If no available port is found (e.g., because of a
+ firewall) before the <literal>end_port</literal>, the server throws an exception. If no <literal>end_port</literal> is provided or <literal>end_port < start_port</literal> then there is no upper limit on the port range. If <literal>start_port == end_port</literal>, then we force JGroups to use the given port (start fails if port is not available). The default is 7800. If set to 0, then the operating system will pick a port. Please, bear in mind that setting it to 0 will work only if we use MPING or TCPGOSSIP as discovery protocol because <literal>TCCPING</literal> requires listing the nodes and their corresponding ports.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">loopback</emphasis> specifies whether to loop outgoing message
+ back up the stack. In <literal>unicast</literal> mode, the messages are sent to self. In
+ <literal>mcast</literal> mode, a copy of the mcast message is sent. The default is false.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">recv_buf_size, send_buf_size</emphasis> define receive and send buffer sizes. It is good to have a large receiver buffer size, so packets are less likely to get dropped due to buffer overflow.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">conn_expire_time</emphasis> specifies the time (in milliseconds)
+ after which a connection can be closed by the reaper if no traffic has been
+ received.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">reaper_interval</emphasis> specifies interval (in milliseconds)
+ to run the reaper. If both values are 0, no reaping will be done. If either value is
+ > 0, reaping will be enabled. By default, reaper_interval is 0, which means no reaper.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">sock_conn_timeout</emphasis> specifies max time in millis for a socket creation. When doing the initial discovery, and a peer hangs, don't wait forever but go on after the timeout to ping other members. Reduces chances of *not* finding any members at all. The default is 2000.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">use_send_queues</emphasis> specifies whether to use separate send queues for each connection. This prevents blocking on write if the peer hangs. The default is true.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">external_addr</emphasis> specifies external IP address to broadcast to other group members (if different to local address). This is useful when you have use (Network Address Translation) NAT, e.g. a node on a private network, behind a firewall, but you can only route to it via an externally visible address, which is different from the local address it is bound to. Therefore, the node can be configured to broadcast its external address, while still able to bind to the local one. This avoids having to use the TUNNEL protocol, (and hence a requirement for a central gossip router) because nodes outside the firewall can still route to the node inside the firewall, but only on its external address. Without setting the external_addr, the node behind the firewall will broadcast its private address to the other nodes which will not be able to route to it.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">skip_suspected_members</emphasis> specifies whether unicast messages should not be sent to suspected members. The default is true.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">tcp_nodelay</emphasis> specifies TCP_NODELAY. TCP by default nagles messages, that is, conceptually, smaller messages are bundled into larger ones. If we want to invoke synchronous cluster method calls, then we need to disable nagling in addition to disabling message bundling (by setting <literal>enable_bundling</literal> to false). Nagling is disabled by setting <literal>tcp_nodelay</literal> to true. The default is false.
+ </para>
+ </listitem>
+
+ </itemizedlist>
+ </section>
+
+ <section id="jgroups-transport-tunnel">
+ <title>TUNNEL configuration</title>
+ <para>The TUNNEL protocol uses an external router to send messages. The external router is known as
+ a <literal>GossipRouter</literal>. Each node has to register with the router. All messages are sent to the router and forwarded on to their destinations. The TUNNEL approach can be used to setup communication with nodes behind firewalls. A node can establish a TCP connection to the GossipRouter through the firewall (you can use port 80). The same connection is used by the router to send messages to nodes behind the firewall as most firewalls do not permit outside hosts to initiate a TCP connection to a host inside the firewall. The TUNNEL configuration is defined in the TUNNEL element in the JGroups Config element. Here is an example..
+ </para>
+
+
+ <programlisting>
+<TUNNEL router_port="12001"
+ router_host="192.168.5.1"
+ down_thread="false" up_thread="false/>
+ </programlisting>
+
+
+<para>The available attributes in the <literal>TUNNEL</literal> element are listed below.</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">router_host</emphasis> specifies the host on which the
+ GossipRouter is running.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">router_port</emphasis> specifies the port on which the
+ GossipRouter is listening.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">loopback</emphasis> specifies whether to loop messages back up
+ the stack. The default is <literal>true</literal>.</para>
+ </listitem>
+ </itemizedlist>
+ </section>
+ </section>
+
+
+
+
+ <section id="jgroups-discovery">
+ <title>Discovery Protocols</title>
+ <para>
+ The cluster needs to maintain a list of current member nodes at all times so that the load balancer and client interceptor know how to route their requests. Discovery protocols are used to discover active nodes in the cluster and detect the oldest member of the cluster, which is the coordinator. All initial nodes are discovered when the cluster starts up.
+ When a new node joins the cluster later, it is only discovered after the group membership protocol
+ (GMS, see <xref linkend="jgroups-other-gms"/>) admits it into the group.</para>
+ <para>Since the discovery protocols sit on top of the transport protocol, you can choose to use different discovery protocols based on your transport protocol. These are also configured as sub-elements in the JGroups MBean <literal>Config</literal> element.</para>
+
+
+
+
+ <section id="jgroups-discovery-ping">
+ <title>PING</title>
+ <para>
+ PING is a discovery protocol that works by either multicasting PING requests to an IP multicast address or connecting to a gossip router. As such, PING normally sits on top of the UDP or TUNNEL transport protocols. Each node responds with a packet {C, A}, where C=coordinator's address and A=own address. After timeout milliseconds or num_initial_members replies, the joiner determines the coordinator from the responses, and sends a JOIN request to it (handled by). If nobody responds, we assume we are the first member of a group.
+ </para>
+ <para>Here is an example PING configuration for IP multicast.
+ </para>
+
+
+ <programlisting>
+<PING timeout="2000"
+ num_initial_members="2"
+ down_thread="false" up_thread="false"/>
+ </programlisting>
+<para>
+ Here is another example PING configuration for contacting a Gossip Router.
+<programlisting><![CDATA[
+<PING gossip_host="localhost"
+ gossip_port="1234"
+ timeout="3000"
+ num_initial_members="3"
+ down_thread="false" up_thread="false"/>]]>
+</programlisting>
+
+ </para>
+
+
+ <para>The available attributes in the <literal>PING</literal> element are listed below.</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">timeout</emphasis> specifies the maximum number of milliseconds
+ to wait for any responses. The default is 3000.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">num_initial_members</emphasis> specifies the maximum number of
+ responses to wait for unless timeout has expired. The default is 2.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">gossip_host</emphasis> specifies the host on which the
+ GossipRouter is running.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">gossip_port</emphasis> specifies the port on which the
+ GossipRouter is listening on.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">gossip_refresh</emphasis> specifies the interval (in
+ milliseconds) for the lease from the GossipRouter. The default is 20000.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">initial_hosts</emphasis> is a comma-seperated list of addresses
+ (e.g., <literal>host1[12345],host2[23456]</literal>), which are pinged for
+ discovery.</para>
+ </listitem>
+ </itemizedlist>
+ <para>If both <literal>gossip_host</literal> and <literal>gossip_port</literal> are defined, the
+ cluster uses the GossipRouter for the initial discovery. If the <literal>initial_hosts</literal>
+ is specified, the cluster pings that static list of addresses for discovery. Otherwise, the
+ cluster uses IP multicasting for discovery.</para>
+ <note>
+ <para>The discovery phase returns when the <literal>timeout</literal> ms have elapsed or the
+ <literal>num_initial_members</literal> responses have been received.</para>
+ </note>
+ </section>
+
+
+
+ <section id="jgroups-discovery-tcpgossip">
+ <title>TCPGOSSIP</title>
+ <para>The TCPGOSSIP protocol only works with a GossipRouter. It works essentially the same way as
+ the PING protocol configuration with valid <literal>gossip_host</literal> and
+ <literal>gossip_port</literal> attributes. It works on top of both UDP and TCP transport protocols. Here is an example.</para>
+<programlisting><![CDATA[
+<TCPGOSSIP timeout="2000"
+ initial_hosts="192.168.5.1[12000],192.168.0.2[12000]"
+ num_initial_members="3"
+ down_thread="false" up_thread="false"/>]]>
+</programlisting>
+
+
+ <para>The available attributes in the <literal>TCPGOSSIP</literal> element are listed below.</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">timeout</emphasis> specifies the maximum number of milliseconds
+ to wait for any responses. The default is 3000.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">num_initial_members</emphasis> specifies the maximum number of
+ responses to wait for unless timeout has expired. The default is 2.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">initial_hosts</emphasis> is a comma-seperated list of addresses
+ (e.g., <literal>host1[12345],host2[23456]</literal>) for GossipRouters to register
+ with.</para>
+ </listitem>
+ </itemizedlist>
+ </section>
+
+
+
+ <section id="jgroups-discovery-tcpping">
+ <title>TCPPING</title>
+ <para>The TCPPING protocol takes a set of known members and ping them for discovery. This is
+ essentially a static configuration. It works on top of TCP. Here is an example of the
+ <literal>TCPPING</literal> configuration element in the JGroups <literal>Config</literal>
+ element.</para>
+ <programlisting>
+<TCPPING timeout="2000"
+ initial_hosts="hosta[2300],hostb[3400],hostc[4500]"
+ port_range="3"
+ num_initial_members="3"
+ down_thread="false" up_thread="false"/>
+</programlisting>
+
+
+ <para>The available attributes in the <literal>TCPPING</literal> element are listed below.</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">timeout</emphasis> specifies the maximum number of milliseconds
+ to wait for any responses. The default is 3000.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">num_initial_members</emphasis> specifies the maximum number of
+ responses to wait for unless timeout has expired. The default is 2.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">initial_hosts</emphasis> is a comma-seperated list of addresses
+ (e.g., <literal>host1[12345],host2[23456]</literal>) for pinging.</para>
+ </listitem>
+ <listitem>
+ <para>
+ <emphasis role="bold">port_range</emphasis> specifies the number of consecutive ports to be probed when getting the initial membership, starting with the port specified in the initial_hosts parameter. Given the current values of port_range and initial_hosts above, the TCPPING layer will try to connect to hosta:2300, hosta:2301, hosta:2302, hostb:3400, hostb:3401, hostb:3402, hostc:4500, hostc:4501, hostc:4502. The configuration options allows for multiple nodes on the same host to be pinged.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </section>
+
+
+
+
+ <section id="jgroups-discovery-mping">
+ <title>MPING</title>
+ <para>
+ MPING uses IP multicast to discover the initial membership. It can be used with all transports, but usually this is used in combination with TCP. TCP usually requires TCPPING, which has to list all group members explicitly, but MPING doesn't have this requirement. The typical use case for this is when we want TCP as transport, but multicasting for discovery so we don't have to define a static list of initial hosts in TCPPING or require external Gossip Router.
+ </para>
+
+<programlisting>
+<MPING timeout="2000"
+ bind_to_all_interfaces="true"
+ mcast_addr="228.8.8.8"
+ mcast_port="7500"
+ ip_ttl="8"
+ num_initial_members="3"
+ down_thread="false" up_thread="false"/>
+</programlisting>
+
+
+ <para>The available attributes in the <literal>MPING</literal> element are listed below.</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">timeout</emphasis> specifies the maximum number of milliseconds
+ to wait for any responses. The default is 3000.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">num_initial_members</emphasis> specifies the maximum number of
+ responses to wait for unless timeout has expired. The default is 2..</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">bind_addr</emphasis> specifies the interface on which to send
+ and receive multicast packets.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">bind_to_all_interfaces</emphasis> overrides the
+ <literal>bind_addr</literal> and uses all interfaces in multihome nodes.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">mcast_addr, mcast_port, ip_ttl</emphasis> attributes are the
+ same as related attributes in the UDP protocol configuration.</para>
+ </listitem>
+ </itemizedlist>
+ </section>
+ </section>
+
+
+
+ <section id="jgroups-fd">
+ <title>Failure Detection Protocols</title>
+ <para>The failure detection protocols are used to detect failed nodes. Once a failed node is detected, a suspect verification phase can occur after which, if the node is still considered dead, the cluster updates its view so that the load balancer and client interceptors know to avoid the dead node. The failure detection protocols are configured as sub-elements in the JGroups MBean
+ <literal>Config</literal> element.</para>
+
+
+
+<section id="jgroups-fd-fd">
+ <title>FD</title>
+ <para>
+ FD is a failure detection protocol based on heartbeat messages. This protocol requires each node to periodically send are-you-alive messages to its neighbour. If the neighbour fails to respond, the calling node sends a SUSPECT message to the cluster. The current group coordinator can optionally double check whether the suspected node is indeed dead after which, if the node is still considered dead, updates the cluster's view. Here is an example FD configuration.
+ </para>
+
+<programlisting>
+<FD timeout="2000"
+ max_tries="3"
+ shun="true"
+ down_thread="false" up_thread="false"/>
+</programlisting>
+
+
+ <para>The available attributes in the <literal>FD</literal> element are listed below.</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">timeout</emphasis> specifies the maximum number of milliseconds
+ to wait for the responses to the are-you-alive messages. The default is 3000.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">max_tries</emphasis> specifies the number of missed
+ are-you-alive messages from a node before the node is suspected. The default is 2.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">shun</emphasis> specifies whether a failed node will be shunned.
+ Once shunned, the node will be expelled from the cluster even if it comes back later.
+ The shunned node would have to re-join the cluster through the discovery process. JGroups allows to configure itself such that shunning leads to automatic rejoins and state transfer, which is the default behaivour within JBoss Application Server.</para>
+ </listitem>
+ </itemizedlist>
+ <note>
+ <para>Regular traffic from a node counts as if it is a live. So, the are-you-alive messages are
+ only sent when there is no regular traffic to the node for sometime.</para>
+ </note>
+ </section>
+
+
+
+
+ <section id="jgroups-fd-fdsock">
+ <title>FD_SOCK</title>
+ <para>
+FD_SOCK is a failure detection protocol based on a ring of TCP sockets created between group members. Each member in a group connects to its neighbor (last member connects to first) thus forming a ring. Member B is suspected when its neighbor A detects abnormally closed TCP socket (presumably due to a node B crash). However, if a member B is about to leave gracefully, it lets its neighbor A know, so that it does not become suspected. The simplest FD_SOCK configuration does not take any attribute. You can just declare an empty <literal>FD_SOCK</literal> element in JGroups's <literal>Config</literal> element.</para>
+
+
+<programlisting>
+<FD_SOCK_down_thread="false" up_thread="false"/>
+</programlisting>
+
+<para>There available attributes in the <literal>FD_SOCK</literal> element are listed below.</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">bind_addr</emphasis> specifies the interface to which the server socket should bind to. If -Djgroups.bind_address system property is defined, XML value will be ignore. This behaivour can be reversed setting -Djgroups.ignore.bind_addr=true system property.</para>
+ </listitem>
+ </itemizedlist>
+
+ </section>
+
+
+
+ <section><title>VERIFY_SUSPECT</title>
+ <para>
+ This protocol verifies whether a suspected member is really dead by pinging that member once again. This verification is performed by the coordinator of the cluster. The suspected member is dropped from the cluster group if confirmed to be dead. The aim of this protocol is to minimize false suspicions. Here's an example.
+ </para>
+
+<programlisting><![CDATA[
+<VERIFY_SUSPECT timeout="1500"
+ down_thread="false" up_thread="false"/>]]>
+</programlisting>
+
+ <para>
+ The available attributes in the FD_SOCK element are listed below.
+ </para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ timeout specifies how long to wait for a response from the suspected member before considering it dead.
+ </para>
+</listitem>
+</itemizedlist>
+
+ </section>
+
+
+
+
+ <section><title>FD versus FD_SOCK</title>
+ <para>
+ FD and FD_SOCK, each taken individually, do not provide a solid failure detection layer. Let's look at the the differences between these failure detection protocols to understand how they complement each other:
+ </para>
+ <itemizedlist>
+ <listitem><para><emphasis>FD</emphasis></para>
+ </listitem>
+ </itemizedlist>
+ <itemizedlist>
+ <listitem>
+ <para>
+ An overloaded machine might be slow in sending are-you-alive responses.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ A member will be suspected when suspended in a debugger/profiler.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Low timeouts lead to higher probability of false suspicions and higher network traffic.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ High timeouts will not detect and remove crashed members for some time.
+ </para>
+ </listitem>
+</itemizedlist>
+
+<itemizedlist>
+<listitem><para><emphasis>FD_SOCK</emphasis>:</para>
+</listitem>
+</itemizedlist>
+
+<itemizedlist>
+ <listitem>
+ <para>
+ Suspended in a debugger is no problem because the TCP connection is still open.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ High load no problem either for the same reason.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Members will only be suspected when TCP connection breaks
+ </para>
+ </listitem>
+</itemizedlist>
+
+
+ <itemizedlist>
+ <listitem>
+ <para>
+ So hung members will not be detected.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+
+ Also, a crashed switch will not be detected until the connection runs into the TCP timeout (between 2-20 minutes, depending on TCP/IP stack implementation).
+ </para>
+ </listitem>
+</itemizedlist>
+
+<para>
+ The aim of a failure detection layer is to report real failures and therefore avoid false suspicions. There are two solutions:
+ </para>
+ <orderedlist>
+ <listitem>
+ <para>
+ By default, JGroups configures the FD_SOCK socket with KEEP_ALIVE, which means that TCP sends a heartbeat on socket on which no traffic has been received in 2 hours. If a host crashed (or an intermediate switch or router crashed) without closing the TCP connection properly, we would detect this after 2 hours (plus a few minutes). This is of course better than never closing the connection (if KEEP_ALIVE is off), but may not be of much help. So, the first solution would be to lower the timeout value for KEEP_ALIVE. This can only be done for the entire kernel in most operating systems, so if this is lowered to 15 minutes, this will affect all TCP sockets.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ The second solution is to combine FD_SOCK and FD; the timeout in FD can be set such that it is much lower than the TCP timeout, and this can be configured individually per process. FD_SOCK will already generate a suspect message if the socket was closed abnormally. However, in the case of a crashed switch or host, FD will make sure the socket is eventually closed and the suspect message generated. Example:
+ </para>
+ </listitem>
+</orderedlist>
+<programlisting><![CDATA[
+<FD_SOCK down_thread="false" up_thread="false"/>
+<FD timeout="10000" max_tries="5" shun="true"
+down_thread="false" up_thread="false" /> ]]>
+</programlisting>
+
+<para>
+ This suspects a member when the socket to the neighbor has been closed abonormally (e.g. process crash, because the OS closes all sockets). However, f a host or switch crashes, then the sockets won't be closed, therefore, as a seond line of defense, FD will suspect the neighbor after 50 seconds. Note that with this example, if you have your system stopped in a breakpoint in the debugger, the node you're debugging will be suspected after ca 50 seconds.
+ </para>
+<para>
+ A combination of FD and FD_SOCK provides a solid failure detection layer and for this reason, such technique is used accross JGroups configurations included within JBoss Application Server.
+ </para>
+ </section>
+ </section>
+
+
+
+ <section id="jgroups-reliable">
+ <title>Reliable Delivery Protocols</title>
+ <para>
+ Reliable delivery protocols within the JGroups stack ensure that data pockets are actually delivered in the right order (FIFO) to the destination node. The basis for reliable message delivery is positive and negative delivery acknowledgments (ACK and NAK). In the ACK mode, the sender resends the message until the acknowledgment is received from the receiver. In the NAK mode, the receiver requests retransmission when it discovers a gap.
+ </para>
+
+
+
+ <section id="jgroups-reliable-unicast">
+ <title>UNICAST</title>
+ <para>
+ The UNICAST protocol is used for unicast messages. It uses ACK. It is configured as a sub-element under the JGroups Config element. If the JGroups stack is configured with TCP transport protocol, UNICAST is not necessary because TCP itself guarantees FIFO delivery of unicast messages. Here is an example configuration for the <literal>UNICAST</literal> protocol.</para>
+
+<programlisting>
+<UNICAST timeout="100,200,400,800"
+down_thread="false" up_thread="false"/>
+</programlisting>
+
+<para>There is only one configurable attribute in the <literal>UNICAST</literal> element.</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">timeout</emphasis> specifies the retransmission timeout (in
+ milliseconds). For instance, if the timeout is "100,200,400,800", the sender resends the
+ message if it hasn't received an ACK after 100 ms the first time, and the second time it
+ waits for 200 ms before resending, and so on.</para>
+ </listitem>
+ </itemizedlist>
+ </section>
+
+
+ <section id="jgroups-reliable-nakack">
+ <title>NAKACK</title>
+ <para>The NAKACK protocol is used for multicast messages. It uses NAK. Under this protocol, each
+ message is tagged with a sequence number. The receiver keeps track of the sequence numbers and
+ deliver the messages in order. When a gap in the sequence number is detected, the receiver asks
+ the sender to retransmit the missing message. The NAKACK protocol is configured as the
+ <literal>pbcast.NAKACK</literal> sub-element under the JGroups <literal>Config</literal>
+ element. Here is an example configuration.</para>
+
+<programlisting>
+<pbcast.NAKACK max_xmit_size="60000" use_mcast_xmit="false"
+
+ retransmit_timeout="300,600,1200,2400,4800" gc_lag="0"
+ discard_delivered_msgs="true"
+ down_thread="false" up_thread="false"/>
+</programlisting>
+
+
+<para>The configurable attributes in the <literal>pbcast.NAKACK</literal> element are as follows.</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">retransmit_timeout</emphasis> specifies the retransmission
+ timeout (in milliseconds). It is the same as the <literal>timeout</literal> attribute in
+ the UNICAST protocol.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">use_mcast_xmit</emphasis> determines whether the sender should
+ send the retransmission to the entire cluster rather than just the node requesting it.
+ This is useful when the sender drops the pocket -- so we do not need to retransmit for
+ each node.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">max_xmit_size</emphasis> specifies maximum size for a bundled
+ retransmission, if multiple packets are reported missing.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">discard_delivered_msgs</emphasis> specifies whether to discard
+ delivery messages on the receiver nodes. By default, we save all delivered messages.
+ However, if we only ask the sender to resend their messages, we can enable this option
+ and discard delivered messages.</para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold">gc_lag specifies</emphasis> the number of messages garbage collection lags behind.
+ </para>
+ </listitem>
+
+ </itemizedlist>
+ </section>
+ </section>
+
+
+
+ <section id="jgroups-other">
+ <title>Other Configuration Options</title>
+ <para>In addition to the protocol stacks, you can also configure JGroups network services in the <literal>Config</literal> element.</para>
+
+
+ <section id="jgroups-other-gms">
+ <title>Group Membership</title>
+ <para>The group membership service in the JGroups stack maintains a list of active nodes. It handles
+ the requests to join and leave the cluster. It also handles the SUSPECT messages sent by failure
+ detection protocols. All nodes in the cluster, as well as the load balancer and client side
+ interceptors, are notified if the group membership changes. The group membership service is
+ configured in the <literal>pbcast.GMS</literal> sub-element under the JGroups
+ <literal>Config</literal> element. Here is an example configuration.</para>
+<programlisting>
+<pbcast.GMS print_local_addr="true"
+ join_timeout="3000"
+ down_thread="false" up_thread="false"
+ join_retry_timeout="2000"
+ shun="true"
+ view_bundling="true"/>
+</programlisting>
+
+
+
+<para>The configurable attributes in the <literal>pbcast.GMS</literal> element are as follows.</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">join_timeout</emphasis> specifies the maximum number of
+ milliseconds to wait for a new node JOIN request to succeed. Retry afterwards.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">join_retry_timeout</emphasis> specifies the maximum number of
+ milliseconds to wait after a failed JOIN to re-submit it.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">print_local_addr</emphasis> specifies whether to dump the node's
+ own address to the output when started.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">shun</emphasis> specifies whether a node should shun itself if
+ it receives a cluster view that it is not a member node.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">disable_initial_coord</emphasis> specifies whether to prevent
+ this node as the cluster coordinator.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">view_bundling</emphasis> specifies whether multiple JOIN or LEAVE request arriving at the same time are bundled and handled together at the same time, only sending out 1 new view / bundle. This is is more efficient than handling each request separately.
+ </para>
+ </listitem>
+
+ </itemizedlist>
+ </section>
+
+
+ <section id="jgroups-other-fc">
+ <title>Flow Control</title>
+ <para>The flow control service tries to adapt the sending data rate and the receiving data among
+ nodes. If a sender node is too fast, it might overwhelm the receiver node and result in dropped
+ packets that have to be retransmitted. In JGroups, the flow control is implemented via a
+ credit-based system. The sender and receiver nodes have the same number of credits (bytes) to
+ start with. The sender subtracts credits by the number of bytes in messages it sends. The
+ receiver accumulates credits for the bytes in the messages it receives. When the sender's credit
+ drops to a threshold, the receivers sends some credit to the sender. If the sender's credit is
+ used up, the sender blocks until it receives credits from the receiver. The flow control service
+ is configured in the <literal>FC</literal> sub-element under the JGroups
+ <literal>Config</literal> element. Here is an example configuration.</para>
+
+<programlisting>
+<FC max_credits="1000000"
+down_thread="false" up_thread="false"
+ min_threshold="0.10"/>
+</programlisting>
+
+
+<para>The configurable attributes in the <literal>FC</literal> element are as follows.</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">max_credits</emphasis> specifies the maximum number of credits
+ (in bytes). This value should be smaller than the JVM heap size.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">min_credits</emphasis> specifies the threshold credit on the
+ sender, below which the receiver should send in more credits.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">min_threshold</emphasis> specifies percentage value of the
+ threshold. It overrides the <literal>min_credits</literal> attribute.</para>
+ </listitem>
+ </itemizedlist>
+
+<note><title>Note</title>
+ <para>
+ Applications that use synchronous group RPC calls primarily do not require FC protocol in their JGroups protocol stack because synchronous communication, where the hread that makes the call blocks waiting for responses from all the members of the group, already slows overall rate of calls. Even though TCP provides flow control by itself, FC is still required in TCP based JGroups stacks because of group communication, where we essentially have to send group messages at the highest speed the slowest receiver can keep up with. TCP flow control only takes into account individual node communications and has not a notion of who's the slowest in the group, which is why FC is required.
+ </para>
+</note>
+
+<section>
+ <title>Why is FC needed on top of TCP ? TCP has its own flow control !</title>
+ <para>
+
+ The reason is group communication, where we essentially have to send group messages at the highest speed the slowest receiver can keep up with. Let's say we have a cluster {A,B,C,D}. D is slow (maybe overloaded), the rest is fast. When A sends a group message, it establishes TCP connections A-A (conceptually), A-B, A-C and A-D (if they don't yet exist). So let's say A sends 100 million messages to the cluster. Because TCP's flow control only applies to A-B, A-C and A-D, but not to A-{B,C,D}, where {B,C,D} is the group, it is possible that A, B and C receive the 100M, but D only received 1M messages. (BTW: this is also the reason why we need NAKACK, although TCP does its own retransmission).
+ </para>
+ <para>
+ Now JGroups has to buffer all messages in memory for the case when the original sender S dies and a node asks for retransmission of a message of S. Because all members buffer all messages they received, they need to purge stable messages (= messages seen by everyone) every now and then. This is done by the STABLE protocol, which can be configured to run the stability protocol round time based (e.g. every 50s) or size based (whenever 400K data has been received).
+ </para>
+ <para>
+ In the above case, the slow node D will prevent the group from purging messages above 1M, so every member will buffer 99M messages ! This in most cases leads to OOM exceptions. Note that - although the sliding window protocol in TCP will cause writes to block if the window is full - we assume in the above case that this is still much faster for A-B and A-C than for A-D.
+ </para>
+ <para>
+ So, in summary, we need to send messages at a rate the slowest receiver (D) can handle.
+ </para>
+</section>
+
+<section>
+ <title>So do I always need FC?</title>
+ <para>
+ This depends on how the application uses the JGroups channel. Referring to the example above, if there was something about the application that would naturally cause A to slow down its rate of sending because D wasn't keeping up, then FC would not be needed.
+ </para>
+ <para>
+ A good example of such an application is one that makes synchronous group RPC calls (typically using a JGroups RpcDispatcher.) By synchronous, we mean the thread that makes the call blocks waiting for responses from all the members of the group. In that kind of application, the threads on A that are making calls would block waiting for responses from D, thus naturally slowing the overall rate of calls.
+ </para>
+ <para>
+ A JBoss Cache cluster configured for REPL_SYNC is a good example of an application that makes synchronous group RPC calls. If a channel is only used for a cache configured for REPL_SYNC, we recommend you remove FC from its protocol stack.
+ </para>
+ <para>
+ And, of course, if your cluster only consists of two nodes, including FC in a TCP-based protocol stack is unnecessary. There is no group beyond the single peer-to-peer relationship, and TCP's internal flow control will handle that just fine.
+ </para>
+ <para>
+ Another case where FC may not be needed is for a channel used by a JBoss Cache configured for buddy replication and a single buddy. Such a channel will in many respects act like a two node cluster, where messages are only exchanged with one other node, the buddy. (There may be other messages related to data gravitation that go to all members, but in a properly engineered buddy replication use case these should be infrequent. But if you remove FC be sure to load test your application.)
+ </para>
+</section>
+
+
+
+ </section>
+
+
+<section><title>Fragmentation</title>
+ <para>
+ This protocol fragments messages larger than certain size. Unfragments at the receiver's side. It works for both unicast and multicast messages. It is configured in the FRAG2 sub-element under the JGroups Config element. Here is an example configuration.
+ </para>
+<programlisting><![CDATA[
+ <FRAG2 frag_size="60000" down_thread="false" up_thread="false"/>]]>
+</programlisting>
+
+<para>
+The configurable attributes in the FRAG2 element are as follows.
+</para>
+
+<itemizedlist>
+ <listitem><para><emphasis role="bold">frag_size</emphasis> specifies the max frag size in bytes. Messages larger than that are fragmented.</para></listitem>
+</itemizedlist>
+
+<note><title>Note</title>
+ <para>
+ TCP protocol already provides fragmentation but a fragmentation JGroups protocol is still needed if FC is used. The reason for this is that if you send a message larger than FC.max_bytes, FC protocol would block. So, frag_size within FRAG2 needs to be set to always be less than FC.max_bytes.
+ </para>
+</note>
+
+
+</section>
+
+ <section id="jgroups-other-st">
+ <title>State Transfer</title>
+ <para>The state transfer service transfers the state from an existing node (i.e., the cluster
+ coordinator) to a newly joining node. It is configured in the
+ <literal>pbcast.STATE_TRANSFER</literal> sub-element under the JGroups <literal>Config</literal>
+ element. It does not have any configurable attribute. Here is an example configuration.</para>
+<programlisting>
+<pbcast.STATE_TRANSFER down_thread="false" up_thread="false"/>
+</programlisting>
+ </section>
+
+ <section id="jgroups-other-gc">
+ <title>Distributed Garbage Collection</title>
+ <para>
+ In a JGroups cluster, all nodes have to store all messages received for potential retransmission in case of a failure. However, if we store all messages forever, we will run out of memory. So, the distributed garbage collection service in JGroups periodically purges messages that have seen by all nodes from the memory in each node. The distributed garbage collection service is configured in the <literal>pbcast.STABLE</literal> sub-element under the JGroups <literal>Config</literal> element. Here is an example configuration.
+ </para>
+
+<programlisting>
+<pbcast.STABLE stability_delay="1000"
+ desired_avg_gossip="5000"
+ down_thread="false" up_thread="false"
+ max_bytes="400000"/>
+</programlisting>
+
+<para>The configurable attributes in the <literal>pbcast.STABLE</literal> element are as follows.</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">desired_avg_gossip</emphasis> specifies intervals (in
+ milliseconds) of garbage collection runs. Value <literal>0</literal> disables this
+ service.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">max_bytes</emphasis> specifies the maximum number of bytes
+ received before the cluster triggers a garbage collection run. Value
+ <literal>0</literal> disables this service.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">stability_delay</emphasis> specifies delay before we send STABILITY msg (give others a change to send first). If used together with max_bytes, this attribute should be set to a small number.</para>
+ </listitem>
+ </itemizedlist>
+ <note>
+ <para>Set the <literal>max_bytes</literal> attribute when you have a high traffic
+ cluster.</para>
+ </note>
+ </section>
+ <section id="jgroups-other-merge">
+ <title>Merging</title>
+ <para>
+ When a network error occurs, the cluster might be partitioned into several different partitions. JGroups has a MERGE service that allows the coordinators in partitions to communicate with each other and form a single cluster back again. The flow control service is configured in the <literal>MERGE2</literal> sub-element under the JGroups <literal>Config</literal> element. Here is an example configuration.
+ </para>
+
+<programlisting>
+<MERGE2 max_interval="10000"
+ min_interval="2000"
+ down_thread="false" up_thread="false"/>
+</programlisting>
+
+
+ <para>The configurable attributes in the <literal>FC</literal> element are as follows.</para>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">max_interval</emphasis> specifies the maximum number of
+ milliseconds to send out a MERGE message.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">min_interval</emphasis> specifies the minimum number of
+ milliseconds to send out a MERGE message.</para>
+ </listitem>
+ </itemizedlist>
+ <para>JGroups chooses a random value between <literal>min_interval</literal> and
+ <literal>max_interval</literal> to send out the MERGE message.</para>
+ <note>
+ <para>
+ The cluster states are not merged in a merger. This has to be done by the application. If <literal>MERGE2</literal> is used in conjunction with TCPPING, the <literal>initial_hosts</literal> attribute must contain all the nodes that could potentially be merged back, in order for the merge process to work properly. Otherwise, the merge process would not merge all the nodes even though shunning is disabled. Alternatively use MPING, which is commonly used with TCP to provide multicast member discovery capabilities, instead of TCPPING to avoid having to specify all the nodes.
+ </para>
+ </note>
+ </section>
+
+ <section><title>Binding JGroups Channels to a particular interface</title>
+ <para>
+ In the Transport Protocols section above, we briefly touched on how the interface to which JGroups will bind sockets is configured. Let's get into this topic in more depth:
+ </para>
+ <para>
+ First, it's important to understand that the value set in any bind_addr element in an XML configuration file will be ignored by JGroups if it finds that system property jgroups.bind_addr (or a deprecated earlier name for the same thing, <literal>bind.address</literal>) has been set. The system property trumps XML. If JBoss AS is started with the -b (a.k.a. --host) switch, the AS will set <literal>jgroups.bind_addr</literal> to the specified value.
+ </para>
+ <para>
+ Beginning with AS 4.2.0, for security reasons the AS will bind most services to localhost if -b is not set. The effect of this is that in most cases users are going to be setting -b and thus jgroups.bind_addr is going to be set and any XML setting will be ignored.
+ </para>
+ <para>
+ So, what are <emphasis>best practices</emphasis> for managing how JGroups binds to interfaces?
+ </para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ Binding JGroups to the same interface as other services. Simple, just use -b:
+ <screen>./run.sh -b 192.168.1.100 -c all</screen>
+ </para>
+</listitem>
+<listitem>
+ <para>
+ Binding services (e.g., JBoss Web) to one interface, but use a different one for JGroups:
+ <screen>./run.sh -b 10.0.0.100 -Djgroups.bind_addr=192.168.1.100 -c all</screen>
+
+ Specifically setting the system property overrides the -b value. This is a common usage pattern; put client traffic on one network, with intra-cluster traffic on another.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+
+ Binding services (e.g., JBoss Web) to all interfaces. This can be done like this:
+ <screen>./run.sh -b 0.0.0.0 -c all</screen>
+ However, doing this will not cause JGroups to bind to all interfaces! Instead , JGroups will bind to the machine's default interface. See the Transport Protocols section for how to tell JGroups to receive or send on all interfaces, if that is what you really want.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Binding services (e.g., JBoss Web) to all interfaces, but specify the JGroups interface:
+ <screen>./run.sh -b 0.0.0.0 -Djgroups.bind_addr=192.168.1.100 -c all</screen>
+
+ Again, specifically setting the system property overrides the -b value.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Using different interfaces for different channels:
+ <screen>./run.sh -b 10.0.0.100 -Djgroups.ignore.bind_addr=true -c all</screen>
+ </para>
+ </listitem>
+</itemizedlist>
+
+<para>
+This setting tells JGroups to ignore the <literal>jgroups.bind_addr</literal> system property, and instead use whatever is specfied in XML. You would need to edit the various XML configuration files to set the <literal>bind_addr</literal> to the desired interfaces.
+ </para>
+ </section>
+
+ <section><title>Isolating JGroups Channels</title>
+ <para>
+ Within JBoss AS, there are a number of services that independently create JGroups channels -- 3 different JBoss Cache services (used for HttpSession replication, EJB3 SFSB replication and EJB3 entity replication) along with the general purpose clustering service called HAPartition that underlies most other JBossHA services.
+ </para>
+ <para>
+ It is critical that these channels only communicate with their intended peers; not with the channels used by other services and not with channels for the same service opened on machines not meant to be part of the group. Nodes improperly communicating with each other is one of the most common issues users have with JBoss AS clustering.
+ </para>
+ <para>
+ Whom a JGroups channel will communicate with is defined by its group name, multicast address, and multicast port, so isolating JGroups channels comes down to ensuring different channels use different values for the group name, multicast address and multicast port.
+ </para>
+ <para>
+ To isolate JGroups channels for different services on the same set of AS instances from each other, you MUST change the group name and the multicast port. In other words, each channel must have its own set of values.
+ </para>
+ <para>
+ For example, say we have a production cluster of 3 machines, each of which has an HAPartition deployed along with a JBoss Cache used for web session clustering. The HAPartition channels should not communicate with the JBoss Cache channels. They should use a different group name and multicast port. They can use the same multicast address, although they don't need to.
+ </para>
+ <para>
+ To isolate JGroups channels for the same service from other instances of the service on the network, you MUST change ALL three values. Each channel must have its own group name, multicast address, and multicast port.
+ </para>
+ <para>
+ For example, say we have a production cluster of 3 machines, each of which has an HAPartition deployed. On the same network there is also a QA cluster of 3 machines, which also has an HAPartition deployed. The HAPartition group name, multicast address, and multicast port for the production machines must be different from those used on the QA machines.
+ </para>
+ </section>
+
+
+ <section><title>Changing the Group Name</title>
+ <para>
+ The group name for a JGroups channel is configured via the service that starts the channel. Unfortunately, different services use different attribute names for configuring this. For HAPartition and related services configured in the deploy/cluster-service.xml file, this is configured via a PartitionName attribute. For JBoss Cache services, the name of the attribute is ClusterName.
+ </para>
+ <para>
+ Starting with JBoss AS 4.0.4, for the HAPartition and all the standard JBoss Cache services, we make it easy for you to create unique groups names simply by using the -g (a.k.a. –partition) switch when starting JBoss:
+ <screen>./run.sh -g QAPartition -b 192.168.1.100 -c all</screen>
+ This switch sets the jboss.partition.name system property, which is used as a component in the configuration of the group name in all the standard clustering configuration files. For example,
+<screen><![CDATA[<attribute name="ClusterName">Tomcat-${jboss.partition.name:Cluster}</attribute>]]></screen>
+ </para>
+ </section>
+
+
+
+<section><title>Changing the multicast address and port</title>
+ <para>
+ The -u (a.k.a. --udp) command line switch may be used to control the multicast address used by the JGroups channels opened by all standard AS services.
+<screen><![CDATA[/run.sh -u 230.1.2.3 -g QAPartition -b 192.168.1.100 -c all]]></screen>
+ This switch sets the jboss.partition.udpGroup system property, which you can see referenced in all of the standard protocol stack configs in JBoss AS:
+ </para>
+
+<programlisting><![CDATA[<Config>
+<UDP mcast_addr="${jboss.partition.udpGroup:228.1.2.3}"
+ ....]]>
+</programlisting>
+<para>
+ Unfortunately, setting the multicast ports is not so simple. As described above, by default there are four separate JGroups channels in the standard JBoss AS all configuration, and each should be given a unique port. There are no command line switches to set these, but the standard configuration files do use system properties to set them. So, they can be configured from the command line by using -D. For example,
+ </para>
+<programlisting>
+ /run.sh -u 230.1.2.3 -g QAPartition -Djboss.hapartition.mcast_port=12345 -Djboss.webpartition.mcast_port=23456 -Djboss.ejb3entitypartition.mcast_port=34567 -Djboss.ejb3sfsbpartition.mcast_port=45678 -b 192.168.1.100 -c all
+</programlisting>
+
+<para><emphasis>Why isn't it sufficient to change the group name?</emphasis></para>
+<para>
+ If channels with different group names share the same multicast address and port, the lower level JGroups protocols in each channel will see, process and eventually discard messages intended for the other group. This will at a minimum hurt performance and can lead to anomalous behavior.
+ </para>
+
+ <para><emphasis>Why do I need to change the multicast port if I change the address?</emphasis></para>
+ <para>
+ It should be sufficient to just change the address, but there is a problem on several operating systems whereby packets addressed to a particular multicast port are delivered to all listeners on that port, regardless of the multicast address they are listening on. So the recommendation is to change both the address and the port.
+ </para>
+</section>
+
+
+<section><title>JGroups Troubleshooting</title>
+ <para><emphasis>Nodes do not form a cluster</emphasis></para>
+
+ <para>
+ Make sure your machine is set up correctly for IP multicast. There are 2 test programs that can be used to detect this: McastReceiverTest and McastSenderTest. Go to the <literal>$JBOSS_HOME/server/all/lib</literal> directory and start McastReceiverTest, for example:
+<screen>java -cp jgroups.jar org.jgroups.tests.McastReceiverTest -mcast_addr 224.10.10.10 -port 5555 </screen>
+</para>
+
+<para>
+Then in another window start <literal>McastSenderTest</literal>:
+<screen>java -cp jgroups.jar org.jgroups.tests.McastSenderTest -mcast_addr 224.10.10.10 -port 5555</screen>
+</para>
+
+<para>
+ If you want to bind to a specific network interface card (NIC), use <literal>-bind_addr 192.168.0.2</literal>, where 192.168.0.2 is the IP address of the NIC to which you want to bind. Use this parameter in both the sender and the receiver.
+</para>
+<para>
+ You should be able to type in the <literal>McastSenderTest</literal> window and see the output in the <literal>McastReceiverTest</literal> window. If not, try to use -ttl 32 in the sender. If this still fails, consult a system administrator to help you setup IP multicast correctly, and ask the admin to make sure that multicast will work on the interface you have chosen or, if the machines have multiple interfaces, ask to be told the correct interface.
+Once you know multicast is working properly on each machine in your cluster, you can repeat the above test to test the network, putting the sender on one machine and the receiver on another.
+
+ </para>
+</section>
+
+<section><title>Causes of missing heartbeats in FD</title>
+ <para>
+ Sometimes a member is suspected by FD because a heartbeat ack has not been received for some time T (defined by timeout and max_tries). This can have multiple reasons, e.g. in a cluster of A,B,C,D; C can be suspected if (note that A pings B, B pings C, C pings D and D pings A):
+ </para>
+
+ <itemizedlist>
+ <listitem>
+ <para>
+ B or C are running at 100% CPU for more than T seconds. So even if C sends a heartbeat ack to B, B may not be able to process it because it is at 100%
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ B or C are garbage collecting, same as above.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ A combination of the 2 cases above
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ The network loses packets. This usually happens when there is a lot of traffic on the network, and the switch starts dropping packets (usually broadcasts first, then IP multicasts, TCP packets last).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ B or C are processing a callback. Let's say C received a remote method call over its channel and takes T+1 seconds to process it. During this time, C will not process any other messages, including heartbeats, and therefore B will not receive the heartbeat ack and will suspect C.
+ </para>
+ </listitem>
+</itemizedlist>
+
+</section>
+</section>
+
+ </chapter>
Property changes on: projects/docs/community/5/Clustering_Guide/en-US/Clustering_Guide_JGroups.xml
___________________________________________________________________
Name: svn:mergeinfo
+
Copied: projects/docs/community/5/Clustering_Guide/en-US/images/JGroupsStack.png (from rev 82580, projects/docs/community/5/Clustering_Guide/en-US/images/jbosscache-JGroupsStack.png)
===================================================================
(Binary files differ)
Property changes on: projects/docs/community/5/Clustering_Guide/en-US/images/JGroupsStack.png
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Name: svn:mergeinfo
+
Deleted: projects/docs/community/5/Clustering_Guide/en-US/images/jbosscache-JGroupsStack.png
===================================================================
(Binary files differ)
More information about the jboss-cvs-commits
mailing list