5:02 p.m.
The short version of this question is: I have two nodes, A & B, with TreeCache
hibernate 2nd-level cache replication. Node B will have several GCs during the day that
make it unresponsive for minutes at a time. During these, Node A throws
ReplicationExceptions. I want to set my jbosscache/jgroups configuration to enable node A
to withstand these GC events on node B, either by temporarily shunning or by having
appropriate timeouts, or some combination of both.
Long Version:
I have inherited a jboss cluster. I have been doing a great deal of reading, I have gone
through the jgroups, jbosscache, & hibernate jbosscache wikis here on jboss.org.
We have 2 nodes , A & B, in a cluster. Both are running:
JBoss 4.0.5
JBossCache 1.4.1.SP3
JGroups 2.4.1-SP1.
The jboss-config.xml for the hibernate treecache (identical on both):
| <?xml version="1.0" encoding="UTF-8"?>
| <server>
| <mbean code="org.jboss.cache.TreeCache"
| name="jboss.cache:service=HibernateTreeCache">
|
| <depends>jboss:service=Naming</depends>
| <depends>jboss:service=TransactionManager</depends>
|
| <attribute
name="ClusterName">Hibernate-${jboss.partition.name:Cluster}</attribute>
|
| <attribute
name="IsolationLevel">READ_COMMITTED</attribute>
|
| <attribute name="CacheMode">REPL_SYNC</attribute>
|
| <attribute name="UseRegionBasedMarshalling">false</attribute>
|
| <attribute name="InactiveOnStartup">false</attribute>
|
| <attribute
name="TransactionManagerLookupClass">org.jboss.cache.BatchModeTransactionManagerLookup</attribute>
|
| <attribute name="ClusterConfig">
| <config>
| <TCP bind_addr="${partition.tcphost:HIB3-MISCONFIGURED}"
| start_port="${partition.tcpport.hib3:HIB3-MISCONFIGURED}"
loopback="false"
| tcp_nodelay="false" up_thread="false"
down_thread="false"/>
| <TCPPING
initial_hosts="${partition.tcphosts.hib3:HIB3-MISCONFIGURED}"
| port_range="3" timeout="3500"
| num_initial_members="3" up_thread="false"
down_thread="false"/>
| <MERGE2 min_interval="20000"
max_interval="100000"
| down_thread="false" up_thread="false"/>
| <FD_SOCK down_thread="false"
up_thread="false"/>
| <FD shun="true" down_thread="false"
up_thread="false"
| timeout="20000" max_tries="5"/>
| <VERIFY_SUSPECT timeout="1500"
down_thread="false" up_thread="false"/>
| <pbcast.NAKACK up_thread="false"
down_thread="false" gc_lag="100"
| retransmit_timeout="60000"/>
| <pbcast.STABLE desired_avg_gossip="50000"
up_thread="false" down_thread="false" />
| <pbcast.GMS join_timeout="5000"
join_retry_timeout="2000" shun="true"
| print_local_addr="true" down_thread="false"
up_thread="false"/>
| <pbcast.STATE_TRANSFER up_thread="false"
down_thread="false"/>
| </config>
|
|
| </attribute>
|
| <attribute name="FetchInMemoryState">false</attribute>
| <attribute
name="InitialStateRetrievalTimeout">20000</attribute>
|
| <attribute name="SyncReplTimeout">20000</attribute>
|
| <attribute
name="LockAcquisitionTimeout">15000</attribute>
|
| <attribute name="BuddyReplicationConfig">
| <config>
| <buddyReplicationEnabled>false</buddyReplicationEnabled>
|
<buddyLocatorClass>org.jboss.cache.buddyreplication.NextMemberBuddyLocator</buddyLocatorClass>
| <buddyLocatorProperties>
| numBuddies = 1
| ignoreColocatedBuddies = true
| </buddyLocatorProperties>
|
| <buddyPoolName>default</buddyPoolName>
|
<buddyCommunicationTimeout>2000</buddyCommunicationTimeout>
|
| <autoDataGravitation>false</autoDataGravitation>
|
<dataGravitationRemoveOnFind>true</dataGravitationRemoveOnFind>
|
<dataGravitationSearchBackupTrees>true</dataGravitationSearchBackupTrees>
|
| </config>
| </attribute>
|
| </mbean>
|
| </server>
|
Our cluster-service.xml:
| <?xml version="1.0" encoding="UTF-8"?>
|
| <server>
|
| <mbean code="org.jboss.ha.framework.server.ClusterPartition"
| name="jboss:service=${jboss.partition.name:DefaultPartition}">
|
| <attribute
name="PartitionName">${jboss.partition.name:DefaultPartition}</attribute>
|
| <attribute
name="NodeAddress">${jboss.bind.address}</attribute>
|
| <attribute name="DeadlockDetection">False</attribute>
|
| <attribute name="StateTransferTimeout">30000</attribute>
|
| <attribute name="PartitionConfig">
| <Config>
| <TCP
bind_addr="${partition.tcphost:CLUSTERCONFIG-MISCONFIGURED}"
start_port="${partition.tcpport.cluster:CLUSTERCONFIG-MISCONFIGURED}"
loopback="false"
| recv_buf_size="2000000" send_buf_size="640000"
| tcp_nodelay="true" up_thread="true"
down_thread="true"/>
| <TCPPING
initial_hosts="${partition.tcphosts.cluster:CLUSTERCONFIG-MISCONFIGURED}"
| port_range="3" timeout="3500"
| num_initial_members="3" up_thread="true"
down_thread="true"/>
| <MERGE2 min_interval="10000" max_interval="20000"
/>
| <FD_SOCK down_thread="true" up_thread="true"/>
| <FD shun="true" up_thread="true"
down_thread="true"
| timeout="10000" max_tries="5"/>
| <VERIFY_SUSPECT timeout="3000" down_thread="true"
up_thread="true" />
| <pbcast.NAKACK up_thread="true" down_thread="true"
gc_lag="100"
| retransmit_timeout="300,600,1200,2400,4800"/>
| <pbcast.STABLE desired_avg_gossip="20000"
max_bytes="400000"
| down_thread="true" up_thread="true" />
| <pbcast.GMS join_timeout="5000"
join_retry_timeout="2000" shun="true"
| print_local_addr="true" up_thread="true"
down_thread="true"/>
| <FC max_credits="2000000" down_thread="true"
up_thread="true"
| min_threshold="0.10"/>
| <FRAG2 frag_size="60000" down_thread="true"
up_thread="true"/>
| <pbcast.STATE_TRANSFER up_thread="true"
down_thread="true"/>
| </Config>
| </attribute>
| <depends>jboss:service=Naming</depends>
| </mbean>
|
|
| <mbean
code="org.jboss.ha.hasessionstate.server.HASessionStateService"
| name="jboss:service=HASessionState">
| <depends>jboss:service=Naming</depends>
| <!-- We now inject the partition into the HAJNDI service instead
| of requiring that the partition name be passed -->
| <depends optional-attribute-name="ClusterPartition"
|
proxy-type="attribute">jboss:service=${jboss.partition.name:DefaultPartition}</depends>
| <!-- JNDI name under which the service is bound -->
| <attribute
name="JndiName">/HASessionState/Default</attribute>
| <!-- Max delay before cleaning unreclaimed state.
| Defaults to 30*60*1000 => 30 minutes -->
| <attribute name="BeanCleaningDelay">0</attribute>
| </mbean>
|
| <mbean code="org.jboss.ha.jndi.HANamingService"
| name="jboss:service=HAJNDI">
| <!-- We now inject the partition into the HAJNDI service instead
| of requiring that the partition name be passed -->
| <depends optional-attribute-name="ClusterPartition"
|
proxy-type="attribute">jboss:service=${jboss.partition.name:DefaultPartition}</depends>
| <!-- Bind address of bootstrap and HA-JNDI RMI endpoints -->
| <attribute
name="BindAddress">${jboss.bind.address}</attribute>
| <!-- Port on which the HA-JNDI stub is made available -->
| <attribute name="Port">1100</attribute>
| <!-- RmiPort to be used by the HA-JNDI service once bound. 0 => auto.
-->
| <attribute name="RmiPort">1101</attribute>
| <!-- Accept backlog of the bootstrap socket -->
| <attribute name="Backlog">50</attribute>
| <!-- The thread pool service used to control the bootstrap and
| auto discovery lookups -->
| <depends optional-attribute-name="LookupPool"
|
proxy-type="attribute">jboss.system:service=ThreadPool</depends>
|
| <!-- A flag to disable the auto discovery via multicast -->
| <attribute name="DiscoveryDisabled">false</attribute>
| <!-- Set the auto-discovery bootstrap multicast bind address. If not
| specified and a BindAddress is specified, the BindAddress will be used. -->
| <attribute
name="AutoDiscoveryBindAddress">${jboss.bind.address}</attribute>
| <!-- Multicast Address and group port used for auto-discovery -->
| <attribute
name="AutoDiscoveryAddress">${jboss.partition.udpGroup:230.0.0.4}</attribute>
| <attribute name="AutoDiscoveryGroup">1102</attribute>
| <!-- The TTL (time-to-live) for autodiscovery IP multicast packets -->
| <attribute name="AutoDiscoveryTTL">16</attribute>
| <!-- The load balancing policy for HA-JNDI -->
| <attribute
name="LoadBalancePolicy">org.jboss.ha.framework.interfaces.RoundRobin</attribute>
|
| <!-- Client socket factory to be used for client-server
| RMI invocations during JNDI queries
| <attribute name="ClientSocketFactory">custom</attribute>
| -->
| <!-- Server socket factory to be used for client-server
| RMI invocations during JNDI queries
| <attribute name="ServerSocketFactory">custom</attribute>
| -->
| </mbean>
|
| <mbean code="org.jboss.invocation.jrmp.server.JRMPInvokerHA"
| name="jboss:service=invoker,type=jrmpha">
| <attribute
name="ServerAddress">${jboss.bind.address}</attribute>
| <attribute name="RMIObjectPort">4447</attribute>
| <!--
| <attribute
name="RMIClientSocketFactory">custom</attribute>
| <attribute
name="RMIServerSocketFactory">custom</attribute>
| -->
| <depends>jboss:service=Naming</depends>
| </mbean>
|
| <!-- the JRMPInvokerHA creates a thread per request. This implementation uses a
pool of threads -->
| <mbean code="org.jboss.invocation.pooled.server.PooledInvokerHA"
| name="jboss:service=invoker,type=pooledha">
| <attribute name="NumAcceptThreads">1</attribute>
| <attribute name="MaxPoolSize">300</attribute>
| <attribute name="ClientMaxPoolSize">300</attribute>
| <attribute name="SocketTimeout">60000</attribute>
| <attribute
name="ServerBindAddress">${jboss.bind.address}</attribute>
| <attribute name="ServerBindPort">4446</attribute>
| <attribute
name="ClientConnectAddress">${jboss.bind.address}</attribute>
| <attribute name="ClientConnectPort">0</attribute>
| <attribute name="EnableTcpNoDelay">false</attribute>
| <depends
optional-attribute-name="TransactionManagerService">jboss:service=TransactionManager</depends>
| <depends>jboss:service=Naming</depends>
| </mbean>
|
| <mbean
code="org.jboss.cache.invalidation.bridges.JGCacheInvalidationBridge"
| name="jboss.cache:service=InvalidationBridge,type=JavaGroups">
| <!-- We now inject the partition into the HAJNDI service instead
| of requiring that the partition name be passed -->
| <depends optional-attribute-name="ClusterPartition"
|
proxy-type="attribute">jboss:service=${jboss.partition.name:DefaultPartition}</depends>
| <depends>jboss.cache:service=InvalidationManager</depends>
| <attribute
name="InvalidationManager">jboss.cache:service=InvalidationManager</attribute>
| <attribute name="BridgeName">DefaultJGBridge</attribute>
| </mbean>
|
| </server>
|
We occasionally get ReplicationExceptions on A, and we have been able to verify that these
occur during long (up to 4-5 minutes) GCs on B, where the jvm becomes unresponsive.
As I read the config snippets above, node A will not shun until it doesn't receive a
heartbeat for at least (20x5 + 1.5) = 101.5 seconds, but the replication timeout is 20
seconds.
So my questions are -
1) If I change the config so A will shun B before SyncReplTimeout, will this prevent
replication during the time where B is unresponsive (hence preventing the
ReplicationExceptions)?
2) I then am expecting B to be shunned somewhate regularly, but I always want B to be able
to rejoin the cluster when it becomes responsive again. From what I'm reading, this
means setting shun=false. Without shunning, how do I prevent replication to the
unresponsive node B?
3) Are there further caveats that I need to consider? Will I need to make similar
timeout/config changes to the cluster-config.xml?
Thank you very much for your time.
View the original post :
http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4062708#...
Reply to the post :
http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&a...