[jboss-user] [JBossCache] - Initial State Transfer Failure
alauro
do-not-reply at jboss.com
Thu Nov 1 16:17:12 EDT 2007
Hi - I'm new to JBossCache and we are having an initial state transfer failure. Once the clustered caches are up - they talk fine and new objects or updates replicate perfectly. The problem is that if one box goes down, when it comes back up, the cache coming up doesn't get the old data from the other node. New data replicates or any objects that get updated. We have a pretty strict firewall setup with 48866 open for multicast traffic. We're using JBoss 4.0.4 and JBossCache 2.x.
Here's our config and some relevant logging. I wonder if our Eviction Policy is the problem.
<?xml version="1.0" encoding="UTF-8"?>
<!-- ===================================================================== -->
<!-- Service to deploy and configure a replicating cache -->
<!-- This is the default file used for installations. -->
<!-- ===================================================================== -->
<!-- ==================================================================== -->
<!-- Defines TreeCache configuration -->
<!-- ==================================================================== -->
jboss:service=Naming
jboss:service=TransactionManager
jboss.aop:service=AspectDeployer
<!--
Configure the TransactionManager
-->
org.jboss.cache.BatchModeTransactionManagerLookup
<!--
Isolation level : SERIALIZABLE
REPEATABLE_READ (default)
READ_COMMITTED
READ_UNCOMMITTED
NONE
-->
REPEATABLE_READ
<!--
Valid modes are LOCAL
REPL_ASYNC
REPL_SYNC
INVALIDATION_ASYNC
INVALIDATION_SYNC
-->
REPL_ASYNC
<!--
Just used for async repl: use a replication queue
-->
false
<!--
Replication interval for replication queue (in ms)
-->
0
<!--
Max number of elements which trigger replication
-->
0
<!-- Name of cluster. Needs to be the same for all clusters, in order
to find each other
-->
TreeCache-Cluster
<!-- JGroups protocol stack properties. Can also be a URL,
e.g. file:/home/bela/default.xml
-->
<UDP mcast_addr="228.1.2.3" mcast_port="48866"
ip_ttl="64" ip_mcast="true"
mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
loopback="false" bind_addr="192.168.181.190" />
<PING timeout="2000" num_initial_members="3"
up_thread="false" down_thread="false"
initial_hosts="192.168.181.191[48866]"/>
<MERGE2 min_interval="10000" max_interval="20000"/>
<FD_SOCK/>
<VERIFY_SUSPECT timeout="1500"
up_thread="false" down_thread="false"/>
<pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800"
max_xmit_size="8192" up_thread="false" down_thread="false"/>
<UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10"
down_thread="false"/>
<pbcast.STABLE desired_avg_gossip="20000"
up_thread="false" down_thread="false"/>
<FRAG frag_size="8192"
down_thread="false" up_thread="false"/>
<pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
shun="true" print_local_addr="true"/>
<pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>
<!--
Whether or not to fetch state on joining a cluster
NOTE this used to be called FetchStateOnStartup and has been renamed to be more descriptive.
-->
true
<!--
The max amount of time (in milliseconds) we wait until the
initial state (ie. the contents of the cache) are retrieved from
existing members in a clustered environment
-->
20000
<!--
Number of milliseconds to wait until all responses for a
synchronous call have been received.
-->
20000
<!-- Max number of milliseconds to wait for a lock acquisition -->
15000
<!-- Eviction policy defines how long nodes in defined regions stay -->
<!-- in cache before being removed. In this case a cache entry can -->
<!-- stay in cache w/out any activity for the specified number of -->
<!-- seconds. BAM will update timestamp on the record every time -->
<!-- an interium accounting record is received. Likewise the record -->
<!-- will be removed when a accounting stop record is received. -->
<!-- So, this policy is used to remove nodes in cases where an -->
<!-- accounting stop record is lost. -->
<!-- -->
<!-- NOTE: the timeToLiveSeconds must be greater than the time -->
<!-- used to send interium accouning records. Also /SessionKeys -->
<!-- and Sessions must have a timeToLiveSeconds > /SessionKeyIndex -->
<!-- as the indexs have to be removed before the acutal data to -->
<!-- avoid a race condition. -->
5
<!-- See notes above -->
0
3900
<!-- See notes above, time must be > than /SessionKeyuIndex -->
0
3901
<!-- See notes above, time must be > than /sessionKeyIndex -->
0
3902
<!-- Default Eviction policy for all other Nodes in cache -->
0
0
<!--
Indicate whether to use marshalling or not. Set this to true if you are running under a scoped
class loader, e.g., inside an application server. Default is "false".
-->
false
false
2007-11-01 08:39:39,009 DEBUG [org.jgroups.conf.ClassConfigurator] mapping is:
1: class org.jgroups.stack.IpAddress
2: class org.jgroups.protocols.CAUSAL$CausalHeader
3: class org.jgroups.protocols.FD$FdHeader
4: class org.jgroups.protocols.FD_PID$FdHeader
5: class org.jgroups.protocols.FD_PROB$FdHeader
6: class org.jgroups.protocols.FD_SOCK$FdHeader
7: class org.jgroups.protocols.FragHeader
8: class org.jgroups.protocols.MERGE$MergeHeader
9: class org.jgroups.protocols.NakAckHeader
10: class org.jgroups.protocols.PARTITIONER$PartitionerHeader
11: class org.jgroups.protocols.PerfHeader
12: class org.jgroups.protocols.PIGGYBACK$PiggybackHeader
13: class org.jgroups.protocols.PingHeader
14: class org.jgroups.protocols.TcpHeader
15: class org.jgroups.protocols.TOTAL$Header
16: class org.jgroups.protocols.TOTAL_OLD$TotalHeader
17: class org.jgroups.protocols.TOTAL_TOKEN$TotalTokenHeader
18: class org.jgroups.protocols.TOTAL_TOKEN$RingTokenHeader
19: class org.jgroups.protocols.TunnelHeader
20: class org.jgroups.protocols.UdpHeader
21: class org.jgroups.protocols.UNICAST$UnicastHeader
22: class org.jgroups.protocols.VERIFY_SUSPECT$VerifyHeader
23: class org.jgroups.protocols.WANPIPE$WanPipeHeader
24: class org.jgroups.protocols.pbcast.GMS$GmsHeader
25: class org.jgroups.protocols.pbcast.NakAckHeader
26: class org.jgroups.protocols.pbcast.PbcastHeader
27: class org.jgroups.protocols.pbcast.STABLE$StableHeader
28: class org.jgroups.protocols.pbcast.STATE_TRANSFER$StateHeader
29: class org.jgroups.protocols.SMACK$SmackHeader
30: class org.jgroups.Message
31: class org.jgroups.View
32: class org.jgroups.ViewId
33: class org.jgroups.util.List
34: interface org.jgroups.Address
35: class [Lorg.jgroups.Address;
36: class org.jgroups.protocols.PingRsp
37: class [Ljava.lang.Object;
38: class java.util.Vector
39: class org.jgroups.protocols.pbcast.JoinRsp
40: class org.jgroups.protocols.pbcast.Digest
41: class java.util.Hashtable
50: class org.jgroups.service.lease.LeaseRequestHeader
51: class org.jgroups.service.lease.LeaseResponseHeader
52: class org.jgroups.service.lease.DenyResponseHeader
53: class org.jgroups.protocols.COMPRESS$CompressHeader
54: class org.jgroups.protocols.FC$FcHeader
55: class org.jgroups.protocols.WanPipeAddress
2007-11-01 08:39:39,024 DEBUG [org.jgroups.protocols.AUTOCONF] frag_size=64000
2007-11-01 08:39:39,061 DEBUG [org.jgroups.protocols.pbcast.GMS] changed role to org.jgroups.protocols.pbcast.ClientGmsImpl
2007-11-01 08:39:39,091 DEBUG [org.jboss.cache.marshall.VersionAwareMarshaller] Initialised with version 1.4.0 and versionInt 14
2007-11-01 08:39:39,091 DEBUG [org.jboss.cache.marshall.VersionAwareMarshaller] Using default marshaller class org.jboss.cache.marshall.TreeCacheMarshaller140
2007-11-01 08:39:39,091 DEBUG [org.jboss.cache.aop.TreeCacheAop] Created jboss.cache:service=TomcatClusteringCache
2007-11-01 08:39:39,091 DEBUG [org.jboss.cache.aop.TreeCacheAop] Starting jboss.cache:service=TomcatClusteringCache
2007-11-01 08:39:39,173 DEBUG [org.jgroups.protocols.UDP] creating sockets and starting threads
2007-11-01 08:39:39,173 INFO [org.jgroups.protocols.UDP] sockets will use interface 192.168.181.191
2007-11-01 08:39:39,187 INFO [org.jgroups.protocols.UDP] socket information:
local_addr=192.168.181.191:37990, mcast_addr=228.1.2.3:48866, bind_addr=/192.168.181.191, ttl=64
sock: bound to 192.168.181.191:37990, receive buffer size=80000, send buffer size=131071
mcast_recv_sock: bound to 192.168.181.191:48866, send buffer size=131071, receive buffer size=80000
mcast_send_sock: bound to 192.168.181.191:37991, send buffer size=131071, receive buffer size=80000
2007-11-01 08:39:39,188 DEBUG [org.jgroups.protocols.PING] adding my address (192.168.181.191:37990) to initial_hosts; initial_hosts=[[192.168.181.190:48866 ] [192.168.181.191:37990 ] ]
2007-11-01 08:39:39,190 INFO [STDOUT]
-------------------------------------------------------
GMS: address is 192.168.181.191:37990
-------------------------------------------------------
2007-11-01 08:39:39,190 DEBUG [org.jgroups.protocols.UDP] created unicast receiver thread
2007-11-01 08:39:41,193 DEBUG [org.jgroups.protocols.PingWaiter] initial mbrs are []
2007-11-01 08:39:41,194 DEBUG [org.jgroups.protocols.pbcast.ClientGmsImpl] initial_mbrs are []
2007-11-01 08:39:41,194 DEBUG [org.jgroups.protocols.pbcast.ClientGmsImpl] no initial members discovered: creating group as first member
2007-11-01 08:39:41,200 DEBUG [org.jgroups.protocols.pbcast.GMS] [local_addr=192.168.181.191:37990] view is [192.168.181.191:37990|0] [192.168.181.191:37990]
2007-11-01 08:39:41,204 INFO [org.jboss.cache.TreeCache] TreeCache local address is 192.168.181.191:37990
2007-11-01 08:39:41,206 DEBUG [org.jgroups.protocols.pbcast.STATE_TRANSFER] GET_STATE: first member (no state)
2007-11-01 08:39:41,206 INFO [org.jboss.cache.TreeCache] viewAccepted(): [192.168.181.191:37990|0] [192.168.181.191:37990]
2007-11-01 08:39:41,212 DEBUG [org.jboss.cache.TreeCache] transferred state is null (may be first member in cluster)
2007-11-01 08:39:41,216 DEBUG [org.jgroups.protocols.pbcast.GMS] 192.168.181.191:37990 changed role to org.jgroups.protocols.pbcast.CoordGmsImpl
2007-11-01 08:39:41,216 DEBUG [org.jgroups.protocols.pbcast.GMS] 192.168.181.191:37990 changed role to org.jgroups.protocols.pbcast.CoordGmsImpl
2007-11-01 08:39:41,217 DEBUG [org.jgroups.protocols.pbcast.ClientGmsImpl] created group (first member). My view is [192.168.181.191:37990|0], impl is org.jgroups.protocols.pbcast.CoordGmsImpl
2007-11-01 08:39:41,217 DEBUG [org.jgroups.protocols.FD_SOCK] VIEW_CHANGE received: [192.168.181.191:37990]
2007-11-01 08:39:41,217 DEBUG [org.jgroups.protocols.FD_SOCK] first member; cache is empty
2007-11-01 08:39:41,217 DEBUG [org.jgroups.protocols.MERGE2] merge task started as I'm the coordinator
2007-11-01 08:39:41,217 INFO [org.jboss.cache.TreeCache] State could not be retrieved (we are the first member in group)
2007-11-01 08:39:41,217 DEBUG [org.jboss.cache.eviction.RegionManager] Starting eviction timer
2007-11-01 08:39:41,218 INFO [org.jboss.cache.TreeCache] parseConfig(): PojoCacheConfig is empty
2007-11-01 08:39:41,218 DEBUG [org.jboss.cache.aop.TreeCacheAop] Started jboss.cache:service=TomcatClusteringCache
2007-11-01 08:39:41,235 DEBUG [org.jboss.cache.invalidation.InvalidationManager] Creating jboss.cache:service=InvalidationManager
2007-11-01 08:39:41,235 DEBUG [org.jboss.cache.invalidation.InvalidationManager] Created jboss.cache:service=InvalidationManager
2007-11-01 08:39:41,236 DEBUG [org.jboss.cache.invalidation.InvalidationManager] Starting jboss.cache:service=InvalidationManager
2007-11-01 08:39:41,236 DEBUG [org.jboss.cache.invalidation.InvalidationManager] Starting Invalidation Manager jboss.cache:service=InvalidationManager
2007-11-01 08:39:41,236 DEBUG [org.jboss.cache.invalidation.InvalidationManager] Started jboss.cache:service=InvalidationManager
2007-11-01 08:39:48,625 DEBUG [org.jgroups.protocols.pbcast.CoordGmsImpl] sender=192.168.181.190:37863, merge_id=[192.168.181.190:37863|1193927988632]
2007-11-01 08:39:48,626 DEBUG [org.jgroups.protocols.pbcast.CoordGmsImpl] response=GmsHeader[MERGE_RSP]: view=[192.168.181.191:37990|0] [192.168.181.191:37990], digest=[192.168.181.191:37990: [0 : 0], merge_rejected=false, merge_id=[192.168.181.190:37863|1193927988632]
2007-11-01 08:39:48,630 WARN [org.jgroups.protocols.pbcast.NAKACK] 192.168.181.191:37990] discarded message from non-member 192.168.181.190:37863
2007-11-01 08:39:48,632 DEBUG [org.jgroups.protocols.FD_SOCK] [SUSPECT] hdr=SUSPECT, mbrs=[192.168.181.191:37990]
2007-11-01 08:39:48,632 DEBUG [org.jgroups.protocols.FD_SOCK] member is 192.168.181.191:37990
2007-11-01 08:39:48,633 DEBUG [org.jgroups.protocols.pbcast.GMS] mcasting view {[192.168.181.190:37863|3] [192.168.181.190:37863, 192.168.181.191:37990]} (2 mbrs)
2007-11-01 08:39:48,633 DEBUG [org.jgroups.protocols.pbcast.CoordGmsImpl] view=[192.168.181.190:37863|3] [192.168.181.190:37863, 192.168.181.191:37990], digest=[192.168.181.190:37863: [433 : 486, 192.168.181.191:37990: [0 : 0]
2007-11-01 08:39:48,633 DEBUG [org.jgroups.protocols.pbcast.GMS] [local_addr=192.168.181.191:37990] view is [192.168.181.190:37863|3] [192.168.181.190:37863, 192.168.181.191:37990]
2007-11-01 08:39:48,633 DEBUG [org.jgroups.protocols.FD_SOCK] VIEW_CHANGE received: [192.168.181.190:37863, 192.168.181.191:37990]
2007-11-01 08:39:48,634 DEBUG [org.jgroups.protocols.FD_SOCK] determinePingDest()=192.168.181.190:37863, pingable_mbrs=[192.168.181.190:37863, 192.168.181.191:37990]
2007-11-01 08:39:48,634 INFO [org.jboss.cache.TreeCache] viewAccepted(): [192.168.181.190:37863|3] [192.168.181.190:37863, 192.168.181.191:37990]
2007-11-01 08:39:48,634 DEBUG [org.jgroups.protocols.pbcast.GMS] 192.168.181.191:37990 changed role to org.jgroups.protocols.pbcast.ParticipantGmsImpl
2007-11-01 08:39:48,937 DEBUG [org.jgroups.protocols.FD_SOCK] could not create socket to 192.168.181.190:37863; suspecting 192.168.181.190:37863
2007-11-01 08:39:48,937 DEBUG [org.jgroups.protocols.FD_SOCK] suspecting 192.168.181.190:37863 (own address is 192.168.181.191:37990)
2007-11-01 08:39:48,937 DEBUG [org.jgroups.protocols.FD_SOCK] mbr=192.168.181.190:37863 (size=1)
2007-11-01 08:39:48,938 DEBUG [org.jgroups.protocols.FD_SOCK] determinePingDest()=null, pingable_mbrs=[192.168.181.191:37990]
2007-11-01 08:39:48,938 DEBUG [org.jgroups.protocols.FD_SOCK] pinger thread terminated
2007-11-01 08:39:48,938 DEBUG [org.jgroups.protocols.FD_SOCK] [SUSPECT] hdr=SUSPECT, mbrs=[192.168.181.190:37863]
2007-11-01 08:39:48,939 DEBUG [org.jgroups.protocols.FD_SOCK] member is 192.168.181.190:37863
2007-11-01 08:39:58,068 DEBUG [org.jgroups.protocols.pbcast.ParticipantGmsImpl] view=[192.168.181.190:37863|3] [192.168.181.190:37863, 192.168.181.191:37990]
2007-11-01 08:39:58,068 DEBUG [org.jgroups.protocols.pbcast.GMS] [local_addr=192.168.181.191:37990] view is [192.168.181.190:37863|3] [192.168.181.190:37863, 192.168.181.191:37990]
2007-11-01 08:39:58,068 DEBUG [org.jgroups.protocols.pbcast.GMS] [192.168.181.191:37990] received view <= current view; discarding it (current vid: [192.168.181.190:37863|3], new vid: [192.168.181.190:37863|3])
2007-11-01 08:39:58,627 DEBUG [org.jgroups.protocols.pbcast.CoordGmsImpl] cancelling merge (merge_id=[192.168.181.190:37863|1193927988632])
2007-11-01 08:40:01,234 DEBUG [org.jboss.cache.eviction.BaseEvictionAlgorithm] Adding element /SessionKeyIndex/IP_ADDR/70.1.2.3 for a node that doesn't exist yet. Process as an add.
2007-11-01 08:40:01,235 DEBUG [org.jboss.cache.eviction.BaseEvictionAlgorithm] Adding element /SessionKeyIndex/DEVICE_IP_LIST/10.10.10.10 for a node that doesn't exist yet. Process as an add.
2007-11-01 08:40:01,235 DEBUG [org.jboss.cache.eviction.BaseEvictionAlgorithm] Adding element /SessionKeyIndex/USER_NAME/testuser2 for a node that doesn't exist yet. Process as an add.
2007-11-01 08:40:01,235 DEBUG [org.jboss.cache.eviction.BaseEvictionAlgorithm] Adding element /SessionKeyIndex/SERVICE_LIST/internet2 for a node that doesn't exist yet. Process as an add.
2007-11-01 08:40:01,235 DEBUG [org.jboss.cache.eviction.BaseEvictionAlgorithm] Adding element /Sessions/42E2E6FCD2EC4345 for a node that doesn't exist yet. Process as an add.
2007-11-01 08:40:01,235 DEBUG [org.jboss.cache.eviction.BaseEvictionAlgorithm] Adding element /SessionKeys/42E2E6FCD2EC4345 for a node that doesn't exist yet. Process as an add.
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4101073#4101073
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4101073
More information about the jboss-user
mailing list