3:17 p.m.
Hi - I'm new to JBossCache and we are having an initial state transfer failure. Once
the clustered caches are up - they talk fine and new objects or updates replicate
perfectly. The problem is that if one box goes down, when it comes back up, the cache
coming up doesn't get the old data from the other node. New data replicates or any
objects that get updated. We have a pretty strict firewall setup with 48866 open for
multicast traffic. We're using JBoss 4.0.4 and JBossCache 2.x.
Here's our config and some relevant logging. I wonder if our Eviction Policy is the
problem.
<?xml version="1.0" encoding="UTF-8"?>
<!-- ===================================================================== -->
<!-- Service to deploy and configure a replicating cache -->
<!-- This is the default file used for installations. -->
<!-- ===================================================================== -->
<!-- ==================================================================== -->
<!-- Defines TreeCache configuration -->
<!-- ==================================================================== -->
jboss:service=Naming
jboss:service=TransactionManager
jboss.aop:service=AspectDeployer
<!--
Configure the TransactionManager
-->
org.jboss.cache.BatchModeTransactionManagerLookup
<!--
Isolation level : SERIALIZABLE
REPEATABLE_READ (default)
READ_COMMITTED
READ_UNCOMMITTED
NONE
-->
REPEATABLE_READ
<!--
Valid modes are LOCAL
REPL_ASYNC
REPL_SYNC
INVALIDATION_ASYNC
INVALIDATION_SYNC
-->
REPL_ASYNC
<!--
Just used for async repl: use a replication queue
-->
false
<!--
Replication interval for replication queue (in ms)
-->
0
<!--
Max number of elements which trigger replication
-->
0
<!-- Name of cluster. Needs to be the same for all clusters, in order
to find each other
-->
TreeCache-Cluster
<!-- JGroups protocol stack properties. Can also be a URL,
e.g. file:/home/bela/default.xml
-->
<UDP mcast_addr="228.1.2.3" mcast_port="48866"
ip_ttl="64" ip_mcast="true"
mcast_send_buf_size="150000"
mcast_recv_buf_size="80000"
ucast_send_buf_size="150000"
ucast_recv_buf_size="80000"
loopback="false" bind_addr="192.168.181.190" />
<PING timeout="2000" num_initial_members="3"
up_thread="false" down_thread="false"
initial_hosts="192.168.181.191[48866]"/>
<MERGE2 min_interval="10000"
max_interval="20000"/>
<FD_SOCK/>
<VERIFY_SUSPECT timeout="1500"
up_thread="false" down_thread="false"/>
<pbcast.NAKACK gc_lag="50"
retransmit_timeout="600,1200,2400,4800"
max_xmit_size="8192" up_thread="false"
down_thread="false"/>
<UNICAST timeout="600,1200,2400" window_size="100"
min_threshold="10"
down_thread="false"/>
<pbcast.STABLE desired_avg_gossip="20000"
up_thread="false" down_thread="false"/>
<FRAG frag_size="8192"
down_thread="false" up_thread="false"/>
<pbcast.GMS join_timeout="5000"
join_retry_timeout="2000"
shun="true" print_local_addr="true"/>
<pbcast.STATE_TRANSFER up_thread="true"
down_thread="true"/>
<!--
Whether or not to fetch state on joining a cluster
NOTE this used to be called FetchStateOnStartup and has been renamed to be more
descriptive.
-->
true
<!--
The max amount of time (in milliseconds) we wait until the
initial state (ie. the contents of the cache) are retrieved from
existing members in a clustered environment
-->
20000
<!--
Number of milliseconds to wait until all responses for a
synchronous call have been received.
-->
20000
<!-- Max number of milliseconds to wait for a lock acquisition -->
15000
<!-- Eviction policy defines how long nodes in defined regions stay -->
<!-- in cache before being removed. In this case a cache entry can -->
<!-- stay in cache w/out any activity for the specified number of -->
<!-- seconds. BAM will update timestamp on the record every time -->
<!-- an interium accounting record is received. Likewise the record -->
<!-- will be removed when a accounting stop record is received. -->
<!-- So, this policy is used to remove nodes in cases where an -->
<!-- accounting stop record is lost. -->
<!-- -->
<!-- NOTE: the timeToLiveSeconds must be greater than the time -->
<!-- used to send interium accouning records. Also /SessionKeys -->
<!-- and Sessions must have a timeToLiveSeconds > /SessionKeyIndex -->
<!-- as the indexs have to be removed before the acutal data to -->
<!-- avoid a race condition. -->
5
<!-- See notes above -->
0
3900
<!-- See notes above, time must be > than /SessionKeyuIndex -->
0
3901
<!-- See notes above, time must be > than /sessionKeyIndex -->
0
3902
<!-- Default Eviction policy for all other Nodes in cache -->
0
0
<!--
Indicate whether to use marshalling or not. Set this to true if you are running
under a scoped
class loader, e.g., inside an application server. Default is "false".
-->
false
false
2007-11-01 08:39:39,009 DEBUG [org.jgroups.conf.ClassConfigurator] mapping is:
1: class org.jgroups.stack.IpAddress
2: class org.jgroups.protocols.CAUSAL$CausalHeader
3: class org.jgroups.protocols.FD$FdHeader
4: class org.jgroups.protocols.FD_PID$FdHeader
5: class org.jgroups.protocols.FD_PROB$FdHeader
6: class org.jgroups.protocols.FD_SOCK$FdHeader
7: class org.jgroups.protocols.FragHeader
8: class org.jgroups.protocols.MERGE$MergeHeader
9: class org.jgroups.protocols.NakAckHeader
10: class org.jgroups.protocols.PARTITIONER$PartitionerHeader
11: class org.jgroups.protocols.PerfHeader
12: class org.jgroups.protocols.PIGGYBACK$PiggybackHeader
13: class org.jgroups.protocols.PingHeader
14: class org.jgroups.protocols.TcpHeader
15: class org.jgroups.protocols.TOTAL$Header
16: class org.jgroups.protocols.TOTAL_OLD$TotalHeader
17: class org.jgroups.protocols.TOTAL_TOKEN$TotalTokenHeader
18: class org.jgroups.protocols.TOTAL_TOKEN$RingTokenHeader
19: class org.jgroups.protocols.TunnelHeader
20: class org.jgroups.protocols.UdpHeader
21: class org.jgroups.protocols.UNICAST$UnicastHeader
22: class org.jgroups.protocols.VERIFY_SUSPECT$VerifyHeader
23: class org.jgroups.protocols.WANPIPE$WanPipeHeader
24: class org.jgroups.protocols.pbcast.GMS$GmsHeader
25: class org.jgroups.protocols.pbcast.NakAckHeader
26: class org.jgroups.protocols.pbcast.PbcastHeader
27: class org.jgroups.protocols.pbcast.STABLE$StableHeader
28: class org.jgroups.protocols.pbcast.STATE_TRANSFER$StateHeader
29: class org.jgroups.protocols.SMACK$SmackHeader
30: class org.jgroups.Message
31: class org.jgroups.View
32: class org.jgroups.ViewId
33: class org.jgroups.util.List
34: interface org.jgroups.Address
35: class [Lorg.jgroups.Address;
36: class org.jgroups.protocols.PingRsp
37: class [Ljava.lang.Object;
38: class java.util.Vector
39: class org.jgroups.protocols.pbcast.JoinRsp
40: class org.jgroups.protocols.pbcast.Digest
41: class java.util.Hashtable
50: class org.jgroups.service.lease.LeaseRequestHeader
51: class org.jgroups.service.lease.LeaseResponseHeader
52: class org.jgroups.service.lease.DenyResponseHeader
53: class org.jgroups.protocols.COMPRESS$CompressHeader
54: class org.jgroups.protocols.FC$FcHeader
55: class org.jgroups.protocols.WanPipeAddress
2007-11-01 08:39:39,024 DEBUG [org.jgroups.protocols.AUTOCONF] frag_size=64000
2007-11-01 08:39:39,061 DEBUG [org.jgroups.protocols.pbcast.GMS] changed role to
org.jgroups.protocols.pbcast.ClientGmsImpl
2007-11-01 08:39:39,091 DEBUG [org.jboss.cache.marshall.VersionAwareMarshaller]
Initialised with version 1.4.0 and versionInt 14
2007-11-01 08:39:39,091 DEBUG [org.jboss.cache.marshall.VersionAwareMarshaller] Using
default marshaller class org.jboss.cache.marshall.TreeCacheMarshaller140
2007-11-01 08:39:39,091 DEBUG [org.jboss.cache.aop.TreeCacheAop] Created
jboss.cache:service=TomcatClusteringCache
2007-11-01 08:39:39,091 DEBUG [org.jboss.cache.aop.TreeCacheAop] Starting
jboss.cache:service=TomcatClusteringCache
2007-11-01 08:39:39,173 DEBUG [org.jgroups.protocols.UDP] creating sockets and starting
threads
2007-11-01 08:39:39,173 INFO [org.jgroups.protocols.UDP] sockets will use interface
192.168.181.191
2007-11-01 08:39:39,187 INFO [org.jgroups.protocols.UDP] socket information:
local_addr=192.168.181.191:37990, mcast_addr=228.1.2.3:48866, bind_addr=/192.168.181.191,
ttl=64
sock: bound to 192.168.181.191:37990, receive buffer size=80000, send buffer size=131071
mcast_recv_sock: bound to 192.168.181.191:48866, send buffer size=131071, receive buffer
size=80000
mcast_send_sock: bound to 192.168.181.191:37991, send buffer size=131071, receive buffer
size=80000
2007-11-01 08:39:39,188 DEBUG [org.jgroups.protocols.PING] adding my address
(192.168.181.191:37990) to initial_hosts; initial_hosts=[[192.168.181.190:48866 ]
[192.168.181.191:37990 ] ]
2007-11-01 08:39:39,190 INFO [STDOUT]
-------------------------------------------------------
GMS: address is 192.168.181.191:37990
-------------------------------------------------------
2007-11-01 08:39:39,190 DEBUG [org.jgroups.protocols.UDP] created unicast receiver thread
2007-11-01 08:39:41,193 DEBUG [org.jgroups.protocols.PingWaiter] initial mbrs are []
2007-11-01 08:39:41,194 DEBUG [org.jgroups.protocols.pbcast.ClientGmsImpl] initial_mbrs
are []
2007-11-01 08:39:41,194 DEBUG [org.jgroups.protocols.pbcast.ClientGmsImpl] no initial
members discovered: creating group as first member
2007-11-01 08:39:41,200 DEBUG [org.jgroups.protocols.pbcast.GMS]
[local_addr=192.168.181.191:37990] view is [192.168.181.191:37990|0]
[192.168.181.191:37990]
2007-11-01 08:39:41,204 INFO [org.jboss.cache.TreeCache] TreeCache local address is
192.168.181.191:37990
2007-11-01 08:39:41,206 DEBUG [org.jgroups.protocols.pbcast.STATE_TRANSFER] GET_STATE:
first member (no state)
2007-11-01 08:39:41,206 INFO [org.jboss.cache.TreeCache] viewAccepted():
[192.168.181.191:37990|0] [192.168.181.191:37990]
2007-11-01 08:39:41,212 DEBUG [org.jboss.cache.TreeCache] transferred state is null (may
be first member in cluster)
2007-11-01 08:39:41,216 DEBUG [org.jgroups.protocols.pbcast.GMS] 192.168.181.191:37990
changed role to org.jgroups.protocols.pbcast.CoordGmsImpl
2007-11-01 08:39:41,216 DEBUG [org.jgroups.protocols.pbcast.GMS] 192.168.181.191:37990
changed role to org.jgroups.protocols.pbcast.CoordGmsImpl
2007-11-01 08:39:41,217 DEBUG [org.jgroups.protocols.pbcast.ClientGmsImpl] created group
(first member). My view is [192.168.181.191:37990|0], impl is
org.jgroups.protocols.pbcast.CoordGmsImpl
2007-11-01 08:39:41,217 DEBUG [org.jgroups.protocols.FD_SOCK] VIEW_CHANGE received:
[192.168.181.191:37990]
2007-11-01 08:39:41,217 DEBUG [org.jgroups.protocols.FD_SOCK] first member; cache is
empty
2007-11-01 08:39:41,217 DEBUG [org.jgroups.protocols.MERGE2] merge task started as I'm
the coordinator
2007-11-01 08:39:41,217 INFO [org.jboss.cache.TreeCache] State could not be retrieved (we
are the first member in group)
2007-11-01 08:39:41,217 DEBUG [org.jboss.cache.eviction.RegionManager] Starting eviction
timer
2007-11-01 08:39:41,218 INFO [org.jboss.cache.TreeCache] parseConfig(): PojoCacheConfig
is empty
2007-11-01 08:39:41,218 DEBUG [org.jboss.cache.aop.TreeCacheAop] Started
jboss.cache:service=TomcatClusteringCache
2007-11-01 08:39:41,235 DEBUG [org.jboss.cache.invalidation.InvalidationManager] Creating
jboss.cache:service=InvalidationManager
2007-11-01 08:39:41,235 DEBUG [org.jboss.cache.invalidation.InvalidationManager] Created
jboss.cache:service=InvalidationManager
2007-11-01 08:39:41,236 DEBUG [org.jboss.cache.invalidation.InvalidationManager] Starting
jboss.cache:service=InvalidationManager
2007-11-01 08:39:41,236 DEBUG [org.jboss.cache.invalidation.InvalidationManager] Starting
Invalidation Manager jboss.cache:service=InvalidationManager
2007-11-01 08:39:41,236 DEBUG [org.jboss.cache.invalidation.InvalidationManager] Started
jboss.cache:service=InvalidationManager
2007-11-01 08:39:48,625 DEBUG [org.jgroups.protocols.pbcast.CoordGmsImpl]
sender=192.168.181.190:37863, merge_id=[192.168.181.190:37863|1193927988632]
2007-11-01 08:39:48,626 DEBUG [org.jgroups.protocols.pbcast.CoordGmsImpl]
response=GmsHeader[MERGE_RSP]: view=[192.168.181.191:37990|0] [192.168.181.191:37990],
digest=[192.168.181.191:37990: [0 : 0], merge_rejected=false,
merge_id=[192.168.181.190:37863|1193927988632]
2007-11-01 08:39:48,630 WARN [org.jgroups.protocols.pbcast.NAKACK] 192.168.181.191:37990]
discarded message from non-member 192.168.181.190:37863
2007-11-01 08:39:48,632 DEBUG [org.jgroups.protocols.FD_SOCK] [SUSPECT] hdr=SUSPECT,
mbrs=[192.168.181.191:37990]
2007-11-01 08:39:48,632 DEBUG [org.jgroups.protocols.FD_SOCK] member is
192.168.181.191:37990
2007-11-01 08:39:48,633 DEBUG [org.jgroups.protocols.pbcast.GMS] mcasting view
{[192.168.181.190:37863|3] [192.168.181.190:37863, 192.168.181.191:37990]} (2 mbrs)
2007-11-01 08:39:48,633 DEBUG [org.jgroups.protocols.pbcast.CoordGmsImpl]
view=[192.168.181.190:37863|3] [192.168.181.190:37863, 192.168.181.191:37990],
digest=[192.168.181.190:37863: [433 : 486, 192.168.181.191:37990: [0 : 0]
2007-11-01 08:39:48,633 DEBUG [org.jgroups.protocols.pbcast.GMS]
[local_addr=192.168.181.191:37990] view is [192.168.181.190:37863|3]
[192.168.181.190:37863, 192.168.181.191:37990]
2007-11-01 08:39:48,633 DEBUG [org.jgroups.protocols.FD_SOCK] VIEW_CHANGE received:
[192.168.181.190:37863, 192.168.181.191:37990]
2007-11-01 08:39:48,634 DEBUG [org.jgroups.protocols.FD_SOCK]
determinePingDest()=192.168.181.190:37863, pingable_mbrs=[192.168.181.190:37863,
192.168.181.191:37990]
2007-11-01 08:39:48,634 INFO [org.jboss.cache.TreeCache] viewAccepted():
[192.168.181.190:37863|3] [192.168.181.190:37863, 192.168.181.191:37990]
2007-11-01 08:39:48,634 DEBUG [org.jgroups.protocols.pbcast.GMS] 192.168.181.191:37990
changed role to org.jgroups.protocols.pbcast.ParticipantGmsImpl
2007-11-01 08:39:48,937 DEBUG [org.jgroups.protocols.FD_SOCK] could not create socket to
192.168.181.190:37863; suspecting 192.168.181.190:37863
2007-11-01 08:39:48,937 DEBUG [org.jgroups.protocols.FD_SOCK] suspecting
192.168.181.190:37863 (own address is 192.168.181.191:37990)
2007-11-01 08:39:48,937 DEBUG [org.jgroups.protocols.FD_SOCK] mbr=192.168.181.190:37863
(size=1)
2007-11-01 08:39:48,938 DEBUG [org.jgroups.protocols.FD_SOCK] determinePingDest()=null,
pingable_mbrs=[192.168.181.191:37990]
2007-11-01 08:39:48,938 DEBUG [org.jgroups.protocols.FD_SOCK] pinger thread terminated
2007-11-01 08:39:48,938 DEBUG [org.jgroups.protocols.FD_SOCK] [SUSPECT] hdr=SUSPECT,
mbrs=[192.168.181.190:37863]
2007-11-01 08:39:48,939 DEBUG [org.jgroups.protocols.FD_SOCK] member is
192.168.181.190:37863
2007-11-01 08:39:58,068 DEBUG [org.jgroups.protocols.pbcast.ParticipantGmsImpl]
view=[192.168.181.190:37863|3] [192.168.181.190:37863, 192.168.181.191:37990]
2007-11-01 08:39:58,068 DEBUG [org.jgroups.protocols.pbcast.GMS]
[local_addr=192.168.181.191:37990] view is [192.168.181.190:37863|3]
[192.168.181.190:37863, 192.168.181.191:37990]
2007-11-01 08:39:58,068 DEBUG [org.jgroups.protocols.pbcast.GMS] [192.168.181.191:37990]
received view <= current view; discarding it (current vid: [192.168.181.190:37863|3],
new vid: [192.168.181.190:37863|3])
2007-11-01 08:39:58,627 DEBUG [org.jgroups.protocols.pbcast.CoordGmsImpl] cancelling merge
(merge_id=[192.168.181.190:37863|1193927988632])
2007-11-01 08:40:01,234 DEBUG [org.jboss.cache.eviction.BaseEvictionAlgorithm] Adding
element /SessionKeyIndex/IP_ADDR/70.1.2.3 for a node that doesn't exist yet. Process
as an add.
2007-11-01 08:40:01,235 DEBUG [org.jboss.cache.eviction.BaseEvictionAlgorithm] Adding
element /SessionKeyIndex/DEVICE_IP_LIST/10.10.10.10 for a node that doesn't exist yet.
Process as an add.
2007-11-01 08:40:01,235 DEBUG [org.jboss.cache.eviction.BaseEvictionAlgorithm] Adding
element /SessionKeyIndex/USER_NAME/testuser2 for a node that doesn't exist yet.
Process as an add.
2007-11-01 08:40:01,235 DEBUG [org.jboss.cache.eviction.BaseEvictionAlgorithm] Adding
element /SessionKeyIndex/SERVICE_LIST/internet2 for a node that doesn't exist yet.
Process as an add.
2007-11-01 08:40:01,235 DEBUG [org.jboss.cache.eviction.BaseEvictionAlgorithm] Adding
element /Sessions/42E2E6FCD2EC4345 for a node that doesn't exist yet. Process as an
add.
2007-11-01 08:40:01,235 DEBUG [org.jboss.cache.eviction.BaseEvictionAlgorithm] Adding
element /SessionKeys/42E2E6FCD2EC4345 for a node that doesn't exist yet. Process as an
add.
View the original post :
http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4101073#...
Reply to the post :
http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&a...