9:39 a.m.
We are seeing an issue that I'd like to confirm with others before I write it up as a
jira issue.
JBoss 5.1.0.
JBoss Cache 3.2.1
Buddy Replication (two nodes, separate machines)
We're currently using Total Replication for HTTP Session replication but we're
increasing our node count (to three for now) and we'd like to change from total, to
buddy replication. After a significant amount of testing, buddy replication works very
well, except in one case for us.
The problem:
Two servers, A and B.
- Start server A
- Log into web app, creating an HTTP Session, S1, on server A
- Start server B. When B starts (this first time), the session S1 is perfectly replicated
from A to B into B's "_BUDDY_BACKUP_" for server A
- Now restart server B, upon starting the second time, the session S1 from server A is
only partially replicated to the "_BUDDY_BACKUP_" tree on server B. In
particular it appears that (at least) the "DistributableSessionMetadata" was not
replicated (should have a key of "2" in the replicated cache entry)
- Shut down server A, when the user hits server B, JBoss will try to unserialize and use
S1, however it will fail because some data is not present in S1, such as the session
metadata.
Example session S1 that successfully replicated after the initial startup of server B
--- Cache1 ---
| / {}
| /_BUDDY_BACKUP_ {}
| /192.168.71.60_7600 {}
| /JSESSION {}
| /ROOT_localhost {}
| /TYqa9j30si-QlGaVL9OVvQ__ {0=16, 1=1255098672537,
org.jboss.seam.security.rememberMe=org.jboss.ha.framework.server.SimpleCachableMarshalledValue{raw=nullserialized=true},
2=org.jboss.web.tomcat.service.session.distributedcache.spi.DistributableSessionMetadata@c1e908b,
org.jboss.seam.CONVERSATION#1$org.jboss.seam.persistence.persistenceContexts=org.jboss.ha.framework.server.SimpleCachableMarshalledValue{raw=nullserialized=true},
org.jboss.seam.core.conversationEntries=org.jboss.ha.framework.server.SimpleCachableMarshalledValue{raw=nullserialized=true},
org.jboss.seam.international.localeSelector=org.jboss.ha.framework.server.SimpleCachableMarshalledValue{raw=nullserialized=true},
org.jboss.seam.CONVERSATION#1$org.jboss.seam.core.conversation=org.jboss.ha.framework.server.SimpleCachableMarshalledValue{raw=nullserialized=true},
org.jboss.seam.CONVERSATION#1$org.jboss.seam.international.statusMessages=org.jboss.ha.framework.server.SimpleCachableMarshalledValue{raw=nullserialized=true},
org.jboss.seam.security.identity=org.jboss.ha.framework.server.SimpleCachableMarshalledValue{raw=nullserialized=true},
javax.faces.request.charset=UTF-8,
org.jboss.seam.CONVERSATION#1$org.jboss.seam.faces.redirect=org.jboss.ha.framework.server.SimpleCachableMarshalledValue{raw=nullserialized=true},
org.jboss.seam.security.credentials=org.jboss.ha.framework.server.SimpleCachableMarshalledValue{raw=nullserialized=true},
pier=org.jboss.ha.framework.server.SimpleCachableMarshalledValue{raw=nullserialized=true},
org.jboss.seam.web.session=org.jboss.ha.framework.server.SimpleCachableMarshalledValue{raw=nullserialized=true}}
| ------------
|
Example session S1 that failed to replicate fully to B after restart (notice the missing
"2=..." among many other things)
--- Cache1 ---
| / {}
| /_BUDDY_BACKUP_ {}
| /192.168.71.60_7600 {}
| /JSESSION {}
| /ROOT_localhost {}
| /TYqa9j30si-QlGaVL9OVvQ__ {0=153, 1=1255097288878,
foo=org.jboss.ha.framework.server.SimpleCachableMarshalledValue{raw=nullserialized=true}}
| ------------
What appears to be happening is that during the initial startup of the secondary server,
server A properly calls the "AssignToBuddyGroupCommand" on server B, passing the
initial state. However on subsequent restarts of server B, that command is never
executed from server A.
I believe the problem is that server A fails to recognize the server B was shutdown, and
the BuddyManager never removes him as a buddy (at least not in the timespan of server B
restarting). When I look at the BuddyManager on server A in the jmx-console, I can see
that his buddy group is never updated when server B restarts. I believe the data that
does make it to B after a restart is just from the standard "replicate commands"
that occur when something changes in the session S1 on server A.
For example, if server B was restarted at 6:50am, server A's buddy information in JMX
looks like this (last updated 6:40am, which implies he never processed B leaving)
BuddyGroup: (dataOwner: 192.168.71.60:7600, groupName: 192.168.71.60_7600, buddies:
[192.168.71.62:7600],lastModified: Fri Oct 09 06:40:27 PDT 2009)
In conjunction with that, in the log output from server A, I see this when server B is
restarting:
2009-10-09 06:50:52,477 DEBUG [org.jboss.cache.buddyreplication.BuddyManager] Nothing has
changed; new buddy list is identical to the old one.
|
Here are my relevant configurations (let me know if I missed sections):
jboss-web.xml
| <replication-config>
|
<replication-trigger>SET_AND_NON_PRIMITIVE_GET</replication-trigger>
|
<replication-granularity>ATTRIBUTE</replication-granularity>
| </replication-config>
|
<bean name="StandardSessionCacheConfig"
class="org.jboss.cache.config.Configuration">
|
| <!-- Provides batching functionality for caches that don't want to
interact with regular JTA Transactions -->
| <property
name="transactionManagerLookupClass">org.jboss.cache.transaction.BatchModeTransactionManagerLookup</property>
|
| <!-- Name of cluster. Needs to be the same for all members -->
| <property
name="clusterName">${jboss.partition.name:DefaultPartition}-SessionCache</property>
| <!-- Use a UDP (multicast) based stack. Need JGroups flow control (FC)
| because we are using asynchronous replication. -->
| <property
name="multiplexerStack">${jboss.default.jgroups.stack:tcp}</property>
| <property name="fetchInMemoryState">true</property>
|
| <property
name="nodeLockingScheme">PESSIMISTIC</property>
| <property
name="isolationLevel">REPEATABLE_READ</property>
| <property name="useLockStriping">false</property>
| <property name="cacheMode">REPL_ASYNC</property>
|
| <!-- Number of milliseconds to wait until all responses for a
| synchronous call have been received. Make this longer
| than lockAcquisitionTimeout.-->
| <property name="syncReplTimeout">17500</property>
| <!-- Max number of milliseconds to wait for a lock acquisition -->
| <property
name="lockAcquisitionTimeout">15000</property>
| <!-- The max amount of time (in milliseconds) we wait until the
| state (ie. the contents of the cache) are retrieved from
| existing members at startup. -->
| <property
name="stateRetrievalTimeout">60000</property>
|
| <!-- Not needed for a web session cache that doesn't use FIELD -->
| <property
name="useRegionBasedMarshalling">false</property>
| <!-- Must match the value of "useRegionBasedMarshalling" -->
| <property name="inactiveOnStartup">false</property>
|
| <!-- Disable asynchronous RPC marshalling/sending -->
| <property
name="serializationExecutorPoolSize">0</property>
| <!-- We have no asynchronous notification listeners -->
| <property name="listenerAsyncPoolSize">0</property>
|
| <property
name="exposeManagementStatistics">true</property>
|
| <property name="buddyReplicationConfig">
| <bean
class="org.jboss.cache.config.BuddyReplicationConfig">
|
| <!-- Just set to true to turn on buddy replication -->
| <property name="enabled">true</property>
|
| <!-- A way to specify a preferred replication group. We try
| and pick a buddy who shares the same pool name (falling
| back to other buddies if not available). -->
| <property
name="buddyPoolName">default</property>
|
| <property
name="buddyCommunicationTimeout">17500</property>
|
| <!-- Do not change these -->
| <property
name="autoDataGravitation">false</property>
| <property
name="dataGravitationRemoveOnFind">true</property>
| <property
name="dataGravitationSearchBackupTrees">true</property>
|
| <property name="buddyLocatorConfig">
| <bean
class="org.jboss.cache.buddyreplication.NextMemberBuddyLocatorConfig">
| <!-- The number of backup copies we maintain -->
| <property name="numBuddies">1</property>
| <!-- Means that each node will *try* to select a buddy on
| a different physical host. If not able to do so
| though, it will fall back to colocated nodes. -->
| <property
name="ignoreColocatedBuddies">true</property>
| </bean>
| </property>
| </bean>
| </property>
| <property name="cacheLoaderConfig">
| <bean class="org.jboss.cache.config.CacheLoaderConfig">
| <!-- Do not change these -->
| <property name="passivation">true</property>
| <property name="shared">false</property>
|
| <property name="individualCacheLoaderConfigs">
| <list>
| <bean
class="org.jboss.cache.loader.FileCacheLoaderConfig">
| <!-- Where passivated sessions are stored -->
| <property
name="location">${jboss.server.data.dir}${/}session</property>
| <!-- Do not change these -->
| <property
name="async">false</property>
| <property
name="fetchPersistentState">true</property>
| <property
name="purgeOnStartup">true</property>
| <property
name="ignoreModifications">false</property>
| <property
name="checkCharacterPortability">false</property>
| </bean>
| </list>
| </property>
| </bean>
| </property>
| </bean>
I have tried UDP / TCP, passivation / no passivation, and confirmed that things again work
fine when using "total", not "buddy" replication.
Has anyone else seen this? Let me know if more information is needed.
View the original post :
http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4259659#...
Reply to the post :
http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&a...