[jboss-user] [Clustering] - Failure when cleaning up HTTP Session in JBoss AS Cluster?

[RichardTaylor]

Let me know if this should be posted somewhere else.

JBoss 5.1.0
JBoss Cache 3.2.1
Three servers in cluster, HTTP Session replication, UDP, total replication, passivation enabled

We recently rolled out two changes to our system, HTTP Session passivation and an upgrade to JBoss Cache 3.2.1.  The changes were tested for several days under load on two other systems, but now that we've deployed to our primary data center we've run into an issue.

Things ran fine for about 22 hours, then server A and server B went into a loop where each server getting the following two stack traces repeatedly.  This happened twice in 24 hours.  We've not disabled session passivation and have not seen the issue.

In order to get the servers out of the loop we had to restart jboss on both servers (one of them twice).  JBoss was still responding but the cluster was in an obviously degraded state.

2009-10-20 15:40:29,709 ERROR [org.jboss.web.tomcat.service.session.distributedcache.impl.jbc.AttributeBasedJBossCacheService] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) IOException occurred unmarshalling value 
  | java.io.OptionalDataException
  | 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
  | 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
  | 	at javolution.util.FastMap.readObject(Unknown Source)
  | 	at sun.reflect.GeneratedMethodAccessor362.invoke(Unknown Source)
  | 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  | 	at java.lang.reflect.Method.invoke(Method.java:597)
  | 	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
  | 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
  | 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
  | 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
  | 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
  | 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
  | 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
  | 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
  | 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
  | 	at org.jboss.ha.framework.server.SimpleCachableMarshalledValue.get(SimpleCachableMarshalledValue.java:94)
  | 	at org.jboss.web.tomcat.service.session.distributedcache.impl.jbc.AbstractJBossCacheService.getUnMarshalledValue(AbstractJBossCacheService.java:660)
  | 	at org.jboss.web.tomcat.service.session.distributedcache.impl.jbc.AttributeBasedJBossCacheService.getSessionAttributes(AttributeBasedJBossCacheService.java:166)
  | 	at org.jboss.web.tomcat.service.session.distributedcache.impl.jbc.AbstractJBossCacheService.getDistributableSessionData(AbstractJBossCacheService.java:581)
  | 	at org.jboss.web.tomcat.service.session.distributedcache.impl.jbc.AbstractJBossCacheService.getSessionData(AbstractJBossCacheService.java:364)
  | 	at org.jboss.web.tomcat.service.session.JBossCacheManager.loadSession(JBossCacheManager.java:1832)
  | 	at org.jboss.web.tomcat.service.session.JBossCacheManager.findSession(JBossCacheManager.java:489)
  | 	at org.jboss.web.tomcat.service.session.JBossCacheManager.processExpirationPassivation(JBossCacheManager.java:1603)
  | 	at org.jboss.web.tomcat.service.session.JBossManager.backgroundProcess(JBossManager.java:385)
  | 	at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1327)
  | 	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1612)
  | 	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1621)
  | 	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1621)
  | 	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1601)
  | 	at java.lang.Thread.run(Thread.java:619)
  | 
  | 2009-10-20 15:40:29,713 ERROR [org.jboss.web.tomcat.service.session.JBossCacheManager.] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) processExpirationPassivation(): failed handling unloaded session O-lgMwkUSLW1vVpTRQzExg__
  | java.lang.NullPointerException
  | 	at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
  | 	at java.util.concurrent.ConcurrentHashMap.putAll(ConcurrentHashMap.java:909)
  | 	at org.jboss.web.tomcat.service.session.ClusteredSession.populateAttributes(ClusteredSession.java:1661)
  | 	at org.jboss.web.tomcat.service.session.ClusteredSession.update(ClusteredSession.java:1120)
  | 	at org.jboss.web.tomcat.service.session.JBossCacheManager.loadSession(JBossCacheManager.java:1835)
  | 	at org.jboss.web.tomcat.service.session.JBossCacheManager.findSession(JBossCacheManager.java:489)
  | 	at org.jboss.web.tomcat.service.session.JBossCacheManager.processExpirationPassivation(JBossCacheManager.java:1603)
  | 	at org.jboss.web.tomcat.service.session.JBossManager.backgroundProcess(JBossManager.java:385)
  | 	at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1327)
  | 	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1612)
  | 	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1621)
  | 	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1621)
  | 	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1601)
  | 	at java.lang.Thread.run(Thread.java:619)
  | 

It appears that both servers were trying to clean up the exact same "unloaded" session at the same time.  That might be normal, but something in the session data seemed to have gotten corrupted?  It's not clear yet, I'm currently looking at the JBossCacheManager code.

Relevant configs:
>From jboss-web.xml

  | <passivation-config>
  | 	<use-session-passivation>true</use-session-passivation>
  | 	<passivation-min-idle-time>-1</passivation-min-idle-time>
  | 	<passivation-max-idle-time>300</passivation-max-idle-time>
  | </passivation-config>
  | 

>From jboss-cache-manager-jboss-beans.xml
<property name="cacheLoaderConfig">
  |             <bean class="org.jboss.cache.config.CacheLoaderConfig">
  |                    <!-- Do not change these -->
  |                    <property name="passivation">true</property>
  |                    <property name="shared">false</property>
  | 
  |                    <property name="individualCacheLoaderConfigs">
  |                      <list>
  |                         <bean class="org.jboss.cache.loader.FileCacheLoaderConfig">
  |                            <!-- Where passivated sessions are stored -->
  |                            <property name="location">${jboss.server.data.dir}${/}session</property>
  |                            <!-- Do not change these -->
  |                            <property name="async">false</property>
  |                            <property name="fetchPersistentState">true</property>
  |                            <property name="purgeOnStartup">true</property>
  |                            <property name="ignoreModifications">false</property>
  |                            <property name="checkCharacterPortability">false</property>
  |                         </bean>
  |                      </list>
  |                    </property>
  |             </bean>
  |          </property>
  | 


Has anyone seen this issue?  I see other people referencing java.io.OptinalDataException in the context of JBoss, but nothing that sounds like this exact same issue.   Any input would be greatly appreciated, we'd really like to get HTTP session passivation enabled again.



View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4261558#4261558

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4261558