8:24 p.m.
Hi all!
I'm running a webapp and it's successfully clustered in production using JBoss.
First of all, that's awesome and thanks to all who contributed to building this tool.
We just have some issues with Tomcat and/or JBoss restarts. If we hot deploy a WAR file
into JBoss or if we throw some new code into Tomcat's classes directory (on a dev
server) that triggers an application restart we end up some issues.
First, we get the INITIAL ERROR included below. I'm not a master of my MBEAN universe
yet. In fact, I've on Tomcat for years and having moved to JBoss am very new to the
concept. Any interpretation of what's happening would be appreciated.
Next, we get the REPEATING ERROR which looks like a JGroups process gets stuck on while
the rest of the cluster is off. This makes some sense in light of the initial error but I
have no idea how to remedy the problem.
In addition to JBoss's built-in caches we're running a couple TreeCache instances
in the app. One for Hibernate which we set up and configure... we don't use
JBoss's built-in. Both configs included below. One for other stuff we want to cache.
mcast_addr is the same for both but mcast_port is different.
Bela Ban recommended changing the mbean name for each config. We played with that and the
cluster worked perfectly until we deployed something and the app got restarted.
By simply restarting Tomcat (the full service, not just the application) or JBoss the
problem goes away. Seems like a stuck thread or something that doesn't get properly
shut down when the app restarts.
Question: Is there anything special we should be doing when the application restarts to
prevent this sort of thing? We have a ServletContextListener that implements
contextDestroyed(ServletContextEvent cse) and could easily do some better/more graceful
JGroups or MBEAN shutdown.
Apologies if this is a Hibernate-specific issue. The error appears to be from
org.jgroups.protocols.UDP so I thought I'd ping you guys and girls first.
Thanks,
Joe
INITIAL ERROR=================
124125 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] WARN
org.jboss.cache.TreeCache - Error in stop jboss.cache:service=Fbdblog-TreeCache-Cluster
javax.management.InstanceNotFoundException: jboss.system:service=ServiceController
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(Unknown Source)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke (Unknown Source)
at org.jboss.system.ServiceMBeanSupport.stop(ServiceMBeanSupport.java:189)
at org.hibernate.cache.TreeCacheProvider.stop(TreeCacheProvider.java:84)
at org.hibernate.impl.SessionFactoryImpl.close (SessionFactoryImpl.java:800)
at com.fbdblog.dao.hibernate.HibernateUtil.killSessionFactory(HibernateUtil.java:158)
at com.fbdblog.startup.ApplicationStartup.contextDestroyed(ApplicationStartup.java:95)
at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:3733)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4306)
at org.apache.catalina.core.StandardContext.reload (StandardContext.java:2947)
at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:403)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1276)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1568)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java
:1577)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1577)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java
:1557)
at java.lang.Thread.run(Unknown Source)
124125 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] WARN
org.jboss.cache.TreeCache - Error in destroy
jboss.cache:service=Fbdblog-TreeCache-Cluster
javax.management.InstanceNotFoundException: jboss.system:service=ServiceController
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(Unknown Source)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke (Unknown Source)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
at org.jboss.system.ServiceMBeanSupport.destroy(ServiceMBeanSupport.java:204)
at org.hibernate.cache.TreeCacheProvider.stop (TreeCacheProvider.java:85)
at org.hibernate.impl.SessionFactoryImpl.close(SessionFactoryImpl.java:800)
at com.fbdblog.dao.hibernate.HibernateUtil.killSessionFactory(HibernateUtil.java:158)
at com.fbdblog.startup.ApplicationStartup.contextDestroyed (ApplicationStartup.java:95)
at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:3733)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4306)
at org.apache.catalina.core.StandardContext.reload (StandardContext.java:2947)
at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:403)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1276)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1568)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java
:1577)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1577)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java
:1557)
at java.lang.Thread.run(Unknown Source)
124140 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO
com.fbdblog.startup.ApplicationStartup - Unregistering MBean:
jboss.cache:service=Fbdblog-TreeCache-Cluster ,treecache-interceptor=UnlockInterceptor
124140 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO
com.fbdblog.startup.ApplicationStartup - Unregistering MBean:
jboss.cache:service=Fbdblog-TreeCache-Cluster ,treecache-interceptor=TxInterceptor
124140 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO
com.fbdblog.startup.ApplicationStartup - Unregistering MBean:
jboss.cache:service=Fbdblog-TreeCache-Cluster,treecache-interceptor=CacheMgmtInterceptor
124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO
com.fbdblog.startup.ApplicationStartup - Unregistering MBean:
jboss.cache:service=Fbdblog-TreeCache-Cluster,treecache-interceptor=ReplicationInterceptor
124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO
com.fbdblog.startup.ApplicationStartup - Unregistering MBean:
jboss.cache:service=Fbdblog-TreeCache-Cluster,treecache-interceptor=PessimisticLockInterceptor
124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO
com.fbdblog.startup.ApplicationStartup - Unregistering MBean:
jboss.cache:service=Fbdblog-TreeCache-Cluster,treecache-interceptor=CallInterceptor
124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO
com.fbdblog.startup.ApplicationStartup - Unregistering MBean:
jboss.cache:service=Fbdblog-TreeCache-Cluster
124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] WARN
org.jboss.cache.TreeCache - Unexpected error during removal.
jboss.cache:service=Fbdblog-TreeCache-Cluster
javax.management.InstanceNotFoundException: jboss.system:service=ServiceController
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean (Unknown Source)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
at org.jboss.system.ServiceMBeanSupport.postDeregister (ServiceMBeanSupport.java:409)
at com.sun.jmx.mbeanserver.MBeanSupport.postDeregister(Unknown Source)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postDeregisterInvoke(Unknown
Source)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean (Unknown
Source)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(Unknown Source)
at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(Unknown Source)
at com.fbdblog.startup.ApplicationStartup.shutdownCacheMBean
(ApplicationStartup.java:156)
at com.fbdblog.startup.ApplicationStartup.contextDestroyed(ApplicationStartup.java:98)
at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:3733)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4306)
at org.apache.catalina.core.StandardContext.reload(StandardContext.java:2947)
at org.apache.catalina.loader.WebappLoader.backgroundProcess (WebappLoader.java:403)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1276)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java
:1568)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1577)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java
:1577)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1557)
at java.lang.Thread.run(Unknown Source)
Fbdblog: Application shut down! (XPS)
REPEATING ERROR===============
49174016 [Incoming Thread,Fbdblog-TreeCache-Cluster, 192.168.1.103:1852] ERROR
org.jgroups.protocols.UDP - failed handling incoming message
java.lang.NoSuchMethodError: java.lang.ThreadDeath: method (Ljava/lang/String;)V not
found
at org.jgroups.protocols.UNICAST.handleDataReceived (UNICAST.java:539)
at org.jgroups.protocols.UNICAST.up(UNICAST.java:246)
at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:596)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:154)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:298)
at org.jgroups.protocols.MERGE2.up(MERGE2.java:145)
at org.jgroups.protocols.Discovery.up(Discovery.java:220)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage (TP.java:1542)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1496)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (Unknown Source)
at java.lang.Thread.run(Unknown Source)
CONFIG================
<server>
| <classpath codebase="./lib" archives="jboss-cache.jar,
jgroups-all.jar"/>
| <mbean code="org.jboss.cache.TreeCache"
| name="jboss.cache:service=TreeCache">
| <depends>jboss:service=Naming</depends>
| <depends>jboss:service=TransactionManager</depends>
| <attribute
name="TransactionManagerLookupClass">org.jboss.cache.JBossTransactionManagerLookup</attribute>
| <attribute name="IsolationLevel">NONE</attribute>
| <attribute name="CacheMode">REPL_SYNC</attribute>
| <attribute name="UseReplQueue">true</attribute>
| <attribute name="ReplQueueInterval">1000</attribute>
| <attribute name="ReplQueueMaxElements">0</attribute>
| <attribute
name="ClusterName">Fbdblog-TreeCache-Cluster</attribute>
| <attribute name="ClusterConfig">
| <config>
| <UDP mcast_addr="228.1.2.3" mcast_port="41332"
| ip_ttl="64" ip_mcast="true"
| mcast_send_buf_size="150000"
mcast_recv_buf_size="80000"
| ucast_send_buf_size="150000"
ucast_recv_buf_size="80000"
| loopback="true"/>
| <PING timeout="2000" num_initial_members="3"
up_thread="false" down_thread="false"/>
| <MERGE2 min_interval="10000"
max_interval="20000"/>
| <!-- <FD shun="true" up_thread="true"
down_thread="true" />-->
| <FD_SOCK/>
| <VERIFY_SUSPECT timeout="1500"
up_thread="false" down_thread="false"/>
| <pbcast.NAKACK gc_lag="50"
retransmit_timeout="600,1200,2400,4800" max_xmit_size="8192"
up_thread="false" down_thread="false"/>
| <UNICAST timeout="600,1200,2400"
window_size="100" min_threshold="10"
down_thread="false"/>
| <pbcast.STABLE desired_avg_gossip="20000"
up_thread="false" down_thread="false"/>
| <FRAG frag_size="8192" down_thread="false"
up_thread="false"/>
| <pbcast.GMS join_timeout="60000"
join_retry_timeout="2000" shun="true"
print_local_addr="true"/>
| <pbcast.STATE_TRANSFER up_thread="true"
down_thread="true"/>
| </config>
| </attribute>
| <attribute name="FetchStateOnStartup">true</attribute>
| <attribute
name="InitialStateRetrievalTimeout">60000</attribute>
| <attribute name="SyncReplTimeout">15000</attribute>
| <attribute
name="LockAcquisitionTimeout">10000</attribute>
| <attribute name="EvictionPolicyClass">
org.jboss.cache.eviction.LRUPolicy</attribute>
| <attribute name="EvictionPolicyConfig">
| <config>
| <attribute
name="wakeUpIntervalSeconds">5</attribute>
| <region name="/_default_">
| <attribute name="maxNodes">50000</attribute>
| <attribute
name="timeToLiveSeconds">0</attribute>
| </region>
| </config>
| </attribute>
| <attribute name="UseMarshalling">false</attribute>
| </mbean>
| </server>
CONFIG2====================
<server>
| <classpath codebase="./lib" archives="jboss-cache.jar,
jgroups-all.jar"/>
| <mbean code="org.jboss.cache.TreeCache"
| name="jboss.cache:service=TreeCache">
| <depends>jboss:service=Naming</depends>
| <depends>jboss:service=TransactionManager</depends>
| <attribute name="TransactionManagerLookupClass">
org.jboss.cache.JBossTransactionManagerLookup</attribute>
| <attribute name="IsolationLevel">NONE</attribute>
| <attribute name="CacheMode">REPL_SYNC</attribute>
| <attribute name="UseReplQueue">true</attribute>
| <attribute name="ReplQueueInterval">1000</attribute>
| <attribute name="ReplQueueMaxElements">0</attribute>
| <attribute
name="ClusterName">Fbdblog-TreeCache-Cluster-Nondao</attribute>
| <attribute name="ClusterConfig">
| <config>
| <UDP mcast_addr=" 228.1.2.3"
mcast_port="41333"
| ip_ttl="64" ip_mcast="true"
| mcast_send_buf_size="150000"
mcast_recv_buf_size="80000"
| ucast_send_buf_size="150000"
ucast_recv_buf_size="80000"
| loopback="true"/>
| <PING timeout="2000" num_initial_members="3"
up_thread="false" down_thread="false"/>
| <MERGE2 min_interval="10000"
max_interval="20000"/>
| <!-- <FD shun="true" up_thread="true"
down_thread="true" />-->
| <FD_SOCK/>
| <VERIFY_SUSPECT timeout="1500"
up_thread="false" down_thread="false"/>
| <pbcast.NAKACK gc_lag="50"
retransmit_timeout="600,1200,2400,4800" max_xmit_size="8192"
up_thread="false" down_thread="false"/>
| <UNICAST timeout="600,1200,2400"
window_size="100" min_threshold="10"
down_thread="false"/>
| <pbcast.STABLE desired_avg_gossip="20000"
up_thread="false" down_thread="false"/>
| <FRAG frag_size="8192" down_thread="false"
up_thread="false"/>
| <pbcast.GMS join_timeout="60000"
join_retry_timeout="2000" shun="true"
print_local_addr="true"/>
| <pbcast.STATE_TRANSFER up_thread="true"
down_thread="true"/>
| </config>
| </attribute>
| <attribute name="FetchStateOnStartup">true</attribute>
| <attribute
name="InitialStateRetrievalTimeout">60000</attribute>
| <attribute name="SyncReplTimeout">15000</attribute>
| <attribute
name="LockAcquisitionTimeout">10000</attribute>
| <attribute
name="EvictionPolicyClass">org.jboss.cache.eviction.LRUPolicy</attribute>
| <attribute name="EvictionPolicyConfig">
| <config>
| <attribute
name="wakeUpIntervalSeconds">5</attribute>
| <region name="/_default_">
| <attribute name="maxNodes">50000</attribute>
| <attribute
name="timeToLiveSeconds">0</attribute>
| </region>
| </config>
| </attribute>
| <attribute name="UseMarshalling">false</attribute>
| </mbean>
| </server>
View the original post :
http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4089137#...
Reply to the post :
http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&a...