[jboss-user] [Clustering/JBoss] - Errors on Cluster Hot Deploy or Instance Restart
joereger
do-not-reply at jboss.com
Wed Sep 26 21:17:27 EDT 2007
Hi all!
I'm running a webapp and it's successfully clustered in production using JBoss. First of all, that's awesome and thanks to all who contributed to building this tool.
We just have some issues with Tomcat and/or JBoss restarts. If we hot deploy a WAR file into JBoss or if we throw some new code into Tomcat's classes directory (on a dev server) that triggers an application restart we end up some issues.
First, we get the INITIAL ERROR included below. I'm not a master of my MBEAN universe yet. In fact, I've on Tomcat for years and having moved to JBoss am very new to the concept. Any interpretation of what's happening would be appreciated.
Next, we get the REPEATING ERROR which looks like a JGroups process gets stuck on while the rest of the cluster is off. This makes some sense in light of the initial error but I have no idea how to remedy the problem.
In addition to JBoss's built-in caches we're running a couple TreeCache instances in the app. One for Hibernate which we set up and configure... we don't use JBoss's built-in. Both configs included below. One for other stuff we want to cache. mcast_addr is the same for both but mcast_port is different.
By simply restarting Tomcat (the full service, not just the application) or JBoss the problem goes away. Seems like a stuck thread or something that doesn't get properly shut down when the app restarts.
Question: Is there anything special we should be doing when the application restarts to prevent this sort of thing? We have a ServletContextListener that implements contextDestroyed(ServletContextEvent cse) and could easily do some better/more graceful JGroups or MBEAN shutdown.
Apologies if this is a Hibernate-specific issue. The error appears to be from org.jgroups.protocols.UDP so I thought I'd ping you guys and girls first.
Thanks,
Joe
INITIAL ERROR=================
124125 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] WARN org.jboss.cache.TreeCache - Error in stop jboss.cache:service=Fbdblog-TreeCache-Cluster
javax.management.InstanceNotFoundException: jboss.system:service=ServiceController
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(Unknown Source)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke (Unknown Source)
at org.jboss.system.ServiceMBeanSupport.stop(ServiceMBeanSupport.java:189)
at org.hibernate.cache.TreeCacheProvider.stop(TreeCacheProvider.java:84)
at org.hibernate.impl.SessionFactoryImpl.close (SessionFactoryImpl.java:800)
at com.fbdblog.dao.hibernate.HibernateUtil.killSessionFactory(HibernateUtil.java:158)
at com.fbdblog.startup.ApplicationStartup.contextDestroyed(ApplicationStartup.java:95)
at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:3733)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4306)
at org.apache.catalina.core.StandardContext.reload (StandardContext.java:2947)
at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:403)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1276)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1568)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java :1577)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1577)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java :1557)
at java.lang.Thread.run(Unknown Source)
124125 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] WARN org.jboss.cache.TreeCache - Error in destroy jboss.cache:service=Fbdblog-TreeCache-Cluster
javax.management.InstanceNotFoundException: jboss.system:service=ServiceController
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(Unknown Source)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke (Unknown Source)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
at org.jboss.system.ServiceMBeanSupport.destroy(ServiceMBeanSupport.java:204)
at org.hibernate.cache.TreeCacheProvider.stop (TreeCacheProvider.java:85)
at org.hibernate.impl.SessionFactoryImpl.close(SessionFactoryImpl.java:800)
at com.fbdblog.dao.hibernate.HibernateUtil.killSessionFactory(HibernateUtil.java:158)
at com.fbdblog.startup.ApplicationStartup.contextDestroyed (ApplicationStartup.java:95)
at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:3733)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4306)
at org.apache.catalina.core.StandardContext.reload (StandardContext.java:2947)
at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:403)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1276)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1568)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java :1577)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1577)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java :1557)
at java.lang.Thread.run(Unknown Source)
124140 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO com.fbdblog.startup.ApplicationStartup - Unregistering MBean: jboss.cache:service=Fbdblog-TreeCache-Cluster ,treecache-interceptor=UnlockInterceptor
124140 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO com.fbdblog.startup.ApplicationStartup - Unregistering MBean: jboss.cache:service=Fbdblog-TreeCache-Cluster ,treecache-interceptor=TxInterceptor
124140 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO com.fbdblog.startup.ApplicationStartup - Unregistering MBean: jboss.cache:service=Fbdblog-TreeCache-Cluster,treecache-interceptor=CacheMgmtInterceptor
124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO com.fbdblog.startup.ApplicationStartup - Unregistering MBean: jboss.cache:service=Fbdblog-TreeCache-Cluster,treecache-interceptor=ReplicationInterceptor
124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO com.fbdblog.startup.ApplicationStartup - Unregistering MBean: jboss.cache:service=Fbdblog-TreeCache-Cluster,treecache-interceptor=PessimisticLockInterceptor
124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO com.fbdblog.startup.ApplicationStartup - Unregistering MBean: jboss.cache:service=Fbdblog-TreeCache-Cluster,treecache-interceptor=CallInterceptor
124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO com.fbdblog.startup.ApplicationStartup - Unregistering MBean: jboss.cache:service=Fbdblog-TreeCache-Cluster
124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] WARN org.jboss.cache.TreeCache - Unexpected error during removal. jboss.cache:service=Fbdblog-TreeCache-Cluster
javax.management.InstanceNotFoundException: jboss.system:service=ServiceController
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean (Unknown Source)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
at org.jboss.system.ServiceMBeanSupport.postDeregister (ServiceMBeanSupport.java:409)
at com.sun.jmx.mbeanserver.MBeanSupport.postDeregister(Unknown Source)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postDeregisterInvoke(Unknown Source)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean (Unknown Source)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(Unknown Source)
at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(Unknown Source)
at com.fbdblog.startup.ApplicationStartup.shutdownCacheMBean (ApplicationStartup.java:156)
at com.fbdblog.startup.ApplicationStartup.contextDestroyed(ApplicationStartup.java:98)
at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:3733)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4306)
at org.apache.catalina.core.StandardContext.reload(StandardContext.java:2947)
at org.apache.catalina.loader.WebappLoader.backgroundProcess (WebappLoader.java:403)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1276)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java :1568)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1577)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java :1577)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1557)
at java.lang.Thread.run(Unknown Source)
Fbdblog: Application shut down! (XPS)
REPEATING ERROR===============
49174016 [Incoming Thread,Fbdblog-TreeCache-Cluster, 192.168.1.103:1852] ERROR org.jgroups.protocols.UDP - failed handling incoming message
java.lang.NoSuchMethodError: java.lang.ThreadDeath: method (Ljava/lang/String;)V not found
at org.jgroups.protocols.UNICAST.handleDataReceived (UNICAST.java:539)
at org.jgroups.protocols.UNICAST.up(UNICAST.java:246)
at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:596)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:154)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:298)
at org.jgroups.protocols.MERGE2.up(MERGE2.java:145)
at org.jgroups.protocols.Discovery.up(Discovery.java:220)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage (TP.java:1542)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1496)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (Unknown Source)
at java.lang.Thread.run(Unknown Source)
CONFIG================
jboss:service=Naming
jboss:service=TransactionManager
org.jboss.cache.JBossTransactionManagerLookup
NONE
REPL_SYNC
true
1000
0
Fbdblog-TreeCache-Cluster
<UDP mcast_addr="228.1.2.3" mcast_port="41332"
ip_ttl="64" ip_mcast="true"
mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
loopback="true"/>
<PING timeout="2000" num_initial_members="3" up_thread="false" down_thread="false"/>
<MERGE2 min_interval="10000" max_interval="20000"/>
<!-- <FD shun="true" up_thread="true" down_thread="true" />-->
<FD_SOCK/>
<VERIFY_SUSPECT timeout="1500" up_thread="false" down_thread="false"/>
<pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800" max_xmit_size="8192" up_thread="false" down_thread="false"/>
<UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10" down_thread="false"/>
<pbcast.STABLE desired_avg_gossip="20000" up_thread="false" down_thread="false"/>
<FRAG frag_size="8192" down_thread="false" up_thread="false"/>
<pbcast.GMS join_timeout="60000" join_retry_timeout="2000" shun="true" print_local_addr="true"/>
<pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>
true
60000
15000
10000
org.jboss.cache.eviction.LRUPolicy
5
50000
0
false
CONFIG2====================
jboss:service=Naming
jboss:service=TransactionManager
org.jboss.cache.JBossTransactionManagerLookup
NONE
REPL_SYNC
true
1000
0
Fbdblog-TreeCache-Cluster-Nondao
<UDP mcast_addr=" 228.1.2.3" mcast_port="41333"
ip_ttl="64" ip_mcast="true"
mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
loopback="true"/>
<PING timeout="2000" num_initial_members="3" up_thread="false" down_thread="false"/>
<MERGE2 min_interval="10000" max_interval="20000"/>
<!-- <FD shun="true" up_thread="true" down_thread="true" />-->
<FD_SOCK/>
<VERIFY_SUSPECT timeout="1500" up_thread="false" down_thread="false"/>
<pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800" max_xmit_size="8192" up_thread="false" down_thread="false"/>
<UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10" down_thread="false"/>
<pbcast.STABLE desired_avg_gossip="20000" up_thread="false" down_thread="false"/>
<FRAG frag_size="8192" down_thread="false" up_thread="false"/>
<pbcast.GMS join_timeout="60000" join_retry_timeout="2000" shun="true" print_local_addr="true"/>
<pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>
true
60000
15000
10000
org.jboss.cache.eviction.LRUPolicy
5
50000
0
false
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4089136#4089136
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4089136
More information about the jboss-user
mailing list