[jboss-user] [JBoss Cache: Core Edition] - "Response timed out" exception in TreeCache
guruk
do-not-reply at jboss.com
Fri Sep 26 06:23:08 EDT 2008
Hi, folks.
On our environment occasionally were observed numerous org.jboss.cache.lock.TimeoutException's that cause transactions to fail. Moreover the problem seems to progress over time resulting in a system becoming non-operational with restart required.
It was suggested to try to increase SynReplTimeOut up to 100 sec and set loopback attribute to true (something that is recommended on windows machines). As issue reproduces occasionally it is hard to verify that these changes will help.
Any thoughts regarding this problem are appreciated.
We use:
JBossAS - 4.2.2.GA
The JBossCache - 1.4.1.SP5
JGroups - 2.4.1-SP4
The config
<mbean code="org.jboss.cache.TreeCache" name="jboss.cache:service=HibernateTreeCache">
| <depends>jboss:service=Naming</depends>
| <depends>jboss:service=TransactionManager</depends>
| <attribute name="TransactionManagerLookupClass">org.jboss.cache.JBossTransactionManagerLookup</attribute>
| <attribute name="IsolationLevel">REPEATABLE_READ</attribute>
| <attribute name="CacheMode">REPL_SYNC</attribute>
| <attribute name="ClusterName">DEFAULT</attribute>
|
| <attribute name="ClusterConfig">
| <config>
| <UDP mcast_addr="231.12.21.132" mcast_port="45047"
| ip_ttl="64" ip_mcast="true"
| mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
| ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
| loopback="false"/>
| <PING timeout="2000" num_initial_members="3"
| up_thread="false" down_thread="false"/>
| <MERGE2 min_interval="10000" max_interval="20000"/>
| <FD shun="true" up_thread="true" down_thread="true"/>
| <VERIFY_SUSPECT timeout="1500"
| up_thread="false" down_thread="false"/>
| <pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800"
| up_thread="false" down_thread="false"/>
| <pbcast.STABLE desired_avg_gossip="20000"
| up_thread="false" down_thread="false"/>
| <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10"
| down_thread="false"/>
| <FRAG frag_size="8192"
| down_thread="false" up_thread="false"/>
| <pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
| shun="true" print_local_addr="true"/>
| <pbcast.STATE_TRANSFER up_thread="false" down_thread="false"/>
| </config>
| </attribute>
| <attribute name="InitialStateRetrievalTimeout">20000</attribute>
| <attribute name="SyncReplTimeout">10000</attribute>
| <attribute name="LockAcquisitionTimeout">15000</attribute>
| <attribute name="EvictionPolicyClass"></attribute>
| </mbean>
the stack trace:
2008-09-15 04:44:32,548 WARN [http-0.0.0.0-80-2] [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator_2] TwoPhaseCoordinator.beforeCompletion - failed for com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple at f09a652
| java.lang.RuntimeException:
| at org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1182)
| at org.jboss.cache.interceptors.OrderedSynchronizationHandler.beforeCompletion(OrderedSynchronizationHandler.java:75)
| at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:114)
| at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:247)
| at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:86)
| at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:177)
| at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1389)
| at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:135)
| at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:87)
| at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.commit(ServerVMClientUserTransaction.java:140)
| at com.enkata.common.transaction.TransactionManager.commit(TransactionManager.java:75)
| at com.enkata.common.transaction.TransactionManagerHelper.commit(TransactionManagerHelper.java:43)
| at com.enkata.common.transaction.ATransaction.run(ATransaction.java:91)
| at com.enkata.common.transaction.NonCancellableTransaction.run(NonCancellableTransaction.java:33)
| at com.enkata.common.transaction.ReturnObjectTransaction.executeTransaction(ReturnObjectTransaction.java:43)
| at com.enkata.eac.to.analysis.AnalysisUIFacade.drill(AnalysisUIFacade.java:171)
| at com.enkata.eac.ui.analyst.drillpath.DrillPathNavigationCmd.doRun(DrillPathNavigationCmd.java:72)
| at com.enkata.eac.ui.util.ACmd.run(ACmd.java:70)
| at com.enkata.eac.ui.framework.Page.runACmd(Page.java:746)
| at com.enkata.eac.ui.framework.Page.runCmdAndAdjustScrollTop(Page.java:209)
| at com.enkata.eac.ui.framework.Page.validateAndUpdate(Page.java:192)
| at com.enkata.eac.ui.framework.Controller.validateAndUpdate(Controller.java:373)
| at com.enkata.eac.ui.framework.Controller.process2(Controller.java:270)
| at com.enkata.eac.ui.framework.Controller.process(Controller.java:250)
| at com.enkata.eac.ui.framework.UIMgr.processRequest(UIMgr.java:133)
| at com.enkata.eac.ui.servlet.EACServlet.processRequest(EACServlet.java:91)
| at com.enkata.eac.ui.servlet.EACServlet.doPost(EACServlet.java:67)
| at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
| at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
| at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
| at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
| at com.enkata.eac.ui.servlet.DownloadExportFilter.doFilter(DownloadExportFilter.java:41)
| at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
| at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
| at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
| at org.jboss.seam.web.MultipartFilter.doFilter(MultipartFilter.java:85)
| at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
| at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64)
| at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
| at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
| at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
| at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:147)
| at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:276)
| at org.ajax4jsf.Filter.doFilter(Filter.java:175)
| at org.jboss.seam.web.Ajax4jsfFilter.doFilter(Ajax4jsfFilter.java:60)
| at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
| at org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:58)
| at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
| at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)
| at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
| at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
| at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
| at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
| at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
| at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
| at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
| at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
| at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
| at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
| at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
| at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
| at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
| at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
| at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
| at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
| at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
| at java.lang.Thread.run(Unknown Source)
| Caused by: org.jboss.cache.ReplicationException: rsp=sender=172.16.100.17:3216, retval=null, received=false, suspected=false
| at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4436)
| at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4358)
| at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4469)
| at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:110)
| at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:88)
| at org.jboss.cache.interceptors.ReplicationInterceptor.runPreparePhase(ReplicationInterceptor.java:152)
| at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:69)
| at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
| at org.jboss.cache.interceptors.TxInterceptor.runPreparePhase(TxInterceptor.java:873)
| at org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1156)
| ... 66 more
| Caused by: org.jboss.cache.lock.TimeoutException: Response timed out: sender=172.16.100.17:3216, retval=null, received=false, suspected=false
| at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4434)
| ... 75 more
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4179030#4179030
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4179030
More information about the jboss-user
mailing list