5:23 a.m.
Hi, folks.
On our environment occasionally were observed numerous
org.jboss.cache.lock.TimeoutException's that cause transactions to fail. Moreover the
problem seems to progress over time resulting in a system becoming non-operational with
restart required.
It was suggested to try to increase SynReplTimeOut up to 100 sec and set loopback
attribute to true (something that is recommended on windows machines). As issue reproduces
occasionally it is hard to verify that these changes will help.
Any thoughts regarding this problem are appreciated.
We use:
JBossAS - 4.2.2.GA
The JBossCache - 1.4.1.SP5
JGroups - 2.4.1-SP4
The config
<mbean code="org.jboss.cache.TreeCache"
name="jboss.cache:service=HibernateTreeCache">
| <depends>jboss:service=Naming</depends>
| <depends>jboss:service=TransactionManager</depends>
| <attribute
name="TransactionManagerLookupClass">org.jboss.cache.JBossTransactionManagerLookup</attribute>
| <attribute
name="IsolationLevel">REPEATABLE_READ</attribute>
| <attribute name="CacheMode">REPL_SYNC</attribute>
| <attribute name="ClusterName">DEFAULT</attribute>
|
| <attribute name="ClusterConfig">
| <config>
| <UDP mcast_addr="231.12.21.132"
mcast_port="45047"
| ip_ttl="64" ip_mcast="true"
| mcast_send_buf_size="150000"
mcast_recv_buf_size="80000"
| ucast_send_buf_size="150000"
ucast_recv_buf_size="80000"
| loopback="false"/>
| <PING timeout="2000" num_initial_members="3"
| up_thread="false" down_thread="false"/>
| <MERGE2 min_interval="10000"
max_interval="20000"/>
| <FD shun="true" up_thread="true"
down_thread="true"/>
| <VERIFY_SUSPECT timeout="1500"
| up_thread="false" down_thread="false"/>
| <pbcast.NAKACK gc_lag="50"
retransmit_timeout="600,1200,2400,4800"
| up_thread="false" down_thread="false"/>
| <pbcast.STABLE desired_avg_gossip="20000"
| up_thread="false" down_thread="false"/>
| <UNICAST timeout="600,1200,2400"
window_size="100" min_threshold="10"
| down_thread="false"/>
| <FRAG frag_size="8192"
| down_thread="false" up_thread="false"/>
| <pbcast.GMS join_timeout="5000"
join_retry_timeout="2000"
| shun="true" print_local_addr="true"/>
| <pbcast.STATE_TRANSFER up_thread="false"
down_thread="false"/>
| </config>
| </attribute>
| <attribute
name="InitialStateRetrievalTimeout">20000</attribute>
| <attribute name="SyncReplTimeout">10000</attribute>
| <attribute
name="LockAcquisitionTimeout">15000</attribute>
| <attribute name="EvictionPolicyClass"></attribute>
| </mbean>
the stack trace:
2008-09-15 04:44:32,548 WARN [http-0.0.0.0-80-2]
[com.arjuna.ats.arjuna.logging.arjLoggerI18N]
[com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator_2]
TwoPhaseCoordinator.beforeCompletion - failed for
com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple@f09a652
| java.lang.RuntimeException:
| at
org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1182)
| at
org.jboss.cache.interceptors.OrderedSynchronizationHandler.beforeCompletion(OrderedSynchronizationHandler.java:75)
| at
com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:114)
| at
com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:247)
| at
com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:86)
| at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:177)
| at
com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1389)
| at
com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:135)
| at
com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:87)
| at
org.jboss.tm.usertx.client.ServerVMClientUserTransaction.commit(ServerVMClientUserTransaction.java:140)
| at
com.enkata.common.transaction.TransactionManager.commit(TransactionManager.java:75)
| at
com.enkata.common.transaction.TransactionManagerHelper.commit(TransactionManagerHelper.java:43)
| at com.enkata.common.transaction.ATransaction.run(ATransaction.java:91)
| at
com.enkata.common.transaction.NonCancellableTransaction.run(NonCancellableTransaction.java:33)
| at
com.enkata.common.transaction.ReturnObjectTransaction.executeTransaction(ReturnObjectTransaction.java:43)
| at com.enkata.eac.to.analysis.AnalysisUIFacade.drill(AnalysisUIFacade.java:171)
| at
com.enkata.eac.ui.analyst.drillpath.DrillPathNavigationCmd.doRun(DrillPathNavigationCmd.java:72)
| at com.enkata.eac.ui.util.ACmd.run(ACmd.java:70)
| at com.enkata.eac.ui.framework.Page.runACmd(Page.java:746)
| at com.enkata.eac.ui.framework.Page.runCmdAndAdjustScrollTop(Page.java:209)
| at com.enkata.eac.ui.framework.Page.validateAndUpdate(Page.java:192)
| at com.enkata.eac.ui.framework.Controller.validateAndUpdate(Controller.java:373)
| at com.enkata.eac.ui.framework.Controller.process2(Controller.java:270)
| at com.enkata.eac.ui.framework.Controller.process(Controller.java:250)
| at com.enkata.eac.ui.framework.UIMgr.processRequest(UIMgr.java:133)
| at com.enkata.eac.ui.servlet.EACServlet.processRequest(EACServlet.java:91)
| at com.enkata.eac.ui.servlet.EACServlet.doPost(EACServlet.java:67)
| at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
| at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
| at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
| at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
| at
com.enkata.eac.ui.servlet.DownloadExportFilter.doFilter(DownloadExportFilter.java:41)
| at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
| at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
| at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
| at org.jboss.seam.web.MultipartFilter.doFilter(MultipartFilter.java:85)
| at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
| at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64)
| at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
| at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
| at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
| at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:147)
| at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:276)
| at org.ajax4jsf.Filter.doFilter(Filter.java:175)
| at org.jboss.seam.web.Ajax4jsfFilter.doFilter(Ajax4jsfFilter.java:60)
| at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
| at org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:58)
| at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
| at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)
| at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
| at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
| at
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
| at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
| at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
| at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
| at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
| at
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
| at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
| at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
| at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
| at
org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
| at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
| at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
| at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
| at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
| at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
| at java.lang.Thread.run(Unknown Source)
| Caused by: org.jboss.cache.ReplicationException: rsp=sender=172.16.100.17:3216,
retval=null, received=false, suspected=false
| at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4436)
| at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4358)
| at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4469)
| at
org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:110)
| at
org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:88)
| at
org.jboss.cache.interceptors.ReplicationInterceptor.runPreparePhase(ReplicationInterceptor.java:152)
| at
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:69)
| at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
| at
org.jboss.cache.interceptors.TxInterceptor.runPreparePhase(TxInterceptor.java:873)
| at
org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1156)
| ... 66 more
| Caused by: org.jboss.cache.lock.TimeoutException: Response timed out:
sender=172.16.100.17:3216, retval=null, received=false, suspected=false
| at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4434)
| ... 75 more
View the original post :
http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4179030#...
Reply to the post :
http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&a...