5:23 a.m.
Hi, folks.
On our environment occasionally were observed numerous
org.jboss.cache.lock.TimeoutException's that cause transactions to fail. Moreover the
problem seems to progress over time resulting in a system becoming non-operational with
restart required.
It was suggested to try to increase SynReplTimeOut up to 100 sec and set loopback
attribute to true (something that is recommended on windows machines). As issue reproduces
occasionally it is hard to verify that these changes will help.
Any thoughts regarding this problem are appreciated.
We use:
JBossAS - 4.2.2.GA
The JBossCache - 1.4.1.SP5
JGroups - 2.4.1-SP4
The config
<mbean code="org.jboss.cache.TreeCache"
name="jboss.cache:service=HibernateTreeCache">
| <depends>jboss:service=Naming</depends>
| <depends>jboss:service=TransactionManager</depends>
| <attribute
name="TransactionManagerLookupClass">org.jboss.cache.JBossTransactionManagerLookup</attribute>
| <attribute
name="IsolationLevel">REPEATABLE_READ</attribute>
| <attribute name="CacheMode">REPL_SYNC</attribute>
| <attribute name="ClusterName">DEFAULT</attribute>
|
| <attribute name="ClusterConfig">
| <config>
| <UDP mcast_addr="231.12.21.132"
mcast_port="45047"
| ip_ttl="64" ip_mcast="true"
| mcast_send_buf_size="150000"
mcast_recv_buf_size="80000"
| ucast_send_buf_size="150000"
ucast_recv_buf_size="80000"
| loopback="false"/>
| <PING timeout="2000" num_initial_members="3"
| up_thread="false" down_thread="false"/>
| <MERGE2 min_interval="10000"
max_interval="20000"/>
| <FD shun="true" up_thread="true"
down_thread="true"/>
| <VERIFY_SUSPECT timeout="1500"
| up_thread="false" down_thread="false"/>
| <pbcast.NAKACK gc_lag="50"
retransmit_timeout="600,1200,2400,4800"
| up_thread="false" down_thread="false"/>
| <pbcast.STABLE desired_avg_gossip="20000"
| up_thread="false" down_thread="false"/>
| <UNICAST timeout="600,1200,2400"
window_size="100" min_threshold="10"
| down_thread="false"/>
| <FRAG frag_size="8192"
| down_thread="false" up_thread="false"/>
| <pbcast.GMS join_timeout="5000"
join_retry_timeout="2000"
| shun="true" print_local_addr="true"/>
| <pbcast.STATE_TRANSFER up_thread="false"
down_thread="false"/>
| </config>
| </attribute>
| <attribute
name="InitialStateRetrievalTimeout">20000</attribute>
| <attribute name="SyncReplTimeout">10000</attribute>
| <attribute
name="LockAcquisitionTimeout">15000</attribute>
| <attribute name="EvictionPolicyClass"></attribute>
| </mbean>
the stack trace:
2008-09-15 04:44:32,548 WARN [http-0.0.0.0-80-2]
[com.arjuna.ats.arjuna.logging.arjLoggerI18N]
[com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator_2]
TwoPhaseCoordinator.beforeCompletion - failed for
com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple@f09a652
| java.lang.RuntimeException:
| at
org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1182)
| at
org.jboss.cache.interceptors.OrderedSynchronizationHandler.beforeCompletion(OrderedSynchronizationHandler.java:75)
| at
com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:114)
| at
com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:247)
| at
com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:86)
| at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:177)
| at
com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1389)
| at
com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:135)
| at
com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:87)
| at
org.jboss.tm.usertx.client.ServerVMClientUserTransaction.commit(ServerVMClientUserTransaction.java:140)
| at
com.enkata.common.transaction.TransactionManager.commit(TransactionManager.java:75)
| at
com.enkata.common.transaction.TransactionManagerHelper.commit(TransactionManagerHelper.java:43)
| at com.enkata.common.transaction.ATransaction.run(ATransaction.java:91)
| at
com.enkata.common.transaction.NonCancellableTransaction.run(NonCancellableTransaction.java:33)
| at
com.enkata.common.transaction.ReturnObjectTransaction.executeTransaction(ReturnObjectTransaction.java:43)
| at com.enkata.eac.to.analysis.AnalysisUIFacade.drill(AnalysisUIFacade.java:171)
| at
com.enkata.eac.ui.analyst.drillpath.DrillPathNavigationCmd.doRun(DrillPathNavigationCmd.java:72)
| at com.enkata.eac.ui.util.ACmd.run(ACmd.java:70)
| at com.enkata.eac.ui.framework.Page.runACmd(Page.java:746)
| at com.enkata.eac.ui.framework.Page.runCmdAndAdjustScrollTop(Page.java:209)
| at com.enkata.eac.ui.framework.Page.validateAndUpdate(Page.java:192)
| at com.enkata.eac.ui.framework.Controller.validateAndUpdate(Controller.java:373)
| at com.enkata.eac.ui.framework.Controller.process2(Controller.java:270)
| at com.enkata.eac.ui.framework.Controller.process(Controller.java:250)
| at com.enkata.eac.ui.framework.UIMgr.processRequest(UIMgr.java:133)
| at com.enkata.eac.ui.servlet.EACServlet.processRequest(EACServlet.java:91)
| at com.enkata.eac.ui.servlet.EACServlet.doPost(EACServlet.java:67)
| at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
| at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
| at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
| at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
| at
com.enkata.eac.ui.servlet.DownloadExportFilter.doFilter(DownloadExportFilter.java:41)
| at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
| at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
| at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
| at org.jboss.seam.web.MultipartFilter.doFilter(MultipartFilter.java:85)
| at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
| at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64)
| at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
| at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
| at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
| at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:147)
| at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:276)
| at org.ajax4jsf.Filter.doFilter(Filter.java:175)
| at org.jboss.seam.web.Ajax4jsfFilter.doFilter(Ajax4jsfFilter.java:60)
| at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
| at org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:58)
| at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
| at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)
| at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
| at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
| at
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
| at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
| at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
| at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
| at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
| at
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
| at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
| at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
| at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
| at
org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
| at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
| at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
| at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
| at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
| at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
| at java.lang.Thread.run(Unknown Source)
| Caused by: org.jboss.cache.ReplicationException: rsp=sender=172.16.100.17:3216,
retval=null, received=false, suspected=false
| at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4436)
| at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4358)
| at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4469)
| at
org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:110)
| at
org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:88)
| at
org.jboss.cache.interceptors.ReplicationInterceptor.runPreparePhase(ReplicationInterceptor.java:152)
| at
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:69)
| at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
| at
org.jboss.cache.interceptors.TxInterceptor.runPreparePhase(TxInterceptor.java:873)
| at
org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1156)
| ... 66 more
| Caused by: org.jboss.cache.lock.TimeoutException: Response timed out:
sender=172.16.100.17:3216, retval=null, received=false, suspected=false
| at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4434)
| ... 75 more
View the original post :
http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4179030#...
Reply to the post :
http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&a...
6421
days inactive
6421
days old
0 comments
1 participants
participants (1)
-
guruk