[jboss-jira] [JBoss JIRA] (AS7-4143) Failover scenarios often fail with "StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view Y"
Paul Ferraro (JIRA)
jira-events at lists.jboss.org
Fri Mar 23 09:57:47 EDT 2012
[ https://issues.jboss.org/browse/AS7-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12678961#comment-12678961 ]
Paul Ferraro commented on AS7-4143:
-----------------------------------
Rado, can you test this against master? I'd like to get a better idea of the effect of RSVP protocol on the duration of state transfer - which should help with these timeouts.
> Failover scenarios often fail with "StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view Y"
> -----------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: AS7-4143
> URL: https://issues.jboss.org/browse/AS7-4143
> Project: Application Server 7
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 7.1.1.Final
> Environment: Linux
> Reporter: Radoslav Husar
> Assignee: Paul Ferraro
> Priority: Critical
> Labels: eap6_need_triage, failover_testing
> Fix For: 7.1.2.Final
>
>
> Most of the positive failover scenarios under little load result in StateTransferInProgressException.
> Load is on average 50r/s (which is 200x less than AS5 easily handles).
> {noformat}
> [JBossINF] 16:04:42,099 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-perf20-10.16.90.58-8009-25) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 23
> [JBossINF] at org.infinispan.interceptors.StateTransferLockInterceptor.signalStateTransferInProgress(StateTransferLockInterceptor.java:199) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:80) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:89) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:86) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager$ClassLoaderAwareCommandInterceptor.handleDefault(DefaultEmbeddedCacheManager.java:410) [jboss-as-clustering-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:174) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:272) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:321) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:90) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:100) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.jboss.as.clustering.web.impl.TransactionBatchingManager.endBatch(TransactionBatchingManager.java:75) [jboss-as-clustering-web-spi-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
> [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.processSessionRepl(DistributableSessionManager.java:1530)
> [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.storeSession(DistributableSessionManager.java:865)
> [JBossINF] at org.jboss.as.web.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:47)
> [JBossINF] at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:133)
> [JBossINF] at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:91)
> [JBossINF] at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:88)
> [JBossINF] at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:56)
> [JBossINF] at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:154)
> [JBossINF] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
> [JBossINF] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> [JBossINF] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> [JBossINF] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
> [JBossINF] at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505)
> [JBossINF] at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445)
> [JBossINF] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
> [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
> {noformat}
> This likely results in timeout on the clients (that you see in the runs).
> {noformat}
> Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: IO error: java.net.SocketTimeoutException: Read timed out>
> org.jboss.smartfrog.loaddriver.RequestProcessingException: IO error: java.net.SocketTimeoutException: Read timed out
> at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:218)
> at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
> at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
> at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
> at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
> at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
> at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
> at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
> at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
> at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
> at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
> at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:132)
> ... 3 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the jboss-jira
mailing list