[jboss-jira] [JBoss JIRA] (AS7-4143) Failover scenarios often fail with "StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view Y"
Radoslav Husar (JIRA)
jira-events at lists.jboss.org
Fri Apr 13 09:16:49 EDT 2012
[ https://issues.jboss.org/browse/AS7-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Radoslav Husar reopened AS7-4143:
---------------------------------
I don't think this should be resolved as of yet. Shelly has seen some occurrences in the TS even probably after the fix was merged and I am seeing this too (probably in much smaller scope than before).
The problem still exists with SYNC replication and when a JVM crash is simulated. Both REPL and DIST are affected.
WDYT Paul? It sounds like the problem is failure detection not telling statetransfer before the TO kicks in that the member is already dead?
E.g.:
https://hudson.qa.jboss.com/hudson/job/eap-6x-failover-http-session-jvmkill-dist-sync/12/console-perf21/
https://hudson.qa.jboss.com/hudson/job/eap-6x-failover-http-session-jvmkill-repl-sync/17/console-perf21/
> Failover scenarios often fail with "StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view Y"
> -----------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: AS7-4143
> URL: https://issues.jboss.org/browse/AS7-4143
> Project: Application Server 7
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 7.1.1.Final
> Environment: Linux
> Reporter: Radoslav Husar
> Assignee: Paul Ferraro
> Priority: Critical
> Labels: eap6_need_triage, failover_testing
> Fix For: 7.1.2.Final-redhat1
>
>
> Most of the positive failover scenarios under little load result in StateTransferInProgressException.
> Load is on average 50r/s (which is 200x less than AS5 easily handles).
> {noformat}
> [JBossINF] 16:04:42,099 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-perf20-10.16.90.58-8009-25) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 23
> [JBossINF] at org.infinispan.interceptors.StateTransferLockInterceptor.signalStateTransferInProgress(StateTransferLockInterceptor.java:199) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:80) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:89) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:86) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager$ClassLoaderAwareCommandInterceptor.handleDefault(DefaultEmbeddedCacheManager.java:410) [jboss-as-clustering-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:174) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:272) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:321) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:90) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:100) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
> [JBossINF] at org.jboss.as.clustering.web.impl.TransactionBatchingManager.endBatch(TransactionBatchingManager.java:75) [jboss-as-clustering-web-spi-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
> [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.processSessionRepl(DistributableSessionManager.java:1530)
> [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.storeSession(DistributableSessionManager.java:865)
> [JBossINF] at org.jboss.as.web.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:47)
> [JBossINF] at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:133)
> [JBossINF] at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:91)
> [JBossINF] at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:88)
> [JBossINF] at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:56)
> [JBossINF] at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:154)
> [JBossINF] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
> [JBossINF] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> [JBossINF] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> [JBossINF] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
> [JBossINF] at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505)
> [JBossINF] at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445)
> [JBossINF] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
> [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
> {noformat}
> This likely results in timeout on the clients (that you see in the runs).
> {noformat}
> Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: IO error: java.net.SocketTimeoutException: Read timed out>
> org.jboss.smartfrog.loaddriver.RequestProcessingException: IO error: java.net.SocketTimeoutException: Read timed out
> at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:218)
> at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
> at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
> at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
> at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
> at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
> at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
> at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
> at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
> at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
> at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
> at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:132)
> ... 3 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the jboss-jira
mailing list