[JBoss JIRA] (ISPN-7090) Asynchronous invocation doesn't work well with multiple futures
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-7090?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-7090:
-------------------------------
Status: Open (was: New)
> Asynchronous invocation doesn't work well with multiple futures
> ---------------------------------------------------------------
>
> Key: ISPN-7090
> URL: https://issues.jboss.org/browse/ISPN-7090
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: testsuite_stability
> Fix For: 9.0.0.Beta1
>
>
> ISPN-5469 made some more interceptors properly asynchronous, and that exposed a problem in {{ComposedAsyncInvocationStage}}: when its future completes, it assumes that the {{stage}} it receives is also completed, and that's not always the case.
> Blocking with {{stage.get()}} is causing {{ManyTxsDuringStateTransferTest.testManyTxs}} to time out and then actually hang during {{MultipleCacheManagersTest.clearContent}}:
> {noformat}
> "OOB-1,ManyTxsDuringStateTransferTest-NodeA-17899" #138868 prio=5 os_prio=0 tid=0x00007ffa91ac4000 nid=0x571e waiting on condition [0x00007ffa570f4000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000a379b180> (a java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1685)
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3320)
> at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1721)
> at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1926)
> at org.infinispan.interceptors.impl.ComposedAsyncInvocationStage.get(ComposedAsyncInvocationStage.java:49)
> at org.infinispan.interceptors.impl.ComposedAsyncInvocationStage.apply(ComposedAsyncInvocationStage.java:128)
> at org.infinispan.interceptors.impl.ComposedAsyncInvocationStage.apply(ComposedAsyncInvocationStage.java:25)
> at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
> at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:29)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> "ForkThread-1,ManyTxsDuringStateTransferTest" #139051 prio=5 os_prio=0 tid=0x00007ffa7c1e8000 nid=0x57d5 waiting on condition [0x00007ffa09d6b000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0000000084609c60> (a java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1685)
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3320)
> at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1721)
> at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1926)
> at org.infinispan.interceptors.impl.ComposedAsyncInvocationStage.get(ComposedAsyncInvocationStage.java:49)
> at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:246)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
> at org.infinispan.transaction.impl.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:240)
> at org.infinispan.transaction.impl.TransactionCoordinator.rollback(TransactionCoordinator.java:179)
> at org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:148)
> at org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:113)
> at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:123)
> at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68)
> at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:698)
> "testng-ManyTxsDuringStateTransferTest" #17 prio=5 os_prio=0 tid=0x00007ffaacc82800 nid=0x3022 waiting on condition [0x00007ffa74550000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000000008460a118> (a java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1685)
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3320)
> at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1721)
> at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1926)
> at org.infinispan.interceptors.impl.ComposedAsyncInvocationStage.get(ComposedAsyncInvocationStage.java:49)
> at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:246)
> at org.infinispan.cache.impl.CacheImpl.clear(CacheImpl.java:569)
> at org.infinispan.cache.impl.CacheImpl.clear(CacheImpl.java:561)
> at org.infinispan.test.TestingUtil.killCaches(TestingUtil.java:783)
> at org.infinispan.test.TestingUtil.killCacheManagers(TestingUtil.java:631)
> at org.infinispan.test.MultipleCacheManagersTest.clearContent(MultipleCacheManagersTest.java:138)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 3 months
[JBoss JIRA] (ISPN-7090) Asynchronous invocation doesn't work well with multiple futures
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-7090?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-7090:
-------------------------------
Summary: Asynchronous invocation doesn't work well with multiple futures (was: Asynchronous invocation doesn't work well multiple levels of futures)
> Asynchronous invocation doesn't work well with multiple futures
> ---------------------------------------------------------------
>
> Key: ISPN-7090
> URL: https://issues.jboss.org/browse/ISPN-7090
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: testsuite_stability
> Fix For: 9.0.0.Beta1
>
>
> ISPN-5469 made some more interceptors properly asynchronous, and that exposed a problem in {{ComposedAsyncInvocationStage}}: when its future completes, it assumes that the {{stage}} it receives is also completed, and that's not always the case.
> Blocking with {{stage.get()}} is causing {{ManyTxsDuringStateTransferTest.testManyTxs}} to time out and then actually hang during {{MultipleCacheManagersTest.clearContent}}:
> {noformat}
> "OOB-1,ManyTxsDuringStateTransferTest-NodeA-17899" #138868 prio=5 os_prio=0 tid=0x00007ffa91ac4000 nid=0x571e waiting on condition [0x00007ffa570f4000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000a379b180> (a java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1685)
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3320)
> at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1721)
> at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1926)
> at org.infinispan.interceptors.impl.ComposedAsyncInvocationStage.get(ComposedAsyncInvocationStage.java:49)
> at org.infinispan.interceptors.impl.ComposedAsyncInvocationStage.apply(ComposedAsyncInvocationStage.java:128)
> at org.infinispan.interceptors.impl.ComposedAsyncInvocationStage.apply(ComposedAsyncInvocationStage.java:25)
> at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
> at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954)
> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:29)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
> at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
> "ForkThread-1,ManyTxsDuringStateTransferTest" #139051 prio=5 os_prio=0 tid=0x00007ffa7c1e8000 nid=0x57d5 waiting on condition [0x00007ffa09d6b000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0000000084609c60> (a java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1685)
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3320)
> at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1721)
> at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1926)
> at org.infinispan.interceptors.impl.ComposedAsyncInvocationStage.get(ComposedAsyncInvocationStage.java:49)
> at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:246)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
> at org.infinispan.transaction.impl.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:240)
> at org.infinispan.transaction.impl.TransactionCoordinator.rollback(TransactionCoordinator.java:179)
> at org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:148)
> at org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:113)
> at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:123)
> at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68)
> at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:698)
> "testng-ManyTxsDuringStateTransferTest" #17 prio=5 os_prio=0 tid=0x00007ffaacc82800 nid=0x3022 waiting on condition [0x00007ffa74550000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000000008460a118> (a java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1685)
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3320)
> at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1721)
> at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1926)
> at org.infinispan.interceptors.impl.ComposedAsyncInvocationStage.get(ComposedAsyncInvocationStage.java:49)
> at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:246)
> at org.infinispan.cache.impl.CacheImpl.clear(CacheImpl.java:569)
> at org.infinispan.cache.impl.CacheImpl.clear(CacheImpl.java:561)
> at org.infinispan.test.TestingUtil.killCaches(TestingUtil.java:783)
> at org.infinispan.test.TestingUtil.killCacheManagers(TestingUtil.java:631)
> at org.infinispan.test.MultipleCacheManagersTest.clearContent(MultipleCacheManagersTest.java:138)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 3 months
[JBoss JIRA] (ISPN-7093) Unexpected error while replicating
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/ISPN-7093?page=com.atlassian.jira.plugin.... ]
Radoslav Husar moved JBEAP-6363 to ISPN-7093:
---------------------------------------------
Project: Infinispan (was: JBoss Enterprise Application Platform)
Key: ISPN-7093 (was: JBEAP-6363)
Workflow: GIT Pull Request with Triage workflow (was: CDW with loose statuses v1)
Component/s: Core
(was: Clustering)
Affects Version/s: 8.2.4.Final
(was: 7.1.0.DR6)
> Unexpected error while replicating
> ----------------------------------
>
> Key: ISPN-7093
> URL: https://issues.jboss.org/browse/ISPN-7093
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 8.2.4.Final
> Reporter: Radoslav Husar
> Assignee: Paul Ferraro
> Labels: infinispan_clean_shutdown
>
> Occured on server during scenario eap-7x-failover-ejb-ejbservlet-shutdown-dist-async.
> Server log stacktrace:
> {code}
> 11:07:31,390 WARN [org.infinispan.transaction.impl.TransactionTable] (ServerService Thread Pool -- 87) ISPN000100: Stopping, but there are 1 local transactions and 4 remote transactions that did not finish in time.
> [JBossINF] [11:07:31,391 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 87) WFLYCLINF0003: Stopped clusterbench-ee7.ear.clusterbench-ee7-web-default.war cache from web container
> [JBossINF] [11:07:31,393 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000080: Disconnecting JGroups channel web
> [JBossINF] [11:07:31,393 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000082: Stopping the RpcDispatcher for channel web
> [JBossINF] [11:07:31,394 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl] (default task-65) ISPN000073: Unexpected error while replicating: java.lang.NullPointerException
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:611)
> [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:144)
> [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:185)
> [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.releaseLocksOnFailureBeforePrepare(PessimisticLockingInterceptor.java:268)
> [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:79)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:80)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:346)
> [JBossINF] at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:331)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitReadCommand(StateTransferInterceptor.java:177)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitGetKeyValueCommand(StateTransferInterceptor.java:154)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:114)
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:83)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)
> [JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:411)
> [JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:443)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:286)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:74)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:64)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:36)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:60)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:38)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:228)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:142)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:772)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:802)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:806)
> [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:572)
> [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:171)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:327)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1671)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1671)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1671)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1671)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:207)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:810)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> [JBossINF]
> [JBossINF] [11:07:31,394 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl] (default task-65) ISPN000073: Unexpected error while replicating: java.lang.NullPointerException
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:611) ...
> {code}
> There were two occurences of this ERROR in a direct sequence when the server was shutting down.
> This ERROR occured afterwards on the server:
> {code}
> 07:31,397 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-65) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.IllegalLifecycleStateException: ISPN000323: Cache 'routing' is in 'TERMINATED' state and so it does not accept new invocations. Either restart it or recreate the cache container.
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:101) ...
> {code}
> JvmRoute changed occured on client at the same time:
> {code}
> 2016/10/05 11:07:31:452 EDT [INFO ][Runner - 874] HOST dev220.mw.lab.eng.bos.redhat.com:rootProcess:c - Failover detected, JvmRoute changed. dev213 -> dev212
> {code}
> Link to server log:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-e...
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 3 months
[JBoss JIRA] (ISPN-7093) Unexpected error while replicating
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/ISPN-7093?page=com.atlassian.jira.plugin.... ]
Radoslav Husar reassigned ISPN-7093:
------------------------------------
Assignee: Dan Berindei (was: Paul Ferraro)
> Unexpected error while replicating
> ----------------------------------
>
> Key: ISPN-7093
> URL: https://issues.jboss.org/browse/ISPN-7093
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 8.2.4.Final
> Reporter: Radoslav Husar
> Assignee: Dan Berindei
> Labels: infinispan_clean_shutdown
>
> Occured on server during scenario eap-7x-failover-ejb-ejbservlet-shutdown-dist-async.
> Server log stacktrace:
> {code}
> 11:07:31,390 WARN [org.infinispan.transaction.impl.TransactionTable] (ServerService Thread Pool -- 87) ISPN000100: Stopping, but there are 1 local transactions and 4 remote transactions that did not finish in time.
> [JBossINF] [11:07:31,391 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 87) WFLYCLINF0003: Stopped clusterbench-ee7.ear.clusterbench-ee7-web-default.war cache from web container
> [JBossINF] [11:07:31,393 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000080: Disconnecting JGroups channel web
> [JBossINF] [11:07:31,393 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000082: Stopping the RpcDispatcher for channel web
> [JBossINF] [11:07:31,394 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl] (default task-65) ISPN000073: Unexpected error while replicating: java.lang.NullPointerException
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:611)
> [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:144)
> [JBossINF] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:185)
> [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.releaseLocksOnFailureBeforePrepare(PessimisticLockingInterceptor.java:268)
> [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:79)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:80)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:346)
> [JBossINF] at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:331)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitReadCommand(StateTransferInterceptor.java:177)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitGetKeyValueCommand(StateTransferInterceptor.java:154)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:114)
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:83)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)
> [JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:411)
> [JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:443)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:286)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:74)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:64)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:36)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:60)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:38)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:228)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:142)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:772)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:802)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:806)
> [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:572)
> [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:171)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:327)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1671)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1671)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1671)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1671)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:207)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:810)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> [JBossINF]
> [JBossINF] [11:07:31,394 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl] (default task-65) ISPN000073: Unexpected error while replicating: java.lang.NullPointerException
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:611) ...
> {code}
> There were two occurences of this ERROR in a direct sequence when the server was shutting down.
> This ERROR occured afterwards on the server:
> {code}
> 07:31,397 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-65) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.IllegalLifecycleStateException: ISPN000323: Cache 'routing' is in 'TERMINATED' state and so it does not accept new invocations. Either restart it or recreate the cache container.
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:101) ...
> {code}
> JvmRoute changed occured on client at the same time:
> {code}
> 2016/10/05 11:07:31:452 EDT [INFO ][Runner - 874] HOST dev220.mw.lab.eng.bos.redhat.com:rootProcess:c - Failover detected, JvmRoute changed. dev213 -> dev212
> {code}
> Link to server log:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-e...
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 3 months
[JBoss JIRA] (ISPN-7092) Failed to broadcast asynchronous command
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/ISPN-7092?page=com.atlassian.jira.plugin.... ]
Radoslav Husar moved JBEAP-6362 to ISPN-7092:
---------------------------------------------
Project: Infinispan (was: JBoss Enterprise Application Platform)
Key: ISPN-7092 (was: JBEAP-6362)
Workflow: GIT Pull Request with Triage workflow (was: CDW with loose statuses v1)
Component/s: Core
(was: Clustering)
Affects Version/s: 8.2.4.Final
(was: 7.1.0.DR6)
> Failed to broadcast asynchronous command
> ----------------------------------------
>
> Key: ISPN-7092
> URL: https://issues.jboss.org/browse/ISPN-7092
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 8.2.4.Final
> Reporter: Radoslav Husar
> Assignee: Paul Ferraro
> Priority: Minor
> Labels: infinispan_clean_shutdown
>
> Occured on server at the end of server stopping.
> Scenarios affected:
> eap-7x-failover-ejb-ejbremote-jvmkill-dist-sync
> eap-7x-failover-ejb-ejbremote-jvmkill-repl-async
> eap-7x-failover-ejb-ejbremote-jvmkill-repl-sync
> eap-7x-failover-ejb-ejbremote-shutdown-dist-async
> eap-7x-failover-ejb-ejbremote-shutdown-dist-sync-tcpStack
> eap-7x-failover-ejb-ejbremote-shutdown-repl-sync
> eap-7x-failover-ejb-ejbremote-undeploy-repl-sync
> eap-7x-failover-ejb-ejbservlet-shutdown-repl-async
> eap-7x-failover-ejb-ejbstateless-jvmkill-repl-async
> eap-7x-failover-http-granular-shutdown-repl-sync
> eap-7x-failover-http-session-shutdown-repl-async
> eap-7x-failover-http-session-shutdown-repl-async-tcpStack
> eap-7x-failover-http-session-shutdown-repl-sync
> eap-7x-failover-jvmkill-ha-ss
> WARN message has two variants.
> *Variant 1*:
> {code}
> 15:01:15,157 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel web
> [JBossINF] [0m[0m15:01:15,157 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000082: Stopping the RpcDispatcher for channel web
> [JBossINF] [0m[0m15:01:15,158 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-3) WFLYJCA0019: Stopped Driver service with driver-name = h2
> [JBossINF] [0m[33m15:01:15,158 WARN [org.infinispan.topology.CacheTopologyControlCommand] (remote-thread--p7-t13) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=REBALANCE_CONFIRM, sender=dev214, joinInfo=null, topologyId=35, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=12}: org.infinispan.commons.CacheException: Failed to broadcast asynchronous command: CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=STABLE_TOPOLOGY_UPDATE, sender=dev213, joinInfo=null, topologyId=36, rebalanceId=17, currentCH=DefaultConsistentHash{ns=256, owners = (2)[dev214: 124+132, dev215: 132+124]}, pendingCH=null, availabilityMode=null, actualMembers=[dev214, dev215], throwable=null, viewId=12}
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:595)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.broadcastStableTopologyUpdate(ClusterTopologyManagerImpl.java:615)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.startQueuedRebalance(ClusterCacheStatus.java:633)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.endRebalance(ClusterCacheStatus.java:373)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.doConfirmRebalance(ClusterCacheStatus.java:310)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.handleRebalanceCompleted(ClusterTopologyManagerImpl.java:263)
> [JBossINF] at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:176)
> [JBossINF] at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:153)
> [JBossINF] at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$2.run(GlobalInboundInvocationHandler.java:159)
> [JBossINF] at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$RunnableWrapper.run(BlockingTaskAwareExecutorServiceImpl.java:199)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at org.jboss.as.clustering.infinispan.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:48)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> [JBossINF] Caused by: org.infinispan.commons.CacheException: java.lang.IllegalStateException: channel is not connected
> [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.rethrowAsCacheException(CommandAwareRpcDispatcher.java:158)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:138)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:578)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:532)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:592)
> [JBossINF] ... 13 more
> [JBossINF] Caused by: java.lang.IllegalStateException: channel is not connected
> [JBossINF] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:682)
> [JBossINF] at org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:172)
> [JBossINF] at org.jgroups.blocks.GroupRequest.sendRequest(GroupRequest.java:325)
> [JBossINF] at org.jgroups.blocks.GroupRequest.sendRequest(GroupRequest.java:76)
> [JBossINF] at org.jgroups.blocks.Request.execute(Request.java:67)
> [JBossINF] at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:373)
> [JBossINF] at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:386)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java:407)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:133)
> [JBossINF] ... 16 more
> {code}
> *Variant 2*:
> {code}
> 15:53:27,821 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t19) ISPN000336: Finished cluster-wide rebalance for cache routing, topology id = 33
> [JBossINF] [0m[0m15:53:27,829 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-8) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> [JBossINF] [0m[0m15:53:27,830 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel web
> [JBossINF] [0m[0m15:53:27,830 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher for channel web
> [JBossINF] [0m[0m15:53:27,821 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t14) ISPN000336: Finished cluster-wide rebalance for cache clusterbench-ee7.ear.clusterbench-ee7-web-default.war, topology id = 39
> [JBossINF] [0m[0m15:53:27,834 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-3) WFLYJCA0019: Stopped Driver service with driver-name = h2
> [JBossINF] [0m[33m15:53:27,832 WARN [org.infinispan.topology.CacheTopologyControlCommand] (remote-thread--p6-t14) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=REBALANCE_CONFIRM, sender=dev215, joinInfo=null, topologyId=39, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=12}: org.infinispan.commons.CacheException: Failed to broadcast asynchronous command: CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=CH_UPDATE, sender=dev213, joinInfo=null, topologyId=40, rebalanceId=21, currentCH=DefaultConsistentHash{ns=256, owners = (2)[dev214: 129+127, dev215: 127+129]}, pendingCH=null, availabilityMode=null, actualMembers=[dev214, dev215], throwable=null, viewId=12}
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:595)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.broadcastTopologyUpdate(ClusterTopologyManagerImpl.java:606)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.endRebalance(ClusterCacheStatus.java:369)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.doConfirmRebalance(ClusterCacheStatus.java:310)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.handleRebalanceCompleted(ClusterTopologyManagerImpl.java:263)
> [JBossINF] at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:176)
> [JBossINF] at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:153)
> [JBossINF] at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$2.run(GlobalInboundInvocationHandler.java:159)
> [JBossINF] at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$RunnableWrapper.run(BlockingTaskAwareExecutorServiceImpl.java:199)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at org.jboss.as.clustering.infinispan.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:48)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> [JBossINF] Caused by: java.lang.NullPointerException
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:611)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:532)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:592)
> [JBossINF] ... 12 more
> {code}
> Link to server log:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-e...
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 3 months
[JBoss JIRA] (ISPN-7092) Failed to broadcast asynchronous command
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/ISPN-7092?page=com.atlassian.jira.plugin.... ]
Radoslav Husar reassigned ISPN-7092:
------------------------------------
Assignee: Dan Berindei (was: Paul Ferraro)
> Failed to broadcast asynchronous command
> ----------------------------------------
>
> Key: ISPN-7092
> URL: https://issues.jboss.org/browse/ISPN-7092
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 8.2.4.Final
> Reporter: Radoslav Husar
> Assignee: Dan Berindei
> Priority: Minor
> Labels: infinispan_clean_shutdown
>
> Occured on server at the end of server stopping.
> Scenarios affected:
> eap-7x-failover-ejb-ejbremote-jvmkill-dist-sync
> eap-7x-failover-ejb-ejbremote-jvmkill-repl-async
> eap-7x-failover-ejb-ejbremote-jvmkill-repl-sync
> eap-7x-failover-ejb-ejbremote-shutdown-dist-async
> eap-7x-failover-ejb-ejbremote-shutdown-dist-sync-tcpStack
> eap-7x-failover-ejb-ejbremote-shutdown-repl-sync
> eap-7x-failover-ejb-ejbremote-undeploy-repl-sync
> eap-7x-failover-ejb-ejbservlet-shutdown-repl-async
> eap-7x-failover-ejb-ejbstateless-jvmkill-repl-async
> eap-7x-failover-http-granular-shutdown-repl-sync
> eap-7x-failover-http-session-shutdown-repl-async
> eap-7x-failover-http-session-shutdown-repl-async-tcpStack
> eap-7x-failover-http-session-shutdown-repl-sync
> eap-7x-failover-jvmkill-ha-ss
> WARN message has two variants.
> *Variant 1*:
> {code}
> 15:01:15,157 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel web
> [JBossINF] [0m[0m15:01:15,157 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000082: Stopping the RpcDispatcher for channel web
> [JBossINF] [0m[0m15:01:15,158 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-3) WFLYJCA0019: Stopped Driver service with driver-name = h2
> [JBossINF] [0m[33m15:01:15,158 WARN [org.infinispan.topology.CacheTopologyControlCommand] (remote-thread--p7-t13) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=REBALANCE_CONFIRM, sender=dev214, joinInfo=null, topologyId=35, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=12}: org.infinispan.commons.CacheException: Failed to broadcast asynchronous command: CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=STABLE_TOPOLOGY_UPDATE, sender=dev213, joinInfo=null, topologyId=36, rebalanceId=17, currentCH=DefaultConsistentHash{ns=256, owners = (2)[dev214: 124+132, dev215: 132+124]}, pendingCH=null, availabilityMode=null, actualMembers=[dev214, dev215], throwable=null, viewId=12}
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:595)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.broadcastStableTopologyUpdate(ClusterTopologyManagerImpl.java:615)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.startQueuedRebalance(ClusterCacheStatus.java:633)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.endRebalance(ClusterCacheStatus.java:373)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.doConfirmRebalance(ClusterCacheStatus.java:310)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.handleRebalanceCompleted(ClusterTopologyManagerImpl.java:263)
> [JBossINF] at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:176)
> [JBossINF] at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:153)
> [JBossINF] at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$2.run(GlobalInboundInvocationHandler.java:159)
> [JBossINF] at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$RunnableWrapper.run(BlockingTaskAwareExecutorServiceImpl.java:199)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at org.jboss.as.clustering.infinispan.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:48)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> [JBossINF] Caused by: org.infinispan.commons.CacheException: java.lang.IllegalStateException: channel is not connected
> [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.rethrowAsCacheException(CommandAwareRpcDispatcher.java:158)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:138)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:578)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:532)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:592)
> [JBossINF] ... 13 more
> [JBossINF] Caused by: java.lang.IllegalStateException: channel is not connected
> [JBossINF] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:682)
> [JBossINF] at org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:172)
> [JBossINF] at org.jgroups.blocks.GroupRequest.sendRequest(GroupRequest.java:325)
> [JBossINF] at org.jgroups.blocks.GroupRequest.sendRequest(GroupRequest.java:76)
> [JBossINF] at org.jgroups.blocks.Request.execute(Request.java:67)
> [JBossINF] at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:373)
> [JBossINF] at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:386)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java:407)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:133)
> [JBossINF] ... 16 more
> {code}
> *Variant 2*:
> {code}
> 15:53:27,821 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t19) ISPN000336: Finished cluster-wide rebalance for cache routing, topology id = 33
> [JBossINF] [0m[0m15:53:27,829 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-8) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> [JBossINF] [0m[0m15:53:27,830 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel web
> [JBossINF] [0m[0m15:53:27,830 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher for channel web
> [JBossINF] [0m[0m15:53:27,821 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t14) ISPN000336: Finished cluster-wide rebalance for cache clusterbench-ee7.ear.clusterbench-ee7-web-default.war, topology id = 39
> [JBossINF] [0m[0m15:53:27,834 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-3) WFLYJCA0019: Stopped Driver service with driver-name = h2
> [JBossINF] [0m[33m15:53:27,832 WARN [org.infinispan.topology.CacheTopologyControlCommand] (remote-thread--p6-t14) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=REBALANCE_CONFIRM, sender=dev215, joinInfo=null, topologyId=39, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=12}: org.infinispan.commons.CacheException: Failed to broadcast asynchronous command: CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=CH_UPDATE, sender=dev213, joinInfo=null, topologyId=40, rebalanceId=21, currentCH=DefaultConsistentHash{ns=256, owners = (2)[dev214: 129+127, dev215: 127+129]}, pendingCH=null, availabilityMode=null, actualMembers=[dev214, dev215], throwable=null, viewId=12}
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:595)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.broadcastTopologyUpdate(ClusterTopologyManagerImpl.java:606)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.endRebalance(ClusterCacheStatus.java:369)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.doConfirmRebalance(ClusterCacheStatus.java:310)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.handleRebalanceCompleted(ClusterTopologyManagerImpl.java:263)
> [JBossINF] at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:176)
> [JBossINF] at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:153)
> [JBossINF] at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$2.run(GlobalInboundInvocationHandler.java:159)
> [JBossINF] at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$RunnableWrapper.run(BlockingTaskAwareExecutorServiceImpl.java:199)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at org.jboss.as.clustering.infinispan.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:48)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> [JBossINF] Caused by: java.lang.NullPointerException
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:611)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:532)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:592)
> [JBossINF] ... 12 more
> {code}
> Link to server log:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-e...
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 3 months
[JBoss JIRA] (ISPN-7090) Asynchronous invocation doesn't work well multiple levels of futures
by Dan Berindei (JIRA)
Dan Berindei created ISPN-7090:
----------------------------------
Summary: Asynchronous invocation doesn't work well multiple levels of futures
Key: ISPN-7090
URL: https://issues.jboss.org/browse/ISPN-7090
Project: Infinispan
Issue Type: Bug
Components: Core
Reporter: Dan Berindei
Assignee: Dan Berindei
Priority: Critical
Fix For: 9.0.0.Beta1
ISPN-5469 made some more interceptors properly asynchronous, and that exposed a problem in {{ComposedAsyncInvocationStage}}: when its future completes, it assumes that the {{stage}} it receives is also completed, and that's not always the case.
Blocking with {{stage.get()}} is causing {{ManyTxsDuringStateTransferTest.testManyTxs}} to time out and then actually hang during {{MultipleCacheManagersTest.clearContent}}:
{noformat}
"OOB-1,ManyTxsDuringStateTransferTest-NodeA-17899" #138868 prio=5 os_prio=0 tid=0x00007ffa91ac4000 nid=0x571e waiting on condition [0x00007ffa570f4000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000a379b180> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1685)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3320)
at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1721)
at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1926)
at org.infinispan.interceptors.impl.ComposedAsyncInvocationStage.get(ComposedAsyncInvocationStage.java:49)
at org.infinispan.interceptors.impl.ComposedAsyncInvocationStage.apply(ComposedAsyncInvocationStage.java:128)
at org.infinispan.interceptors.impl.ComposedAsyncInvocationStage.apply(ComposedAsyncInvocationStage.java:25)
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954)
at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:29)
at org.jgroups.blocks.Request.checkCompletion(Request.java:152)
at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:116)
"ForkThread-1,ManyTxsDuringStateTransferTest" #139051 prio=5 os_prio=0 tid=0x00007ffa7c1e8000 nid=0x57d5 waiting on condition [0x00007ffa09d6b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000084609c60> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1685)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3320)
at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1721)
at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1926)
at org.infinispan.interceptors.impl.ComposedAsyncInvocationStage.get(ComposedAsyncInvocationStage.java:49)
at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:246)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
at org.infinispan.transaction.impl.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:240)
at org.infinispan.transaction.impl.TransactionCoordinator.rollback(TransactionCoordinator.java:179)
at org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:148)
at org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:113)
at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:123)
at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68)
at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:698)
"testng-ManyTxsDuringStateTransferTest" #17 prio=5 os_prio=0 tid=0x00007ffaacc82800 nid=0x3022 waiting on condition [0x00007ffa74550000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000008460a118> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1685)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3320)
at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1721)
at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1926)
at org.infinispan.interceptors.impl.ComposedAsyncInvocationStage.get(ComposedAsyncInvocationStage.java:49)
at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:246)
at org.infinispan.cache.impl.CacheImpl.clear(CacheImpl.java:569)
at org.infinispan.cache.impl.CacheImpl.clear(CacheImpl.java:561)
at org.infinispan.test.TestingUtil.killCaches(TestingUtil.java:783)
at org.infinispan.test.TestingUtil.killCacheManagers(TestingUtil.java:631)
at org.infinispan.test.MultipleCacheManagersTest.clearContent(MultipleCacheManagersTest.java:138)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 3 months
[JBoss JIRA] (ISPN-7088) Interceptors should not access TransactionManager
by Sebastian Łaskawiec (JIRA)
[ https://issues.jboss.org/browse/ISPN-7088?page=com.atlassian.jira.plugin.... ]
Sebastian Łaskawiec updated ISPN-7088:
--------------------------------------
Fix Version/s: 9.0.0.Final
> Interceptors should not access TransactionManager
> -------------------------------------------------
>
> Key: ISPN-7088
> URL: https://issues.jboss.org/browse/ISPN-7088
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: testsuite_failure
> Fix For: 9.0.0.Beta1, 9.0.0.Final
>
>
> With ISPN-5469, we are no longer executing all the interceptor on the user thread, even on the originator. That means trying to read the thread-local transaction with {{TransactionManager.getTransaction()}} sometimes won't work, so interceptors should avoid it.
> E.g. {{InvocationContextInterceptor.markTxForRollbackAndRethrow()}} wants to rollback the current transaction but doesn't find any transaction bound to the current thread, causing failures in {{PessimisticTxPartitionAndMergeDuringRuntimeTest}}.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 3 months