[JBoss JIRA] (ISPN-7553) Replicated cache with numOwners=1 doesn't send writes to backups
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-7553?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-7553:
-------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 9.0.0.Final
9.0.0.CR3
(was: 9.3.0.Final)
Resolution: Done
> Replicated cache with numOwners=1 doesn't send writes to backups
> ----------------------------------------------------------------
>
> Key: ISPN-7553
> URL: https://issues.jboss.org/browse/ISPN-7553
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.0.0.CR2
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Fix For: 9.0.0.Final, 9.0.0.CR3
>
>
> {{BaseDistributionInterceptor.primaryReturnHandler()}} doesn't replicate writes to the backups if {{cacheConfiguration.clustering().hash().numOwners() == 1}}. That check is incorrect, because a segment can have more owners than configured:
> 1. In a replicated cache, the configured {{numOwners}} is ignored, and all the members are owners.
> 1. Even in a distributed cache, during rebalance, the segments being moved will have more than {{numOwners}} owners.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 1 month
[JBoss JIRA] (ISPN-5610) [stderr] CacheException: javax.transaction.HeuristicRollbackException at server shutdown
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-5610?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-5610:
------------------------------------
[~pferraro] is there anything we can do in Infinispan for this?
> [stderr] CacheException: javax.transaction.HeuristicRollbackException at server shutdown
> ----------------------------------------------------------------------------------------
>
> Key: ISPN-5610
> URL: https://issues.jboss.org/browse/ISPN-5610
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 7.2.3.Final
> Reporter: Michal Vinkler
> Assignee: Paul Ferraro
>
> Seen in our failover tests for remote stateful EJBs during server shutdown.
> This error is intermittent and was sometimes seen on one node in 4-nodes cluster.
> When server is shutting down, this [stderr] output occurs (among other errors) in the server log:
> {code:}
> [JBossINF] [0m[0m13:04:52,864 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0008: Undertow HTTP listener default suspending
> [JBossINF] [0m[31m13:04:52,865 ERROR [stderr] (ServerService Thread Pool -- 84) Exception in thread "ServerService Thread Pool -- 84" org.infinispan.commons.CacheException: javax.transaction.HeuristicRollbackException
> [JBossINF] [0m[0m13:04:52,865 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0007: Undertow HTTP listener default stopped, was bound to perf19/10.16.90.56:8080
> [JBossINF] [0m[31m13:04:52,865 ERROR [stderr] (ServerService Thread Pool -- 84) at org.wildfly.clustering.ee.infinispan.ActiveTransactionBatch.close(ActiveTransactionBatch.java:50)
> [JBossINF] [0m[31m13:04:52,865 ERROR [stderr] (ServerService Thread Pool -- 84) at org.wildfly.clustering.server.registry.CacheRegistry.close(CacheRegistry.java:87)
> [JBossINF] [0m[31m13:04:52,865 ERROR [stderr] (ServerService Thread Pool -- 84) at org.wildfly.clustering.server.registry.CacheRegistryFactory$1.close(CacheRegistryFactory.java:56)
> [JBossINF] [0m[31m13:04:52,866 ERROR [stderr] (ServerService Thread Pool -- 84) at org.wildfly.clustering.server.registry.RegistryBuilder.stop(RegistryBuilder.java:88)
> [JBossINF] [0m[31m13:04:52,866 ERROR [stderr] (ServerService Thread Pool -- 84) at org.wildfly.clustering.service.AsynchronousServiceBuilder$2.run(AsynchronousServiceBuilder.java:130)
> [JBossINF] [0m[31m13:04:52,866 ERROR [stderr] (ServerService Thread Pool -- 84) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] [0m[31m13:04:52,866 ERROR [stderr] (ServerService Thread Pool -- 84) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] [0m[31m13:04:52,866 ERROR [stderr] (ServerService Thread Pool -- 84) at java.lang.Thread.run(Thread.java:745)
> [JBossINF] [0m[0m13:04:52,866 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0004: Undertow 1.3.0.Beta3 stopping
> [JBossINF] [0m[31m13:04:52,866 ERROR [stderr] (ServerService Thread Pool -- 84) at org.jboss.threads.JBossThread.run(JBossThread.java:320)
> [JBossINF] [0m[31m13:04:52,866 ERROR [stderr] (ServerService Thread Pool -- 84) Caused by: javax.transaction.HeuristicRollbackException
> [JBossINF] [0m[31m13:04:52,867 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:398)
> [JBossINF] [0m[31m13:04:52,867 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.transaction.tm.DummyTransaction.commitResources(DummyTransaction.java:413)
> [JBossINF] [0m[0m13:04:52,868 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 88) ISPN000029: Passivating all entries to disk
> [JBossINF] [0m[31m13:04:52,870 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.transaction.tm.DummyTransaction.runCommit(DummyTransaction.java:303)
> [JBossINF] [0m[31m13:04:52,872 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:104)
> [JBossINF] [0m[0m13:04:52,872 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 88) ISPN000030: Passivated 2 entries in 3 milliseconds
> [JBossINF] [0m[31m13:04:52,872 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:73)
> [JBossINF] [0m[31m13:04:52,873 ERROR [stderr] (ServerService Thread Pool -- 84) at org.wildfly.clustering.ee.infinispan.ActiveTransactionBatch.close(ActiveTransactionBatch.java:48)
> [JBossINF] [0m[31m13:04:52,873 ERROR [stderr] (ServerService Thread Pool -- 84) ... 8 more
> [JBossINF] [0m[31m13:04:52,873 ERROR [stderr] (ServerService Thread Pool -- 84) Caused by: javax.transaction.xa.XAException
> [JBossINF] [0m[31m13:04:52,873 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:213)
> [JBossINF] [0m[31m13:04:52,873 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159)
> [JBossINF] [0m[31m13:04:52,874 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:112)
> [JBossINF] [0m[31m13:04:52,874 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:367)
> [JBossINF] [0m[31m13:04:52,874 ERROR [stderr] (ServerService Thread Pool -- 84) ... 13 more
> [JBossINF] [0m[31m13:04:52,874 ERROR [stderr] (ServerService Thread Pool -- 84) Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf20, see cause for remote stack trace
> [JBossINF] [0m[31m13:04:52,874 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:46)
> [JBossINF] [0m[31m13:04:52,874 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:71)
> [JBossINF] [0m[31m13:04:52,874 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:586)
> [JBossINF] [0m[31m13:04:52,874 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:287)
> [JBossINF] [0m[31m13:04:52,875 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:222)
> [JBossINF] [0m[31m13:04:52,875 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:206)
> [JBossINF] [0m[31m13:04:52,876 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
> [JBossINF] [0m[31m13:04:52,876 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> [JBossINF] [0m[31m13:04:52,876 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> [JBossINF] [0m[31m13:04:52,876 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123)
> [JBossINF] [0m[31m13:04:52,876 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
> [JBossINF] [0m[31m13:04:52,876 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> [JBossINF] [0m[31m13:04:52,876 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:102)
> [JBossINF] [0m[31m13:04:52,876 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
> [JBossINF] [0m[31m13:04:52,876 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> [JBossINF] [0m[31m13:04:52,877 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> [JBossINF] [0m[31m13:04:52,877 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123)
> [JBossINF] [0m[31m13:04:52,877 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
> [JBossINF] [0m[31m13:04:52,877 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> [JBossINF] [0m[31m13:04:52,877 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:96)
> [JBossINF] [0m[31m13:04:52,877 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:99)
> [JBossINF] [0m[31m13:04:52,877 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
> [JBossINF] [0m[31m13:04:52,877 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> [JBossINF] [0m[31m13:04:52,878 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:36)
> [JBossINF] [0m[31m13:04:52,878 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
> [JBossINF] [0m[31m13:04:52,878 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> [JBossINF] [0m[31m13:04:52,878 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:138)
> [JBossINF] [0m[31m13:04:52,878 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:125)
> [JBossINF] [0m[31m13:04:52,878 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
> [JBossINF] [0m[31m13:04:52,878 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> [JBossINF] [0m[31m13:04:52,878 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> [JBossINF] [0m[31m13:04:52,879 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123)
> [JBossINF] [0m[31m13:04:52,879 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:42)
> [JBossINF] [0m[31m13:04:52,879 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
> [JBossINF] [0m[31m13:04:52,879 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> [JBossINF] [0m[31m13:04:52,879 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:200)
> [JBossINF] [0m[31m13:04:52,879 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:88)
> [JBossINF] [0m[31m13:04:52,880 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
> [JBossINF] [0m[31m13:04:52,881 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> [JBossINF] [0m[31m13:04:52,881 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> [JBossINF] [0m[31m13:04:52,881 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123)
> [JBossINF] [0m[31m13:04:52,881 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
> [JBossINF] [0m[31m13:04:52,881 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> [JBossINF] [0m[31m13:04:52,881 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
> [JBossINF] [0m[31m13:04:52,881 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
> [JBossINF] [0m[31m13:04:52,882 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123)
> [JBossINF] [0m[31m13:04:52,882 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
> [JBossINF] [0m[31m13:04:52,882 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> [JBossINF] [0m[31m13:04:52,882 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:157)
> [JBossINF] [0m[31m13:04:52,882 ERROR [stderr] (ServerService Thread Pool -- 84) ... 15 more
> [JBossINF] [0m[31m13:04:52,882 ERROR [stderr] (ServerService Thread Pool -- 84) Caused by: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.lang.NullPointerException] while invoking method [public void org.wildfly.clustering.server.registry.CacheRegistry.removed(org.infinispan.notifications.cachelistener.event.CacheEntryRemovedEvent)] on listener instance: org.wildfly.clustering.server.registry.CacheRegistryFactory$1@76942caa
> [JBossINF] [0m[31m13:04:52,882 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:291)
> [JBossINF] [0m[31m13:04:52,882 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:22)
> [JBossINF] [0m[31m13:04:52,883 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:309)
> [JBossINF] [0m[31m13:04:52,883 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1212)
> [JBossINF] [0m[31m13:04:52,883 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invokeNoChecks(CacheNotifierImpl.java:1207)
> [JBossINF] [0m[31m13:04:52,883 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1184)
> [JBossINF] [0m[31m13:04:52,883 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryRemoved(CacheNotifierImpl.java:356)
> [JBossINF] [0m[31m13:04:52,883 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.notifyCommitEntry(ClusteringDependentLogic.java:132)
> [JBossINF] [0m[31m13:04:52,883 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.locking.ClusteringDependentLogic$DistributionLogic.commitSingleEntry(ClusteringDependentLogic.java:493)
> [JBossINF] [0m[31m13:04:52,884 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.commitEntry(ClusteringDependentLogic.java:108)
> [JBossINF] [0m[31m13:04:52,884 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:371)
> [JBossINF] [0m[31m13:04:52,884 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:549)
> [JBossINF] [0m[31m13:04:52,884 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:357)
> [JBossINF] [0m[31m13:04:52,884 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:104)
> [JBossINF] [0m[31m13:04:52,884 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
> [JBossINF] [0m[31m13:04:52,884 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> [JBossINF] [0m[31m13:04:52,884 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> [JBossINF] [0m[31m13:04:52,885 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123)
> [JBossINF] [0m[31m13:04:52,885 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
> [JBossINF] [0m[31m13:04:52,885 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> [JBossINF] [0m[31m13:04:52,885 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:96)
> [JBossINF] [0m[31m13:04:52,885 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:99)
> [JBossINF] [0m[31m13:04:52,885 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
> [JBossINF] [0m[31m13:04:52,885 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> [JBossINF] [0m[31m13:04:52,885 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:36)
> [JBossINF] [0m[31m13:04:52,886 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
> [JBossINF] [0m[31m13:04:52,886 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> [JBossINF] [0m[31m13:04:52,886 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:138)
> [JBossINF] [0m[31m13:04:52,886 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:125)
> [JBossINF] [0m[31m13:04:52,886 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
> [JBossINF] [0m[31m13:04:52,887 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> [JBossINF] [0m[31m13:04:52,887 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> [JBossINF] [0m[31m13:04:52,887 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123)
> [JBossINF] [0m[31m13:04:52,888 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:39)
> [JBossINF] [0m[31m13:04:52,888 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
> [JBossINF] [0m[31m13:04:52,888 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> [JBossINF] [0m[31m13:04:52,888 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:200)
> [JBossINF] [0m[31m13:04:52,888 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:88)
> [JBossINF] [0m[31m13:04:52,889 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
> [JBossINF] [0m[31m13:04:52,889 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> [JBossINF] [0m[31m13:04:52,889 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> [JBossINF] [0m[31m13:04:52,889 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123)
> [JBossINF] [0m[31m13:04:52,889 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
> [JBossINF] [0m[31m13:04:52,889 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> [JBossINF] [0m[31m13:04:52,889 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
> [JBossINF] [0m[31m13:04:52,889 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
> [JBossINF] [0m[31m13:04:52,890 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123)
> [JBossINF] [0m[0m13:04:52,890 INFO [org.infinispan.CLUSTER] (ServerService Thread Pool -- 88) ISPN000310: Starting cluster-wide rebalance for cache dist, topology CacheTopology{id=37, rebalanceId=15, currentCH=DefaultConsistentHash{ns=80, owners = (2)[perf20: 39+15, perf21: 41+12]}, pendingCH=DefaultConsistentHash{ns=80, owners = (2)[perf20: 40+40, perf21: 40+40]}, unionCH=null, actualMembers=[perf20, perf21]}
> [JBossINF] [0m[31m13:04:52,890 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
> [JBossINF] [0m[31m13:04:52,890 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> [JBossINF] [0m[31m13:04:52,890 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:113)
> [JBossINF] [0m[31m13:04:52,890 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:85)
> [JBossINF] [0m[0m13:04:52,890 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 88) WFLYCLINF0003: Stopped dist cache from ejb container
> [JBossINF] [0m[31m13:04:52,890 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:32)
> [JBossINF] [0m[31m13:04:52,891 ERROR [stderr] (ServerService Thread Pool -- 84) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] [0m[31m13:04:52,891 ERROR [stderr] (ServerService Thread Pool -- 84) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] [0m[31m13:04:52,891 ERROR [stderr] (ServerService Thread Pool -- 84) at java.lang.Thread.run(Thread.java:745)
> [JBossINF] [0m[31m13:04:52,891 ERROR [stderr] (ServerService Thread Pool -- 84) Caused by: java.lang.NullPointerException
> [JBossINF] [0m[31m13:04:52,891 ERROR [stderr] (ServerService Thread Pool -- 84) at org.wildfly.clustering.server.registry.CacheRegistry.notifyListeners(CacheRegistry.java:183)
> [JBossINF] [0m[31m13:04:52,891 ERROR [stderr] (ServerService Thread Pool -- 84) at org.wildfly.clustering.server.registry.CacheRegistry.removed(CacheRegistry.java:178)
> [JBossINF] [0m[31m13:04:52,891 ERROR [stderr] (ServerService Thread Pool -- 84) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [JBossINF] [0m[31m13:04:52,892 ERROR [stderr] (ServerService Thread Pool -- 84) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [JBossINF] [0m[31m13:04:52,892 ERROR [stderr] (ServerService Thread Pool -- 84) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [JBossINF] [0m[31m13:04:52,892 ERROR [stderr] (ServerService Thread Pool -- 84) at java.lang.reflect.Method.invoke(Method.java:497)
> [JBossINF] [0m[31m13:04:52,892 ERROR [stderr] (ServerService Thread Pool -- 84) at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:286)
> [JBossINF] [0m[31m13:04:52,892 ERROR [stderr] (ServerService Thread Pool -- 84) ... 54 more
> [JBossINF] [0m[0m13:04:52,902 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t30) ISPN000336: Finished cluster-wide rebalance for cache dist, topology id = 37
> [JBossINF] [0m[0m13:04:52,904 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000080: Disconnecting JGroups channel ejb
> [JBossINF] [0m[0m13:04:52,905 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000082: Stopping the RpcDispatcher for channel ejb
> [JBossINF] [0m[0m13:04:52,957 INFO [org.jboss.as] (MSC service thread 1-4) WFLYSRV0050: WildFly Core 2.0.0.Alpha9 "Kenny" stopped in 13744ms
> {code}
> RA: I believe the problem here is that ActiveTransactionBatch is throwing a raw new CacheException(e). We should have a logger defined for the ee subsystem so that it throws something like this: InfinispanEELogger.ROOT_LOGGER.debug(...)); At the moment, such a logger is not defined.
> Server log:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-e...
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-e...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 1 month
[JBoss JIRA] (ISPN-8682) The events in the server log and DMR are not consistent
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-8682?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-8682:
------------------------------------
[~rmacor] the messages you posted here are not log events, they are just regular log messages logged by {{ClusterTopologyManagerImpl}} with the {{CLUSTER}} category name.
Log events look a bit different, they have a {{Context}} or {{Scope}}:
{noformat}
08:53:01,528 INFO (remote-thread-test-NodeA-p2-t6:[cluster-listener]) [CLUSTER] [Context=cluster-listener] ISPN100002: Starting rebalance with members [test-NodeA-15001, test-NodeB-55628, test-NodeC-62395, test-NodeD-29215], phase READ_OLD_WRITE_ALL, topology id 10
{noformat}
With ISPN-9086 I added some events and I changed most of the non-event {{CLUSTER}} log messages to {{DEBUG}} level, but if you think it would make things clearer I could change the category name as well (to {{ClusterTopologyManagerImpl}} or {{ClusterCacheStatus}}).
> The events in the server log and DMR are not consistent
> -------------------------------------------------------
>
> Key: ISPN-8682
> URL: https://issues.jboss.org/browse/ISPN-8682
> Project: Infinispan
> Issue Type: Bug
> Components: JMX, reporting and management, Server
> Affects Versions: 9.2.0.Beta2
> Reporter: Roman Macor
>
> Some events are displayed in server.log file but not in DMR
> Steps to reproduce:
> start server in the domain mode
> turn off server-two node, turn it back on (e.g. through administration console)
> connect through CLI and get the logs:
> [domain@localhost:9990 /] cd /host=master/server=server-one/subsystem=datagrid-infinispan/cache-container=clustered
> [domain@localhost:9990 cache-container=clustered] :read-event-log(lines=20)
> compare the logs with server.log file
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 1 month
[JBoss JIRA] (ISPN-9139) CacheWriterInterceptor invokes batch persistence operations with potentially empty entry sets
by Ryan Emerson (JIRA)
[ https://issues.jboss.org/browse/ISPN-9139?page=com.atlassian.jira.plugin.... ]
Ryan Emerson updated ISPN-9139:
-------------------------------
Fix Version/s: 9.2.4.Final
> CacheWriterInterceptor invokes batch persistence operations with potentially empty entry sets
> ---------------------------------------------------------------------------------------------
>
> Key: ISPN-9139
> URL: https://issues.jboss.org/browse/ISPN-9139
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 9.2.2.Final
> Reporter: Moritz Becker
> Assignee: Ryan Emerson
> Fix For: 9.3.0.CR1, 9.2.4.Final
>
>
> In {{org.infinispan.interceptors.impl.CacheWriterInterceptor#store}}, you have the following:
> {code:java}
> persistenceManager.writeBatchToAllNonTxStores(sharedMods.getMarshalledEntries(), BOTH, 0);
> persistenceManager.writeBatchToAllNonTxStores(nonSharedMods.getMarshalledEntries(), PRIVATE, 0);
> persistenceManager.deleteBatchFromAllNonTxStores(sharedMods.getKeysToRemove(), BOTH, 0);
> persistenceManager.deleteBatchFromAllNonTxStores(nonSharedMods.getKeysToRemove(), PRIVATE, 0);
> {code}
> The batch deletion causes an error with JPAStore in case of an empty key set because the an SQL query like {{where id in ()}}, i.e. the comparison set for the in clause is empty. At least with postgres, this causes the following exception:
> {code}
> ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (EE-ManagedExecutorService-default-Thread-2) ISPN000136: Error executing command CommitCommand, writing keys []: org.infinispan.persistence.jpa.JpaStoreException: Exception caught in deleteBatch()
> at org.infinispan.persistence.jpa.JpaStore.deleteBatch(JpaStore.java:350)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.lambda$deleteBatchFromAllNonTxStores$14(PersistenceManagerImpl.java:562)
> at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
> at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.deleteBatchFromAllNonTxStores(PersistenceManagerImpl.java:562)
> at org.infinispan.interceptors.impl.CacheWriterInterceptor.store(CacheWriterInterceptor.java:406)
> at org.infinispan.interceptors.impl.CacheWriterInterceptor.commitCommand(CacheWriterInterceptor.java:123)
> at org.infinispan.interceptors.impl.CacheWriterInterceptor.visitCommitCommand(CacheWriterInterceptor.java:102)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:38)
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> at org.infinispan.interceptors.impl.TransactionalStoreInterceptor.visitCommitCommand(TransactionalStoreInterceptor.java:53)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:38)
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> at org.infinispan.interceptors.DDAsyncInterceptor.visitCommitCommand(DDAsyncInterceptor.java:142)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:38)
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndFinally(BaseAsyncInterceptor.java:150)
> at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitCommitCommand(EntryWrappingInterceptor.java:183)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:38)
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:98)
> at org.infinispan.interceptors.impl.NotificationInterceptor.visitCommitCommand(NotificationInterceptor.java:46)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:38)
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndFinally(BaseAsyncInterceptor.java:150)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitCommitCommand(AbstractTxLockingInterceptor.java:51)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:38)
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:98)
> at org.infinispan.interceptors.impl.TxInterceptor.finishCommit(TxInterceptor.java:189)
> at org.infinispan.interceptors.impl.TxInterceptor.visitCommitCommand(TxInterceptor.java:183)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:38)
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
> at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> at org.infinispan.interceptors.DDAsyncInterceptor.visitCommitCommand(DDAsyncInterceptor.java:142)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:38)
> at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:123)
> at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90)
> at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:248)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
> at org.infinispan.transaction.impl.TransactionCoordinator.commitInternal(TransactionCoordinator.java:219)
> at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:161)
> at org.infinispan.transaction.impl.TransactionTable.afterCompletion(TransactionTable.java:851)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:33)
> at org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:196)
> at org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.afterCompletion(AbstractTransaction.java:279)
> at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:542)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:101)
> at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
> at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
> at org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:73)
> at org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.commitIfNeeded(PersistenceManagerImpl.java:874)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.preloadKey(PersistenceManagerImpl.java:830)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.lambda$preload$2(PersistenceManagerImpl.java:279)
> at org.infinispan.persistence.jpa.JpaStore$LoadingProcessTask.call(JpaStore.java:915)
> at org.infinispan.persistence.jpa.JpaStore$LoadingProcessTask.call(JpaStore.java:843)
> at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java)
> at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:20)
> at java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:181)
> at org.infinispan.executors.ExecutorAllCompletionService.submit(ExecutorAllCompletionService.java:31)
> at org.infinispan.persistence.jpa.JpaStore.process(JpaStore.java:644)
> at org.infinispan.persistence.jpa.JpaStore.process(JpaStore.java:583)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.preload(PersistenceManagerImpl.java:271)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:229)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1020)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589)
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447)
> at com.ordami.geo.service.impl.service.MatrixUpdateServiceImpl.updateMatrixEntry(MatrixUpdateServiceImpl.java:157)
> at com.ordami.geo.service.impl.service.MatrixUpdateServiceImpl$Proxy$_$$_WeldSubclass.updateMatrixEntry$$super(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:49)
> at org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:77)
> at com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorBase.invokeInOurTx(TransactionalInterceptorBase.java:120)
> at com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorRequired.doIntercept(TransactionalInterceptorRequired.java:53)
> at com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorBase.intercept(TransactionalInterceptorBase.java:79)
> at com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorRequired.intercept(TransactionalInterceptorRequired.java:47)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73)
> at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84)
> at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72)
> at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56)
> at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79)
> at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68)
> at com.ordami.geo.service.impl.service.MatrixUpdateServiceImpl$Proxy$_$$_WeldSubclass.updateMatrixEntry(Unknown Source)
> at com.ordami.geo.service.impl.service.MatrixUpdateServiceImpl$Proxy$_$$_WeldClientProxy.updateMatrixEntry(Unknown Source)
> at com.ordami.geo.service.impl.service.MatrixUpdateServiceImpl$MatrixEntryUpdateTask.run(MatrixUpdateServiceImpl.java:207)
> at org.jboss.as.ee.concurrent.ControlPointUtils$ControlledRunnable.run(ControlPointUtils.java:105)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java)
> at org.glassfish.enterprise.concurrent.internal.ManagedFutureTask.run(ManagedFutureTask.java:141)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250)
> at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78)
> Caused by: javax.persistence.PersistenceException: org.hibernate.exception.SQLGrammarException: could not execute statement
> at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:149)
> at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:157)
> at org.hibernate.query.internal.AbstractProducedQuery.executeUpdate(AbstractProducedQuery.java:1514)
> at org.infinispan.persistence.jpa.JpaStore.deleteBatch(JpaStore.java:334)
> ... 133 more
> Caused by: org.hibernate.exception.SQLGrammarException: could not execute statement
> at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:106)
> at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
> at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
> at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97)
> at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:178)
> at org.hibernate.hql.internal.ast.exec.BasicExecutor.doExecute(BasicExecutor.java:100)
> at org.hibernate.hql.internal.ast.exec.BasicExecutor.execute(BasicExecutor.java:59)
> at org.hibernate.hql.internal.ast.exec.DeleteExecutor.execute(DeleteExecutor.java:110)
> at org.hibernate.hql.internal.ast.QueryTranslatorImpl.executeUpdate(QueryTranslatorImpl.java:450)
> at org.hibernate.engine.query.spi.HQLQueryPlan.performExecuteUpdate(HQLQueryPlan.java:374)
> at org.hibernate.internal.SessionImpl.executeUpdate(SessionImpl.java:1514)
> at org.hibernate.query.internal.AbstractProducedQuery.doExecuteUpdate(AbstractProducedQuery.java:1526)
> at org.hibernate.query.internal.AbstractProducedQuery.executeUpdate(AbstractProducedQuery.java:1504)
> ... 134 more
> Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at or near ")"
> Position: 105
> at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2477)
> at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2190)
> at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:300)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354)
> at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:169)
> at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:136)
> at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:537)
> at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:175)
> ... 142 more
> {code}
> Since you are doing expensive locking operations in all batch operations, I propose to change is code part to:
> {code:java}
> if (!sharedMods.getMarshalledEntries().isEmpty()) {
> persistenceManager.writeBatchToAllNonTxStores(sharedMods.getMarshalledEntries(), BOTH, 0);
> }
> if (!nonSharedMods.getMarshalledEntries().isEmpty()) {
> persistenceManager.writeBatchToAllNonTxStores(nonSharedMods.getMarshalledEntries(), PRIVATE, 0);
> }
> if (!sharedMods.getKeysToRemove().isEmpty()) {
> persistenceManager.deleteBatchFromAllNonTxStores(sharedMods.getKeysToRemove(), BOTH, 0);
> }
> if (!nonSharedMods.getKeysToRemove().isEmpty()) {
> persistenceManager.deleteBatchFromAllNonTxStores(nonSharedMods.getKeysToRemove(), PRIVATE, 0);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 1 month
[JBoss JIRA] (ISPN-7749) A node leaving during rebalance causes extra segment transfers
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-7749?page=com.atlassian.jira.plugin.... ]
Dan Berindei resolved ISPN-7749.
--------------------------------
Fix Version/s: (was: 9.3.0.Final)
Resolution: Duplicate Issue
Duplicate of ISPN-4587
> A node leaving during rebalance causes extra segment transfers
> --------------------------------------------------------------
>
> Key: ISPN-7749
> URL: https://issues.jboss.org/browse/ISPN-7749
> Project: Infinispan
> Issue Type: Bug
> Components: Core, State Transfer
> Affects Versions: 9.0.0.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
>
> If a node leaves during a rebalance, the only change is that other nodes will no longer request segments from that node. Otherwise the rebalance proceeds as usual, and at the end a node may delete its copy of a segment even if that leaves less than {{numOwners}} copies in the cluster.
> When the leaver is the joiner, a 2nd rebalance is very likely to return to the initial CH, but only after transferring some segments twice:
> {noformat}
> Rebalance starts: current_owners(s) = AB, pending_owners(s) = AC
> C leaves: current_owners(s) = AB, pending_owners(s) = A
> Rebalance finishes: current_owners(s) = A, pending_owners(s) = A
> 2nd rebalance starts: current_owners(s) = A, pending_owners(s) = AB
> {noformat}
> Even if the leaver is one of the old owners and the 2 segment transfers are necessary, the cluster stays for too long with less than {{numOwners}} copies of the segment:
> {noformat}
> Rebalance starts: current_owners(s) = AB, pending_owners(s) = AC
> A leaves: current_owners(s) = B, pending_owners(s) = C
> C transfers segment s from B
> Rebalance finishes: current_owners(s) = C, pending_owners(s) = C
> 2nd rebalance starts: current_owners(s) = C, pending_owners(s) = DC
> {noformat}
> We can fix this by making the pending CH a union of the current and pending CH whenever a node leavers, and only removing extra segment copies after the 2nd rebalance.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 1 month