[JBoss JIRA] (ISPN-6368) Race condition in InternalCacheRegistryImpl.registerInternalCache
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-6368?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-6368:
----------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> Race condition in InternalCacheRegistryImpl.registerInternalCache
> -----------------------------------------------------------------
>
> Key: ISPN-6368
> URL: https://issues.jboss.org/browse/ISPN-6368
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 8.1.0.Final, 8.2.0.Final
> Reporter: Adrian Nistor
> Assignee: Adrian Nistor
> Fix For: 9.0.0.Final, 9.0.0.Alpha1, 8.1.3.Final, 8.2.1.Final
>
>
> Access to internalCaches set can cause race conditions and errors related to duplicate cache definitions can be mistakenly reported.
> {code}
> 17:51:40,668 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-3) MSC000001: Failed to start service jboss.datagrid-infinispan.clustered.disttestcache: org.jboss.msc.service.StartException in service jboss.datagrid-infinispan.clustered.disttestcache: Failed to start service
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1904)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.commons.CacheConfigurationException: ISPN000358: A cache configuration named ___query_known_classes already exists. This cannot be configured externally by the user.
> at org.infinispan.registry.impl.InternalCacheRegistryImpl.registerInternalCache(InternalCacheRegistryImpl.java:56)
> at org.infinispan.registry.impl.InternalCacheRegistryImpl.registerInternalCache(InternalCacheRegistryImpl.java:49)
> at org.infinispan.query.backend.QueryKnownClasses.startInternalCache(QueryKnownClasses.java:219)
> at org.infinispan.query.backend.QueryKnownClasses.start(QueryKnownClasses.java:119)
> at org.infinispan.query.backend.QueryInterceptor.start(QueryInterceptor.java:129)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:859)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:628)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:617)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:542)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:238)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:849)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:635)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:585)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:451)
> at org.infinispan.manager.impl.AbstractDelegatingEmbeddedCacheManager.getCache(AbstractDelegatingEmbeddedCacheManager.java:133)
> at org.infinispan.server.infinispan.SecurityActions$5.run(SecurityActions.java:130)
> at org.infinispan.server.infinispan.SecurityActions$5.run(SecurityActions.java:127)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:63)
> at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:135)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:86)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)
> ... 3 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-6402) Default GMS.join_timeout is too long
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-6402?page=com.atlassian.jira.plugin.... ]
Dan Berindei edited comment on ISPN-6402 at 3/18/16 9:42 AM:
-------------------------------------------------------------
{{jgroups-defaults.xml}} uses the {{jgroups.join_timeout}} system property, if present. Since we don't expect long garbage collection or network problems in our tests, we can use it to set an even lower join_timeout in the tests.
was (Author: dan.berindei):
{{jgroups-defaults.xml}} uses the {[jgroups.join_timeout}} system property, if present. Since we don't expect long garbage collection or network problems in our tests, we can use it to set an even lower join_timeout in the tests.
> Default GMS.join_timeout is too long
> ------------------------------------
>
> Key: ISPN-6402
> URL: https://issues.jboss.org/browse/ISPN-6402
> Project: Infinispan
> Issue Type: Task
> Components: Core, Server, Test Suite - Server
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Minor
>
> {{GMS.join_timeout}} is used by JGroups for two purposes:
> # Wait for {{FIND_INITIAL_MBRS}} responses. If other nodes are running, but they don't answer within {{join_timeout}} ms, the node will start a new partition by itself.
> # If no other nodes are running when the request is sent, but another node starts and sends its own discovery request within {{join_timeout}}, the initial cluster view will contain both nodes, but this isn't really useful in Infinispan (we have {{gcb.transport().initialClusterSize()}} instead).
> # Once a coordinator is located, the node sends a join request and waits for a response for {{join_timeout}} ms. After a timeout, the node re-sends the join request (up to a maximum of {{max_join_attempts}}, which defaults to 10).
> The default {{GMS.join_timeout}} in Infinispan is 15000, vs. 2000 in JGroups (actually 3000 in {{GMS}} itself, but 2000 in the example configurations).
> The higher timeout will only help us when a node is running, but it's inaccessible (e.g. because of a long GC) at the exact time a node is joining. I'd argue that applications that can tolerate multi-second pauses would be better served by {{gcb.transport().initialClusterSize(2)}} and/or an external discovery mechanism (e.g. {{FILE_PING}}, or something based on the WildFly domain controller). For most applications, the current default means just a 15s delay every time the cluster is (re)started.
> In particular, because our integration tests use the default configuration, it means a delay of 15s for every test that starts a cluster.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-6132) Mark Spring dependencies are provided
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-6132?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-6132:
----------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 9.0.0.Final
9.0.0.Alpha1
Resolution: Done
> Mark Spring dependencies are provided
> -------------------------------------
>
> Key: ISPN-6132
> URL: https://issues.jboss.org/browse/ISPN-6132
> Project: Infinispan
> Issue Type: Enhancement
> Components: Spring Integration
> Reporter: Sebastian Łaskawiec
> Assignee: Sebastian Łaskawiec
> Fix For: 9.0.0.Final, 9.0.0.Alpha1
>
>
> After splitting Spring into 2 modules (embedded and remote) and introducing Uber jars, it is very easy to hit an error like this:
> {code}
> java.lang.NoSuchMethodError: org.infinispan.commons.logging.BasicLogFactory.getLog(Ljava/lang/Class;)Linfinispan/org/jboss/logging/BasicLogger;
> {code}
> It turns out that Wildfly ships its own JBoss Modules [1] and it conflicts the relocated one from Uber Jars.
> The easiest fix for it is to make all Spring module dependencies provided.
> [1] https://docs.jboss.org/author/display/AS7/Implicit+module+dependencies+fo...
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-6402) Default GMS.join_timeout is too long
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-6402?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-6402:
------------------------------------
{{jgroups-defaults.xml}} uses the {[jgroups.join_timeout}} system property, if present. Since we don't expect long garbage collection or network problems in our tests, we can use it to set an even lower join_timeout in the tests.
> Default GMS.join_timeout is too long
> ------------------------------------
>
> Key: ISPN-6402
> URL: https://issues.jboss.org/browse/ISPN-6402
> Project: Infinispan
> Issue Type: Task
> Components: Core, Server, Test Suite - Server
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Minor
>
> {{GMS.join_timeout}} is used by JGroups for two purposes:
> # Wait for {{FIND_INITIAL_MBRS}} responses. If other nodes are running, but they don't answer within {{join_timeout}} ms, the node will start a new partition by itself.
> # If no other nodes are running when the request is sent, but another node starts and sends its own discovery request within {{join_timeout}}, the initial cluster view will contain both nodes, but this isn't really useful in Infinispan (we have {{gcb.transport().initialClusterSize()}} instead).
> # Once a coordinator is located, the node sends a join request and waits for a response for {{join_timeout}} ms. After a timeout, the node re-sends the join request (up to a maximum of {{max_join_attempts}}, which defaults to 10).
> The default {{GMS.join_timeout}} in Infinispan is 15000, vs. 2000 in JGroups (actually 3000 in {{GMS}} itself, but 2000 in the example configurations).
> The higher timeout will only help us when a node is running, but it's inaccessible (e.g. because of a long GC) at the exact time a node is joining. I'd argue that applications that can tolerate multi-second pauses would be better served by {{gcb.transport().initialClusterSize(2)}} and/or an external discovery mechanism (e.g. {{FILE_PING}}, or something based on the WildFly domain controller). For most applications, the current default means just a 15s delay every time the cluster is (re)started.
> In particular, because our integration tests use the default configuration, it means a delay of 15s for every test that starts a cluster.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-4696) SuspectException kills operation on ClusterRegistryImpl
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-4696?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration updated ISPN-4696:
------------------------------------------
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=1309181
Bugzilla Update: Perform
> SuspectException kills operation on ClusterRegistryImpl
> -------------------------------------------------------
>
> Key: ISPN-4696
> URL: https://issues.jboss.org/browse/ISPN-4696
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Reporter: Sanne Grinovero
> Assignee: William Burns
> Priority: Blocker
> Fix For: 7.0.0.CR1
>
>
> Apparently a SuspectException is not just "cosmetic" as mentioned in ISPN-2577 but can have critical effects on the operation of caches:
> {noformat}
> 2014-09-07 22:20:24,268 ERROR [InvocationContextInterceptor] (remote-thread-IndexManagerStressTest-NodeB-p9-t6) ISPN000136: Execution error
> org.infinispan.remoting.transport.jgroups.SuspectException: Suspected member: IndexManagerStressTest-NodeD-308
> at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:73)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:562)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:281)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:213)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:197)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:109)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:78)
> at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:87)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:36)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:124)
> at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:111)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:97)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:42)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:217)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:147)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:75)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:97)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:112)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:75)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:97)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:66)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:97)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> at org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:121)
> at org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:104)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:44)
> at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:208)
> at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:220)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:64)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:80)
> at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1540)
> at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1497)
> at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:968)
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:960)
> at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:405)
> at org.infinispan.registry.impl.ClusterRegistryImpl.put(ClusterRegistryImpl.java:69)
> at org.infinispan.query.backend.ReadIntensiveClusterRegistryWrapper.put(ReadIntensiveClusterRegistryWrapper.java:58)
> at org.infinispan.query.backend.SearchFactoryHandler.updateClusterRegistry(SearchFactoryHandler.java:61)
> at org.infinispan.query.backend.SearchFactoryHandler.handleOnDemandRegistration(SearchFactoryHandler.java:56)
> at org.infinispan.query.backend.SearchFactoryHandler.updateKnownTypesIfNeeded(SearchFactoryHandler.java:42)
> at org.infinispan.query.backend.QueryInterceptor.updateKnownTypesIfNeeded(QueryInterceptor.java:248)
> at org.infinispan.query.backend.QueryInterceptor.processPutKeyValueCommand(QueryInterceptor.java:435)
> at org.infinispan.query.backend.QueryInterceptor.visitPutKeyValueCommand(QueryInterceptor.java:157)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:46)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:166)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:95)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:148)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:134)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:112)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:75)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:39)
> at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:96)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:51)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:185)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2014-09-07 22:20:24,272 ERROR [TransactionCoordinator] (remote-thread-IndexManagerStressTest-NodeB-p9-t6) ISPN000255: Error while processing prepare
> org.infinispan.remoting.transport.jgroups.SuspectException: Suspected member: IndexManagerStressTest-NodeD-308
> at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:73)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:562)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:281)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:213)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:197)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:109)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:78)
> at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:87)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:36)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:124)
> at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:111)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:97)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:42)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:217)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:147)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:75)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:97)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:112)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:75)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:97)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:66)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:97)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> at org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:121)
> at org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:104)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:44)
> at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:208)
> at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:220)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:64)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:80)
> at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1540)
> at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1497)
> at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:968)
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:960)
> at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:405)
> at org.infinispan.registry.impl.ClusterRegistryImpl.put(ClusterRegistryImpl.java:69)
> at org.infinispan.query.backend.ReadIntensiveClusterRegistryWrapper.put(ReadIntensiveClusterRegistryWrapper.java:58)
> at org.infinispan.query.backend.SearchFactoryHandler.updateClusterRegistry(SearchFactoryHandler.java:61)
> at org.infinispan.query.backend.SearchFactoryHandler.handleOnDemandRegistration(SearchFactoryHandler.java:56)
> at org.infinispan.query.backend.SearchFactoryHandler.updateKnownTypesIfNeeded(SearchFactoryHandler.java:42)
> at org.infinispan.query.backend.QueryInterceptor.updateKnownTypesIfNeeded(QueryInterceptor.java:248)
> at org.infinispan.query.backend.QueryInterceptor.processPutKeyValueCommand(QueryInterceptor.java:435)
> at org.infinispan.query.backend.QueryInterceptor.visitPutKeyValueCommand(QueryInterceptor.java:157)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:46)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:166)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:95)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:148)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:134)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:112)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:75)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:39)
> at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:96)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:51)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:185)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2014-09-07 22:20:24,278 ERROR [DummyTransaction] (remote-thread-IndexManagerStressTest-NodeB-p9-t6) ISPN000109: beforeCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=[org.infinispan.registry.ScopedKey@f4bc386d], topologyId=6, stateTransferFlag=null} org.infinispan.transaction.synchronization.SyncLocalTransaction@2} org.infinispan.transaction.synchronization.SynchronizationAdapter@21
> org.infinispan.commons.CacheException: Could not prepare.
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:46)
> at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:208)
> at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:220)
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:64)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:80)
> at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1540)
> at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1497)
> at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:968)
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:960)
> at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:405)
> at org.infinispan.registry.impl.ClusterRegistryImpl.put(ClusterRegistryImpl.java:69)
> at org.infinispan.query.backend.ReadIntensiveClusterRegistryWrapper.put(ReadIntensiveClusterRegistryWrapper.java:58)
> at org.infinispan.query.backend.SearchFactoryHandler.updateClusterRegistry(SearchFactoryHandler.java:61)
> at org.infinispan.query.backend.SearchFactoryHandler.handleOnDemandRegistration(SearchFactoryHandler.java:56)
> at org.infinispan.query.backend.SearchFactoryHandler.updateKnownTypesIfNeeded(SearchFactoryHandler.java:42)
> at org.infinispan.query.backend.QueryInterceptor.updateKnownTypesIfNeeded(QueryInterceptor.java:248)
> at org.infinispan.query.backend.QueryInterceptor.processPutKeyValueCommand(QueryInterceptor.java:435)
> at org.infinispan.query.backend.QueryInterceptor.visitPutKeyValueCommand(QueryInterceptor.java:157)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:46)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:166)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:95)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:148)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:134)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:112)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:75)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:39)
> at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:96)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:51)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:185)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: javax.transaction.xa.XAException
> at org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:141)
> at org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:104)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:44)
> ... 51 more
> Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: Suspected member: IndexManagerStressTest-NodeD-308
> at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:73)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:562)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:281)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:213)
> at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:197)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:109)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:78)
> at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:87)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:36)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:124)
> at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:111)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:97)
> at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:42)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:217)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:147)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:75)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:97)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:112)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:75)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:97)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:66)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:97)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> at org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:121)
> ... 53 more
> 2014-09-07 22:20:24,280 WARN [CacheImpl] (remote-thread-IndexManagerStressTest-NodeB-p9-t6) ISPN000160: Could not complete injected transaction.
> javax.transaction.RollbackException: Exception rolled back, status is: 9
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:67)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:80)
> at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1540)
> at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1497)
> at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:968)
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:960)
> at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:405)
> at org.infinispan.registry.impl.ClusterRegistryImpl.put(ClusterRegistryImpl.java:69)
> at org.infinispan.query.backend.ReadIntensiveClusterRegistryWrapper.put(ReadIntensiveClusterRegistryWrapper.java:58)
> at org.infinispan.query.backend.SearchFactoryHandler.updateClusterRegistry(SearchFactoryHandler.java:61)
> at org.infinispan.query.backend.SearchFactoryHandler.handleOnDemandRegistration(SearchFactoryHandler.java:56)
> at org.infinispan.query.backend.SearchFactoryHandler.updateKnownTypesIfNeeded(SearchFactoryHandler.java:42)
> at org.infinispan.query.backend.QueryInterceptor.updateKnownTypesIfNeeded(QueryInterceptor.java:248)
> at org.infinispan.query.backend.QueryInterceptor.processPutKeyValueCommand(QueryInterceptor.java:435)
> at org.infinispan.query.backend.QueryInterceptor.visitPutKeyValueCommand(QueryInterceptor.java:157)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:46)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:166)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:95)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:148)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:134)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:112)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:75)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:39)
> at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:96)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:51)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:185)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2014-09-07 22:20:24,281 ERROR [InvocationContextInterceptor] (remote-thread-IndexManagerStressTest-NodeB-p9-t6) ISPN000136: Execution error
> org.infinispan.commons.CacheException: Could not commit implicit transaction
> at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1543)
> at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1497)
> at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:968)
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:960)
> at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:405)
> at org.infinispan.registry.impl.ClusterRegistryImpl.put(ClusterRegistryImpl.java:69)
> at org.infinispan.query.backend.ReadIntensiveClusterRegistryWrapper.put(ReadIntensiveClusterRegistryWrapper.java:58)
> at org.infinispan.query.backend.SearchFactoryHandler.updateClusterRegistry(SearchFactoryHandler.java:61)
> at org.infinispan.query.backend.SearchFactoryHandler.handleOnDemandRegistration(SearchFactoryHandler.java:56)
> at org.infinispan.query.backend.SearchFactoryHandler.updateKnownTypesIfNeeded(SearchFactoryHandler.java:42)
> at org.infinispan.query.backend.QueryInterceptor.updateKnownTypesIfNeeded(QueryInterceptor.java:248)
> at org.infinispan.query.backend.QueryInterceptor.processPutKeyValueCommand(QueryInterceptor.java:435)
> at org.infinispan.query.backend.QueryInterceptor.visitPutKeyValueCommand(QueryInterceptor.java:157)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:46)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:166)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:95)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:148)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:134)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:112)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:75)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:39)
> at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:96)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:51)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:185)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: javax.transaction.RollbackException: Exception rolled back, status is: 9
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:67)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:80)
> at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1540)
> ... 46 more
> 2014-09-07 22:20:24,282 ERROR [InboundInvocationHandlerImpl] (remote-thread-IndexManagerStressTest-NodeB-p9-t6) ISPN000260: Exception executing command
> org.infinispan.commons.CacheException: Could not commit implicit transaction
> at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1543)
> at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1497)
> at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:968)
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:960)
> at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:405)
> at org.infinispan.registry.impl.ClusterRegistryImpl.put(ClusterRegistryImpl.java:69)
> at org.infinispan.query.backend.ReadIntensiveClusterRegistryWrapper.put(ReadIntensiveClusterRegistryWrapper.java:58)
> at org.infinispan.query.backend.SearchFactoryHandler.updateClusterRegistry(SearchFactoryHandler.java:61)
> at org.infinispan.query.backend.SearchFactoryHandler.handleOnDemandRegistration(SearchFactoryHandler.java:56)
> at org.infinispan.query.backend.SearchFactoryHandler.updateKnownTypesIfNeeded(SearchFactoryHandler.java:42)
> at org.infinispan.query.backend.QueryInterceptor.updateKnownTypesIfNeeded(QueryInterceptor.java:248)
> at org.infinispan.query.backend.QueryInterceptor.processPutKeyValueCommand(QueryInterceptor.java:435)
> at org.infinispan.query.backend.QueryInterceptor.visitPutKeyValueCommand(QueryInterceptor.java:157)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:46)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:166)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:95)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:148)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:134)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:112)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:75)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:39)
> at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:96)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:51)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:185)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: javax.transaction.RollbackException: Exception rolled back, status is: 9
> at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:67)
> at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:80)
> at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1540)
> ... 46 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-6402) Default GMS.join_timeout is too long
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-6402?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-6402:
-------------------------------
Status: Open (was: New)
> Default GMS.join_timeout is too long
> ------------------------------------
>
> Key: ISPN-6402
> URL: https://issues.jboss.org/browse/ISPN-6402
> Project: Infinispan
> Issue Type: Task
> Components: Core, Server, Test Suite - Server
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Minor
>
> {{GMS.join_timeout}} is used by JGroups for two purposes:
> # Wait for {{FIND_INITIAL_MBRS}} responses. If other nodes are running, but they don't answer within {{join_timeout}} ms, the node will start a new partition by itself.
> # If no other nodes are running when the request is sent, but another node starts and sends its own discovery request within {{join_timeout}}, the initial cluster view will contain both nodes, but this isn't really useful in Infinispan (we have {{gcb.transport().initialClusterSize()}} instead).
> # Once a coordinator is located, the node sends a join request and waits for a response for {{join_timeout}} ms. After a timeout, the node re-sends the join request (up to a maximum of {{max_join_attempts}}, which defaults to 10).
> The default {{GMS.join_timeout}} in Infinispan is 15000, vs. 2000 in JGroups (actually 3000 in {{GMS}} itself, but 2000 in the example configurations).
> The higher timeout will only help us when a node is running, but it's inaccessible (e.g. because of a long GC) at the exact time a node is joining. I'd argue that applications that can tolerate multi-second pauses would be better served by {{gcb.transport().initialClusterSize(2)}} and/or an external discovery mechanism (e.g. {{FILE_PING}}, or something based on the WildFly domain controller). For most applications, the current default means just a 15s delay every time the cluster is (re)started.
> In particular, because our integration tests use the default configuration, it means a delay of 15s for every test that starts a cluster.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-6275) Double invalidate of invalid Hot Rod connections
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-6275?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-6275:
----------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 8.2.1.Final
Resolution: Done
> Double invalidate of invalid Hot Rod connections
> ------------------------------------------------
>
> Key: ISPN-6275
> URL: https://issues.jboss.org/browse/ISPN-6275
> Project: Infinispan
> Issue Type: Bug
> Components: Remote Protocols
> Affects Versions: 6.0.2.Final
> Reporter: Dennis Reed
> Assignee: Galder Zamarreño
> Fix For: 9.0.0.Final, 8.2.1.Final
>
>
> When there's a problem with a Hot Rod operation, RetryOnFailureOperation invalidates the connection twice (once in a catch block, and once in a finally block).
> This causes the GenericKeyedObjectPool counts to get off, and anything relying on that count (such as the maxTotal configuration for the pool) to break.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years