[JBoss JIRA] (ISPN-5498) ClientClusterFailoverEventsTest.testEventReplayWithAndWithoutStateAfterFailover random failures
by Tristan Tarrant (Jira)
[ https://issues.redhat.com/browse/ISPN-5498?page=com.atlassian.jira.plugin... ]
Tristan Tarrant updated ISPN-5498:
----------------------------------
Fix Version/s: (was: 10.1.0.Final)
> ClientClusterFailoverEventsTest.testEventReplayWithAndWithoutStateAfterFailover random failures
> -----------------------------------------------------------------------------------------------
>
> Key: ISPN-5498
> URL: https://issues.redhat.com/browse/ISPN-5498
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite
> Affects Versions: 7.2.1.Final
> Reporter: Dan Berindei
> Priority: Blocker
> Labels: testsuite_stability
> Attachments: pr_wburns_ISPN-10364_blocking_queue_20190701-1243_ClientClusterFailoverEventsTest-infinispan-client-hotrod.log.gz
>
>
> It appears the cluster event notifier wants to send some events to a node that no longer exists, and the exception is propagated all the way to the HR client:
> {noformat}
> 12:04:03,088 ERROR (HotRodServerWorker-78-1:) [InvocationContextInterceptor] ISPN000136: Execution errorjava.lang.IllegalArgumentException: Target node ClientClusterFailoverEventsTest-NodeA-33510 is not a cluster member, members are [ClientClusterFailoverEventsTest-NodeB-14182]
> at org.infinispan.distexec.DefaultExecutorService.submit(DefaultExecutorService.java:414)
> at org.infinispan.distexec.DefaultExecutorService.submit(DefaultExecutorService.java:403)
> at org.infinispan.distexec.DistributedExecutionCompletionService.submit(DistributedExecutionCompletionService.java:178)
> at org.infinispan.notifications.cachelistener.cluster.impl.BatchingClusterEventManagerImpl$UnicastEventContext.sendToTargets(BatchingClusterEventManagerImpl.java:97)
> at org.infinispan.notifications.cachelistener.cluster.impl.BatchingClusterEventManagerImpl.sendEvents(BatchingClusterEventManagerImpl.java:50)
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryCreated(CacheNotifierImpl.java:292)
> at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.notifyCommitEntry(ClusteringDependentLogic.java:138)
> at org.infinispan.interceptors.locking.ClusteringDependentLogic$DistributionLogic.commitSingleEntry(ClusteringDependentLogic.java:493)
> at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.commitEntry(ClusteringDependentLogic.java:108)
> at org.infinispan.interceptors.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:367)
> at org.infinispan.interceptors.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:545)
> at org.infinispan.interceptors.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:344)
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:418)
> at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:449)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:195)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:88)
> at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:40)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:55)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:44)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:324)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:256)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:115)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:191)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:177)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:44)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1617)
> at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1097)
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1089)
> at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:522)
> at org.infinispan.server.hotrod.CacheDecodeContext.put(CacheDecodeContext.scala:216)
> at org.infinispan.server.hotrod.HotRodDecoder.org$infinispan$server$hotrod$HotRodDecoder$$decodeValue(HotRodDecoder.scala:132)
> at org.infinispan.server.hotrod.HotRodDecoder$$anonfun$decode$1.apply$mcV$sp(HotRodDecoder.scala:50)
> at org.infinispan.server.hotrod.HotRodDecoder.wrapSecurity(HotRodDecoder.scala:208)
> at org.infinispan.server.hotrod.HotRodDecoder.decode(HotRodDecoder.scala:45)
> org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for message id[922] returned server error (status=0x85): java.lang.IllegalArgumentException: Target node ClientClusterFailoverEventsTest-NodeA-33510 is not a cluster member, members are [ClientClusterFailoverEventsTest-NodeB-14182]
> at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:336)
> at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:126)
> at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:112)
> at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)
> at org.infinispan.client.hotrod.impl.operations.AbstractKeyValueOperation.sendPutOperation(AbstractKeyValueOperation.java:57)
> at org.infinispan.client.hotrod.impl.operations.PutOperation.executeOperation(PutOperation.java:31)
> at org.infinispan.client.hotrod.impl.operations.PutOperation.executeOperation(PutOperation.java:20)
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:52)
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:247)
> at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:79)
> at org.infinispan.client.hotrod.event.ClientClusterFailoverEventsTest.testEventReplayWithAndWithoutStateAfterFailover(ClientClusterFailoverEventsTest.java:59)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:38)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:382)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 4 months
[JBoss JIRA] (ISPN-5357) Hot Rod protocol .NEXT
by Tristan Tarrant (Jira)
[ https://issues.redhat.com/browse/ISPN-5357?page=com.atlassian.jira.plugin... ]
Tristan Tarrant reassigned ISPN-5357:
-------------------------------------
Assignee: (was: Galder Zamarreno)
> Hot Rod protocol .NEXT
> ----------------------
>
> Key: ISPN-5357
> URL: https://issues.redhat.com/browse/ISPN-5357
> Project: Infinispan
> Issue Type: Enhancement
> Reporter: Galder Zamarreno
> Priority: Major
>
> Infinispan should expose a single endpoint which should behave as follows:
> - by default offer an HTTP/1.1 RESTftul interface
> - through HTTP upgrade allow switching to better protocols
> - support an HTTP/2 RESTful interface
> - support Hot Rod 3.0, which would be a gRPC protocol with additional L2 and L3 intelligence for capable clients
> *UPDATE (6/10/2015)*: Hot Rod 3.0 could become a HTTP REST protocol taking advantage of existing infrastructure to deal with HTTP requests efficiently, and if it was based on HTTP 2.0, we could have binary data inside of it. Performance should be on par. Text mode could be available for debug or demos. This essentially would mean REST and Hot Rod would merge into one.
> A binary protocol rethink is needed to take better advantage of CPUs and buffering. This is based on the ideas of Martin Thompson's Simple Binary Encoding [blog post|http://mechanical-sympathy.blogspot.cz/2014/05/simple-binary-encoding.html]
> In essence, we need to move Hot Rod protocol towards having mostly fixed size fields, hence getting rid of `vInt` and `vLong` formats which although safe some bytes, they hinder long stream reading patterns.
> An optional part here would be whether to consider using SBE directly, but this JIRA should be mostly oriented at making sure we have a protocol that can be read fast and efficiently.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 4 months
[JBoss JIRA] (ISPN-7167) ExampleConfigsIT.testTopologyConfigRackAttribute random failures
by Tristan Tarrant (Jira)
[ https://issues.redhat.com/browse/ISPN-7167?page=com.atlassian.jira.plugin... ]
Tristan Tarrant closed ISPN-7167.
---------------------------------
Fix Version/s: (was: 10.1.0.Final)
Resolution: Out of Date
> ExampleConfigsIT.testTopologyConfigRackAttribute random failures
> -----------------------------------------------------------------
>
> Key: ISPN-7167
> URL: https://issues.redhat.com/browse/ISPN-7167
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite
> Affects Versions: 9.0.0.Alpha4
> Reporter: Dan Berindei
> Priority: Critical
> Labels: testsuite_stability
>
> {noformat}
> java.lang.AssertionError: Unexpected number of entries in server1: 0
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.infinispan.server.test.configs.ExampleConfigsIT.verifyTopologyHinting(ExampleConfigsIT.java:482)
> at org.infinispan.server.test.configs.ExampleConfigsIT.testTopologyConfigRackAttribute(ExampleConfigsIT.java:419)
> {noformat}
> The test is also inconsistent about the names of the servers, sometimes it starts counting from 0 and sometimes it starts counting from 1.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 4 months
[JBoss JIRA] (ISPN-5357) Hot Rod protocol .NEXT
by Tristan Tarrant (Jira)
[ https://issues.redhat.com/browse/ISPN-5357?page=com.atlassian.jira.plugin... ]
Tristan Tarrant updated ISPN-5357:
----------------------------------
Fix Version/s: (was: 10.1.0.Final)
> Hot Rod protocol .NEXT
> ----------------------
>
> Key: ISPN-5357
> URL: https://issues.redhat.com/browse/ISPN-5357
> Project: Infinispan
> Issue Type: Enhancement
> Reporter: Galder Zamarreno
> Assignee: Galder Zamarreno
> Priority: Major
>
> Infinispan should expose a single endpoint which should behave as follows:
> - by default offer an HTTP/1.1 RESTftul interface
> - through HTTP upgrade allow switching to better protocols
> - support an HTTP/2 RESTful interface
> - support Hot Rod 3.0, which would be a gRPC protocol with additional L2 and L3 intelligence for capable clients
> *UPDATE (6/10/2015)*: Hot Rod 3.0 could become a HTTP REST protocol taking advantage of existing infrastructure to deal with HTTP requests efficiently, and if it was based on HTTP 2.0, we could have binary data inside of it. Performance should be on par. Text mode could be available for debug or demos. This essentially would mean REST and Hot Rod would merge into one.
> A binary protocol rethink is needed to take better advantage of CPUs and buffering. This is based on the ideas of Martin Thompson's Simple Binary Encoding [blog post|http://mechanical-sympathy.blogspot.cz/2014/05/simple-binary-encoding.html]
> In essence, we need to move Hot Rod protocol towards having mostly fixed size fields, hence getting rid of `vInt` and `vLong` formats which although safe some bytes, they hinder long stream reading patterns.
> An optional part here would be whether to consider using SBE directly, but this JIRA should be mostly oriented at making sure we have a protocol that can be read fast and efficiently.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 4 months
[JBoss JIRA] (ISPN-11101) Purge on JDBC shared stores can cause deadlocks
by Ryan Emerson (Jira)
[ https://issues.redhat.com/browse/ISPN-11101?page=com.atlassian.jira.plugi... ]
Ryan Emerson updated ISPN-11101:
--------------------------------
Git Pull Request: https://github.com/infinispan/infinispan/pull/7692, https://github.com/infinispan/infinispan/pull/7701 (was: https://github.com/infinispan/infinispan/pull/7692)
> Purge on JDBC shared stores can cause deadlocks
> -----------------------------------------------
>
> Key: ISPN-11101
> URL: https://issues.redhat.com/browse/ISPN-11101
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 9.4.17.Final, 10.1.0.CR1
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Priority: Major
> Fix For: 10.1.0.Final, 9.4.18.Final
>
>
> ISPN-10337 ensured that the JdbcStringBasedStore correctly acquired the locks of expired rows during the purging of store entries, however it has exposed an issue with shared stores. When the jdbc store is shared, the coordinator locks the rows of the affected entries and releases them once they have all been removed as part of the purge transaction. However, the call to {{ExpirationManager::handleInStoreExpiration}} also sends a {{RemoveExpiredCommand}} to ensure that entries are removed from memory. This results in a {{java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction}} being thrown by the store's delete method when the purge process is still in progress, because the expired rows are still locked and the purge process cannot complete until the {{RemoveExpiredCommand}} has been executed.
> In summary:
> # purge locks all the rows
> # purge sends a RemoveExpiredCommand and waits for it to complete
> # RemoveExpiredCommand tries to remove the row but it can't as it is locked
> Solution, send the {{RemoveExpiredCommand}} with the {{SKIP_CACHE_STORE}} flag when a store is shared,so that it only removes entries from memory.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 4 months
[JBoss JIRA] (ISPN-11110) JdbcStringBasedStore Purge should not use FOR UPDATE to avoid excessive locking
by Ryan Emerson (Jira)
Ryan Emerson created ISPN-11110:
-----------------------------------
Summary: JdbcStringBasedStore Purge should not use FOR UPDATE to avoid excessive locking
Key: ISPN-11110
URL: https://issues.redhat.com/browse/ISPN-11110
Project: Infinispan
Issue Type: Bug
Components: Loaders and Stores
Affects Versions: 10.1.0.CR1
Reporter: Ryan Emerson
Assignee: Ryan Emerson
Fix For: 11.0.0.Final
https://issues.redhat.com/browse/ISPN-10337 Added the FOR UPDATE to the select query used by the purge method in order to ensure that the purge method would lock all of the affected rows so that no subsequent updates to a key could be incorrectly removed by the purge method.
However, if the purge does a select for update and does a single commit at the end of the purge, that will block write operations on all the expired keys for the entire duration of the purge. The clustered purge listener does cache.removeLifespanExpired(key, null, null).join(), so the entire duration of the purge could mean a really long time.
Instead we should utilise a normal select query followed by a 'delete from table where id = ? and timestamp = ?' to prevent accidentally deleting an update.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 4 months
[JBoss JIRA] (ISPN-11109) Deprecate and remove usages of state-transfer-executor
by Dan Berindei (Jira)
Dan Berindei created ISPN-11109:
-----------------------------------
Summary: Deprecate and remove usages of state-transfer-executor
Key: ISPN-11109
URL: https://issues.redhat.com/browse/ISPN-11109
Project: Infinispan
Issue Type: Enhancement
Components: Configuration, Core
Affects Versions: 10.1.0.CR1
Reporter: Dan Berindei
Assignee: Dan Berindei
Fix For: 10.1.0.Final
With ISPN-10310, {{StateConsumerImpl}} applies state asynchronously, so a thread pool specifically for applying state is no longer necessary.
There are some other users remaining, but they can be migrated to the transport executor.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 4 months