[Red Hat JIRA] (ISPN-12646) IllegalArgumentException when doing Rolling Upgrades on transactional caches
by Gustavo Fernandes (Jira)
[ https://issues.redhat.com/browse/ISPN-12646?page=com.atlassian.jira.plugi... ]
Work on ISPN-12646 started by Gustavo Fernandes.
------------------------------------------------
> IllegalArgumentException when doing Rolling Upgrades on transactional caches
> ----------------------------------------------------------------------------
>
> Key: ISPN-12646
> URL: https://issues.redhat.com/browse/ISPN-12646
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 11.0.9.Final
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
> Priority: Major
>
> {noformat}
> Caused by: java.lang.IllegalArgumentException: Cannot create a transactional context without a valid Transaction instance.
> at org.infinispan.context.impl.TransactionalInvocationContextFactory.createInvocationContext(TransactionalInvocationContextFactory.java:63)
> [...]
> at org.infinispan.cache.impl.DecoratedCache.putIfAbsent(DecoratedCache.java:688)
> at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.putIfAbsent(AbstractDelegatingAdvancedCache.java:328)
> at org.infinispan.cache.impl.EncoderCache.putIfAbsent(EncoderCache.java:450)
> at org.infinispan.persistence.remote.upgrade.MigrationTask.lambda$migrateEntriesWithMetadata$0(MigrationTask.java:128)
> at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1771)
> {noformat}
> The migration component that obtains data from the remote cache does not use transactions to write to the destination cache, and since it runs on a separate thread, it cannot see any ongoing transaction and thus fail to write to the cache.
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
4 years, 8 months
[Red Hat JIRA] (ISPN-12651) CacheEntryCloudEventsTest test failures
by Dan Berindei (Jira)
Dan Berindei created ISPN-12651:
-----------------------------------
Summary: CacheEntryCloudEventsTest test failures
Key: ISPN-12651
URL: https://issues.redhat.com/browse/ISPN-12651
Project: Infinispan
Issue Type: Bug
Components: Integration , Test Suite
Affects Versions: 12.0.0.Final
Reporter: Dan Berindei
Assignee: Dan Berindei
Fix For: 12.1.0.Final
{{jackson-core}} is an optional dependency in {{protostream}}, but it is required for transcoding {{application/x-protostream}} to JSON. Because the dependency is missing, the tests fail with {{OBJECT}} storage:
{noformat}
org.infinispan.util.concurrent.TimeoutException
at org.infinispan.cloudevents.MockKafkaEventSender.completeSend(MockKafkaEventSender.java:44)
at org.infinispan.cloudevents.MockKafkaEventSender.completeSend(MockKafkaEventSender.java:53)
at org.infinispan.cloudevents.CacheEntryCloudEventsTest.testMultiKeyOperations(CacheEntryCloudEventsTest.java:188)
{noformat}
The actual reason is hidden because exceptions in listeners are not handled correctly:
{noformat}
org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.lang.NoClassDefFoundError] while invoking method [public java.util.concurrent.CompletionStage org.infinispan.cloudevents.impl.EntryEventListener.handleCacheEntryEvent(org.infinispan.notifications.cachelistener.event.CacheEntryEvent)] on listener instance: org.infinispan.cloudevents.impl.EntryEventListener@7a524380
at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.lambda$invoke$1(AbstractListenerImpl.java:430)
at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:450)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1817)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1813)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invokeNoChecks(CacheNotifierImpl.java:1808)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1781)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl.doNotifyCreated(CacheNotifierImpl.java:428)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryCreated(CacheNotifierImpl.java:410)
at org.infinispan.notifications.cachelistener.NotifyHelper.entryCommitted(NotifyHelper.java:59)
at org.infinispan.interceptors.locking.ClusteringDependentLogic$DistributionLogic.commitSingleEntry(ClusteringDependentLogic.java:539)
at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.commitEntry(ClusteringDependentLogic.java:198)
at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:651)
at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:904)
at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:631)
at org.infinispan.interceptors.impl.EntryWrappingInterceptor.applyChanges(EntryWrappingInterceptor.java:687)
at org.infinispan.interceptors.impl.EntryWrappingInterceptor.applyAndFixVersion(EntryWrappingInterceptor.java:758)
at org.infinispan.interceptors.InvocationSuccessFunction.apply(InvocationSuccessFunction.java:25)
at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.invokeQueuedHandlers(QueueAsyncInvocationStage.java:124)
at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:87)
at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:33)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2137)
at org.infinispan.util.concurrent.CommandAckCollector$SingleKeyCollector.markReady(CommandAckCollector.java:364)
at org.infinispan.util.concurrent.CommandAckCollector$SingleKeyCollector.backupAck(CommandAckCollector.java:356)
at org.infinispan.util.concurrent.CommandAckCollector.backupAck(CommandAckCollector.java:167)
at org.infinispan.commands.write.BackupAckCommand.ack(BackupAckCommand.java:40)
at org.infinispan.remoting.inboundhandler.TrianglePerCacheInboundInvocationHandler.handleBackupAckCommand(TrianglePerCacheInboundInvocationHandler.java:181)
at org.infinispan.remoting.inboundhandler.TrianglePerCacheInboundInvocationHandler.handle(TrianglePerCacheInboundInvocationHandler.java:96)
at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:164)
at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:111)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1383)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1307)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:131)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1450)
at org.jgroups.JChannel.up(JChannel.java:784)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:913)
at org.jgroups.protocols.FRAG3.up(FRAG3.java:165)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:351)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:359)
at org.jgroups.protocols.tom.TOA.up(TOA.java:119)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:876)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:243)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1049)
at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:772)
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:753)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:405)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:592)
at org.jgroups.protocols.Discovery.up(Discovery.java:300)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1396)
at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/JsonProcessingException
at org.infinispan.protostream.ProtobufUtil.toCanonicalJSON(ProtobufUtil.java:174)
at org.infinispan.protostream.ProtobufUtil.toCanonicalJSON(ProtobufUtil.java:160)
at org.infinispan.encoding.ProtostreamTranscoder.toJsonCascading(ProtostreamTranscoder.java:183)
at org.infinispan.encoding.ProtostreamTranscoder.transcode(ProtostreamTranscoder.java:102)
at org.infinispan.cloudevents.impl.EntryEventListener.writeKey(EntryEventListener.java:165)
at org.infinispan.cloudevents.impl.EntryEventListener.entryEventToKafkaMessage(EntryEventListener.java:133)
at org.infinispan.cloudevents.impl.EntryEventListener.handleCacheEntryEvent(EntryEventListener.java:101)
at jdk.internal.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.lambda$invoke$1(AbstractListenerImpl.java:424)
... 54 more
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.core.JsonProcessingException
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:606)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:168)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
... 65 more
{noformat}
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
4 years, 8 months
[Red Hat JIRA] (ISPN-12630) SIFS does not set the seqId when loading index from disk
by Pedro Ruivo (Jira)
[ https://issues.redhat.com/browse/ISPN-12630?page=com.atlassian.jira.plugi... ]
Pedro Ruivo updated ISPN-12630:
-------------------------------
Fix Version/s: 12.1.0.Final
> SIFS does not set the seqId when loading index from disk
> --------------------------------------------------------
>
> Key: ISPN-12630
> URL: https://issues.redhat.com/browse/ISPN-12630
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 12.0.0.CR1
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
> Priority: Major
> Fix For: 12.1.0.Final
>
>
> {{seqId}} is used to find which key's value is the most recent in the file. However, if the index is loaded from the disk, it is never set and the new entries are stored from {{seqId=1}}.
> Reads/Writes aren't affected but iteration relies on this to discard old values. It has the side effect of (possibly) returning multiple values for the same key.
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
4 years, 8 months
[Red Hat JIRA] (ISPN-12650) ThreadLocalLeakTest random failures (uncaught exception)
by Dan Berindei (Jira)
Dan Berindei created ISPN-12650:
-----------------------------------
Summary: ThreadLocalLeakTest random failures (uncaught exception)
Key: ISPN-12650
URL: https://issues.redhat.com/browse/ISPN-12650
Project: Infinispan
Issue Type: Bug
Components: Test Suite
Affects Versions: 12.0.0.Final
Reporter: Dan Berindei
Assignee: Dan Berindei
Fix For: 12.1.0.Final
{{ThreadLocalLeakTest}} is supposed to invoke a cache operation on a new thread and keep that new thread running, then check that the operation didn't leave any thread locals on that thread. But sometimes the cache manager is stopped before the new thread managed to invoke the cache operation, causing a {{IllegalLifecycleStateException}} that is never caught:
{noformat}
org.infinispan.commons.IllegalLifecycleStateException: Cache container has been stopped and cannot be reused. Recreate the cache container.
at org.infinispan.manager.DefaultCacheManager.assertIsNotTerminated(DefaultCacheManager.java:1074)
at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:517)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:510)
at org.infinispan.util.ThreadLocalLeakTest.lambda$doStuffWithCache$1(ThreadLocalLeakTest.java:103)
at org.infinispan.test.AbstractInfinispanTest$RunnableWrapper.run(AbstractInfinispanTest.java:507)
{noformat}
{noformat}
org.infinispan.commons.IllegalLifecycleStateException: ISPN000324: Cache 'leak' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
at org.infinispan.interceptors.impl.InvocationContextInterceptor.ignoreCommand(InvocationContextInterceptor.java:100)
at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:86)
at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:244)
at org.infinispan.cache.impl.InvocationHelper.doInvoke(InvocationHelper.java:297)
at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:101)
at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:83)
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1292)
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1820)
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:225)
at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:449)
at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:715)
at org.infinispan.util.ThreadLocalLeakTest.lambda$doStuffWithCache$1(ThreadLocalLeakTest.java:104)
at org.infinispan.test.AbstractInfinispanTest$RunnableWrapper.run(AbstractInfinispanTest.java:507)
{noformat}
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
4 years, 8 months
[Red Hat JIRA] (ISPN-12630) SIFS does not set the seqId when loading index from disk
by Pedro Ruivo (Jira)
[ https://issues.redhat.com/browse/ISPN-12630?page=com.atlassian.jira.plugi... ]
Work on ISPN-12630 started by Pedro Ruivo.
------------------------------------------
> SIFS does not set the seqId when loading index from disk
> --------------------------------------------------------
>
> Key: ISPN-12630
> URL: https://issues.redhat.com/browse/ISPN-12630
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 12.0.0.CR1
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
> Priority: Major
>
> {{seqId}} is used to find which key's value is the most recent in the file. However, if the index is loaded from the disk, it is never set and the new entries are stored from {{seqId=1}}.
> Reads/Writes aren't affected but iteration relies on this to discard old values. It has the side effect of (possibly) returning multiple values for the same key.
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
4 years, 8 months
[Red Hat JIRA] (ISPN-12630) SIFS does not set the seqId when loading index from disk
by Pedro Ruivo (Jira)
[ https://issues.redhat.com/browse/ISPN-12630?page=com.atlassian.jira.plugi... ]
Pedro Ruivo updated ISPN-12630:
-------------------------------
Status: Open (was: New)
> SIFS does not set the seqId when loading index from disk
> --------------------------------------------------------
>
> Key: ISPN-12630
> URL: https://issues.redhat.com/browse/ISPN-12630
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 12.0.0.CR1
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
> Priority: Major
>
> {{seqId}} is used to find which key's value is the most recent in the file. However, if the index is loaded from the disk, it is never set and the new entries are stored from {{seqId=1}}.
> Reads/Writes aren't affected but iteration relies on this to discard old values. It has the side effect of (possibly) returning multiple values for the same key.
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
4 years, 8 months
[Red Hat JIRA] (ISPN-12335) Expose rebalancing progress
by Tristan Tarrant (Jira)
[ https://issues.redhat.com/browse/ISPN-12335?page=com.atlassian.jira.plugi... ]
Tristan Tarrant updated ISPN-12335:
-----------------------------------
Fix Version/s: 12.1.0.Final
(was: 12.0.0.Final)
> Expose rebalancing progress
> ---------------------------
>
> Key: ISPN-12335
> URL: https://issues.redhat.com/browse/ISPN-12335
> Project: Infinispan
> Issue Type: Feature Request
> Components: Core
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Fix For: 12.1.0.Final
>
>
> Rebalancing sometimes takes a long time without logging anything (except at TRACE level). We need to expose the progress somehow: either the number of transferred segments, the number of transferred entries, or maybe even the size of the transferred entries when the size is available.
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
4 years, 8 months
[Red Hat JIRA] (ISPN-12350) Persistent UUIDs are only used for initial consistent hash
by Tristan Tarrant (Jira)
[ https://issues.redhat.com/browse/ISPN-12350?page=com.atlassian.jira.plugi... ]
Tristan Tarrant updated ISPN-12350:
-----------------------------------
Fix Version/s: 12.1.0.Final
(was: 12.0.0.Final)
> Persistent UUIDs are only used for initial consistent hash
> ----------------------------------------------------------
>
> Key: ISPN-12350
> URL: https://issues.redhat.com/browse/ISPN-12350
> Project: Infinispan
> Issue Type: Bug
> Components: Core, State Transfer
> Affects Versions: 12.0.0.Dev03, 11.0.3.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Fix For: 12.1.0.Final
>
>
> After a graceful restart, the persisted UUIDs are used to re-create the consistent hash of the cache before shutdown. This initial CH will not be rebalanced, so there is no state transfer immediately after cluster restart.
> However, if something then triggers a rebalance (e.g. a node join/leave), the persistent UUIDs are ignored, and {{SyncConsistentHashFactory}} allocates segments based on the new JGroups addresses instead of the persistent UUIDs.
> I modified {{ThreeNodeDistGlobalStateRestartTest}} to force a rebalance after restart, and I got
> {noformat}
> 11:24:07,424 TRACE (jgroups-7,Test-NodeD:[]) [ClusterCacheStatus] Cache testCache topology updated: CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=DefaultConsistentHash{ns=256, owners = (3)[Test-NodeD: 83+0, Test-NodeE: 87+0, Test-NodeF: 86+0]}, pendingCH=null, unionCH=null, actualMembers=[Test-NodeD, Test-NodeE, Test-NodeF], persistentUUIDs=[1ba71c04-a6b9-4a5c-9f51-e5e358081dc6, 6d3ff549-aafa-4d8a-8617-84ac6f119549, f37f6a8c-32a4-4dda-b1b0-876c24f42c6a]}, members = [Test-NodeD, Test-NodeE, Test-NodeF], joiners = []
> 11:24:07,889 TRACE (testng-Test:[]) [ClusterCacheStatus] Rebalancing consistent hash for cache testCache, members are [Test-NodeD, Test-NodeE, Test-NodeF]
> 11:24:07,909 TRACE (testng-Test:[]) [ClusterCacheStatus] Updating cache testCache topology for rebalance: CacheTopology{id=2, phase=READ_OLD_WRITE_ALL, rebalanceId=2, currentCH=DefaultConsistentHash{ns=256, owners = (3)[Test-NodeD: 83+0, Test-NodeE: 87+0, Test-NodeF: 86+0]}, pendingCH=DefaultConsistentHash{ns=256, owners = (3)[Test-NodeD: 87+0, Test-NodeE: 83+0, Test-NodeF: 86+0]}, unionCH=null, actualMembers=[Test-NodeD, Test-NodeE, Test-NodeF], persistentUUIDs=[1ba71c04-a6b9-4a5c-9f51-e5e358081dc6, 6d3ff549-aafa-4d8a-8617-84ac6f119549, f37f6a8c-32a4-4dda-b1b0-876c24f42c6a]}
> 11:24:07,910 TRACE (testng-Test:[]) [ClusterCacheStatus] Moved segments: [Test-NodeD added 72 removed 68, Test-NodeE added 49 removed 53, Test-NodeF added 59 removed 59]
> {noformat}
> This issue does not affect caches using {{DefaultConsistentHashFactory}}, because it doesn't care about member UUIDs. Since there is no {{SyncScatteredConsistentHashFactory}}, scattered cache are not affected at all. Replicated caches with the default {{SyncReplicateedConsistentHashFactory}} will change primary owners, but they won't need any state transfer.
> {{TestingUtil.waitForNoRebalance()}} works around the issue by not checking whether the initial consistent hash (with topologyId==1) is balanced.
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
4 years, 8 months