[Red Hat JIRA] (ISPN-12652) Cluster is broken after one node is down
by Dmitry Kruglikov (Jira)
Dmitry Kruglikov created ISPN-12652:
---------------------------------------
Summary: Cluster is broken after one node is down
Key: ISPN-12652
URL: https://issues.redhat.com/browse/ISPN-12652
Project: Infinispan
Issue Type: Bug
Affects Versions: 9.4.14.Final
Reporter: Dmitry Kruglikov
We have 3 nodes in cluster: app1, app2 and app3. App1 was shut down not gracefully because of some hardware issue. After that app2 and app3 started to fail with something like
ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p23-t1) ISPN000136: Error executing command RemoveCommand on Cache 'fs.war', writing keys [SessionCreationMetaDataKey(PGARVVdjGKfifzrVfyd7HAllbrwaRG7wLhKha1On)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 422657 from app1
at org.infinispan@9.4.14.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
at org.infinispan@9.4.14.Final//org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
at org.infinispan@9.4.14.Final//org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
So these 2 nodes (app2 and app3) could not serve user requests anymore until app1 recovered. My question is... Is it ok? Should not Infinispan identify that one of nodes is down, remove it from cluster and notify app2 and app3 about it? I know that there is something like VERIFY_SUSPECT but it didn't happen.
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
5 years, 2 months
[Red Hat JIRA] (ISPN-12646) IllegalArgumentException when doing Rolling Upgrades on transactional caches
by Gustavo Fernandes (Jira)
[ https://issues.redhat.com/browse/ISPN-12646?page=com.atlassian.jira.plugi... ]
Gustavo Fernandes updated ISPN-12646:
-------------------------------------
Git Pull Request: https://github.com/infinispan/infinispan/pull/9007
Status: Pull Request Sent (was: Coding In Progress)
> IllegalArgumentException when doing Rolling Upgrades on transactional caches
> ----------------------------------------------------------------------------
>
> Key: ISPN-12646
> URL: https://issues.redhat.com/browse/ISPN-12646
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 11.0.9.Final
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
> Priority: Major
>
> {noformat}
> Caused by: java.lang.IllegalArgumentException: Cannot create a transactional context without a valid Transaction instance.
> at org.infinispan.context.impl.TransactionalInvocationContextFactory.createInvocationContext(TransactionalInvocationContextFactory.java:63)
> [...]
> at org.infinispan.cache.impl.DecoratedCache.putIfAbsent(DecoratedCache.java:688)
> at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.putIfAbsent(AbstractDelegatingAdvancedCache.java:328)
> at org.infinispan.cache.impl.EncoderCache.putIfAbsent(EncoderCache.java:450)
> at org.infinispan.persistence.remote.upgrade.MigrationTask.lambda$migrateEntriesWithMetadata$0(MigrationTask.java:128)
> at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1771)
> {noformat}
> The migration component that obtains data from the remote cache does not use transactions to write to the destination cache, and since it runs on a separate thread, it cannot see any ongoing transaction and thus fail to write to the cache.
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
5 years, 2 months
[Red Hat JIRA] (ISPN-12646) IllegalArgumentException when doing Rolling Upgrades on transactional caches
by Gustavo Fernandes (Jira)
[ https://issues.redhat.com/browse/ISPN-12646?page=com.atlassian.jira.plugi... ]
Work on ISPN-12646 started by Gustavo Fernandes.
------------------------------------------------
> IllegalArgumentException when doing Rolling Upgrades on transactional caches
> ----------------------------------------------------------------------------
>
> Key: ISPN-12646
> URL: https://issues.redhat.com/browse/ISPN-12646
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 11.0.9.Final
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
> Priority: Major
>
> {noformat}
> Caused by: java.lang.IllegalArgumentException: Cannot create a transactional context without a valid Transaction instance.
> at org.infinispan.context.impl.TransactionalInvocationContextFactory.createInvocationContext(TransactionalInvocationContextFactory.java:63)
> [...]
> at org.infinispan.cache.impl.DecoratedCache.putIfAbsent(DecoratedCache.java:688)
> at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.putIfAbsent(AbstractDelegatingAdvancedCache.java:328)
> at org.infinispan.cache.impl.EncoderCache.putIfAbsent(EncoderCache.java:450)
> at org.infinispan.persistence.remote.upgrade.MigrationTask.lambda$migrateEntriesWithMetadata$0(MigrationTask.java:128)
> at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1771)
> {noformat}
> The migration component that obtains data from the remote cache does not use transactions to write to the destination cache, and since it runs on a separate thread, it cannot see any ongoing transaction and thus fail to write to the cache.
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
5 years, 2 months
[Red Hat JIRA] (ISPN-12651) CacheEntryCloudEventsTest test failures
by Dan Berindei (Jira)
Dan Berindei created ISPN-12651:
-----------------------------------
Summary: CacheEntryCloudEventsTest test failures
Key: ISPN-12651
URL: https://issues.redhat.com/browse/ISPN-12651
Project: Infinispan
Issue Type: Bug
Components: Integration , Test Suite
Affects Versions: 12.0.0.Final
Reporter: Dan Berindei
Assignee: Dan Berindei
Fix For: 12.1.0.Final
{{jackson-core}} is an optional dependency in {{protostream}}, but it is required for transcoding {{application/x-protostream}} to JSON. Because the dependency is missing, the tests fail with {{OBJECT}} storage:
{noformat}
org.infinispan.util.concurrent.TimeoutException
at org.infinispan.cloudevents.MockKafkaEventSender.completeSend(MockKafkaEventSender.java:44)
at org.infinispan.cloudevents.MockKafkaEventSender.completeSend(MockKafkaEventSender.java:53)
at org.infinispan.cloudevents.CacheEntryCloudEventsTest.testMultiKeyOperations(CacheEntryCloudEventsTest.java:188)
{noformat}
The actual reason is hidden because exceptions in listeners are not handled correctly:
{noformat}
org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.lang.NoClassDefFoundError] while invoking method [public java.util.concurrent.CompletionStage org.infinispan.cloudevents.impl.EntryEventListener.handleCacheEntryEvent(org.infinispan.notifications.cachelistener.event.CacheEntryEvent)] on listener instance: org.infinispan.cloudevents.impl.EntryEventListener@7a524380
at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.lambda$invoke$1(AbstractListenerImpl.java:430)
at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:450)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1817)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1813)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invokeNoChecks(CacheNotifierImpl.java:1808)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1781)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl.doNotifyCreated(CacheNotifierImpl.java:428)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryCreated(CacheNotifierImpl.java:410)
at org.infinispan.notifications.cachelistener.NotifyHelper.entryCommitted(NotifyHelper.java:59)
at org.infinispan.interceptors.locking.ClusteringDependentLogic$DistributionLogic.commitSingleEntry(ClusteringDependentLogic.java:539)
at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.commitEntry(ClusteringDependentLogic.java:198)
at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:651)
at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:904)
at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:631)
at org.infinispan.interceptors.impl.EntryWrappingInterceptor.applyChanges(EntryWrappingInterceptor.java:687)
at org.infinispan.interceptors.impl.EntryWrappingInterceptor.applyAndFixVersion(EntryWrappingInterceptor.java:758)
at org.infinispan.interceptors.InvocationSuccessFunction.apply(InvocationSuccessFunction.java:25)
at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.invokeQueuedHandlers(QueueAsyncInvocationStage.java:124)
at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:87)
at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:33)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2137)
at org.infinispan.util.concurrent.CommandAckCollector$SingleKeyCollector.markReady(CommandAckCollector.java:364)
at org.infinispan.util.concurrent.CommandAckCollector$SingleKeyCollector.backupAck(CommandAckCollector.java:356)
at org.infinispan.util.concurrent.CommandAckCollector.backupAck(CommandAckCollector.java:167)
at org.infinispan.commands.write.BackupAckCommand.ack(BackupAckCommand.java:40)
at org.infinispan.remoting.inboundhandler.TrianglePerCacheInboundInvocationHandler.handleBackupAckCommand(TrianglePerCacheInboundInvocationHandler.java:181)
at org.infinispan.remoting.inboundhandler.TrianglePerCacheInboundInvocationHandler.handle(TrianglePerCacheInboundInvocationHandler.java:96)
at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:164)
at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:111)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1383)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1307)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:131)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1450)
at org.jgroups.JChannel.up(JChannel.java:784)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:913)
at org.jgroups.protocols.FRAG3.up(FRAG3.java:165)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:351)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:359)
at org.jgroups.protocols.tom.TOA.up(TOA.java:119)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:876)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:243)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1049)
at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:772)
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:753)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:405)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:592)
at org.jgroups.protocols.Discovery.up(Discovery.java:300)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1396)
at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/JsonProcessingException
at org.infinispan.protostream.ProtobufUtil.toCanonicalJSON(ProtobufUtil.java:174)
at org.infinispan.protostream.ProtobufUtil.toCanonicalJSON(ProtobufUtil.java:160)
at org.infinispan.encoding.ProtostreamTranscoder.toJsonCascading(ProtostreamTranscoder.java:183)
at org.infinispan.encoding.ProtostreamTranscoder.transcode(ProtostreamTranscoder.java:102)
at org.infinispan.cloudevents.impl.EntryEventListener.writeKey(EntryEventListener.java:165)
at org.infinispan.cloudevents.impl.EntryEventListener.entryEventToKafkaMessage(EntryEventListener.java:133)
at org.infinispan.cloudevents.impl.EntryEventListener.handleCacheEntryEvent(EntryEventListener.java:101)
at jdk.internal.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.lambda$invoke$1(AbstractListenerImpl.java:424)
... 54 more
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.core.JsonProcessingException
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:606)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:168)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
... 65 more
{noformat}
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
5 years, 2 months
[Red Hat JIRA] (ISPN-12630) SIFS does not set the seqId when loading index from disk
by Pedro Ruivo (Jira)
[ https://issues.redhat.com/browse/ISPN-12630?page=com.atlassian.jira.plugi... ]
Pedro Ruivo updated ISPN-12630:
-------------------------------
Fix Version/s: 12.1.0.Final
> SIFS does not set the seqId when loading index from disk
> --------------------------------------------------------
>
> Key: ISPN-12630
> URL: https://issues.redhat.com/browse/ISPN-12630
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 12.0.0.CR1
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
> Priority: Major
> Fix For: 12.1.0.Final
>
>
> {{seqId}} is used to find which key's value is the most recent in the file. However, if the index is loaded from the disk, it is never set and the new entries are stored from {{seqId=1}}.
> Reads/Writes aren't affected but iteration relies on this to discard old values. It has the side effect of (possibly) returning multiple values for the same key.
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
5 years, 2 months
[Red Hat JIRA] (ISPN-12650) ThreadLocalLeakTest random failures (uncaught exception)
by Dan Berindei (Jira)
Dan Berindei created ISPN-12650:
-----------------------------------
Summary: ThreadLocalLeakTest random failures (uncaught exception)
Key: ISPN-12650
URL: https://issues.redhat.com/browse/ISPN-12650
Project: Infinispan
Issue Type: Bug
Components: Test Suite
Affects Versions: 12.0.0.Final
Reporter: Dan Berindei
Assignee: Dan Berindei
Fix For: 12.1.0.Final
{{ThreadLocalLeakTest}} is supposed to invoke a cache operation on a new thread and keep that new thread running, then check that the operation didn't leave any thread locals on that thread. But sometimes the cache manager is stopped before the new thread managed to invoke the cache operation, causing a {{IllegalLifecycleStateException}} that is never caught:
{noformat}
org.infinispan.commons.IllegalLifecycleStateException: Cache container has been stopped and cannot be reused. Recreate the cache container.
at org.infinispan.manager.DefaultCacheManager.assertIsNotTerminated(DefaultCacheManager.java:1074)
at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:517)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:510)
at org.infinispan.util.ThreadLocalLeakTest.lambda$doStuffWithCache$1(ThreadLocalLeakTest.java:103)
at org.infinispan.test.AbstractInfinispanTest$RunnableWrapper.run(AbstractInfinispanTest.java:507)
{noformat}
{noformat}
org.infinispan.commons.IllegalLifecycleStateException: ISPN000324: Cache 'leak' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
at org.infinispan.interceptors.impl.InvocationContextInterceptor.ignoreCommand(InvocationContextInterceptor.java:100)
at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:86)
at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:244)
at org.infinispan.cache.impl.InvocationHelper.doInvoke(InvocationHelper.java:297)
at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:101)
at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:83)
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1292)
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1820)
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:225)
at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:449)
at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:715)
at org.infinispan.util.ThreadLocalLeakTest.lambda$doStuffWithCache$1(ThreadLocalLeakTest.java:104)
at org.infinispan.test.AbstractInfinispanTest$RunnableWrapper.run(AbstractInfinispanTest.java:507)
{noformat}
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
5 years, 2 months
[Red Hat JIRA] (ISPN-12630) SIFS does not set the seqId when loading index from disk
by Pedro Ruivo (Jira)
[ https://issues.redhat.com/browse/ISPN-12630?page=com.atlassian.jira.plugi... ]
Work on ISPN-12630 started by Pedro Ruivo.
------------------------------------------
> SIFS does not set the seqId when loading index from disk
> --------------------------------------------------------
>
> Key: ISPN-12630
> URL: https://issues.redhat.com/browse/ISPN-12630
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 12.0.0.CR1
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
> Priority: Major
>
> {{seqId}} is used to find which key's value is the most recent in the file. However, if the index is loaded from the disk, it is never set and the new entries are stored from {{seqId=1}}.
> Reads/Writes aren't affected but iteration relies on this to discard old values. It has the side effect of (possibly) returning multiple values for the same key.
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
5 years, 2 months
[Red Hat JIRA] (ISPN-12630) SIFS does not set the seqId when loading index from disk
by Pedro Ruivo (Jira)
[ https://issues.redhat.com/browse/ISPN-12630?page=com.atlassian.jira.plugi... ]
Pedro Ruivo updated ISPN-12630:
-------------------------------
Status: Open (was: New)
> SIFS does not set the seqId when loading index from disk
> --------------------------------------------------------
>
> Key: ISPN-12630
> URL: https://issues.redhat.com/browse/ISPN-12630
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 12.0.0.CR1
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
> Priority: Major
>
> {{seqId}} is used to find which key's value is the most recent in the file. However, if the index is loaded from the disk, it is never set and the new entries are stored from {{seqId=1}}.
> Reads/Writes aren't affected but iteration relies on this to discard old values. It has the side effect of (possibly) returning multiple values for the same key.
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
5 years, 2 months