[infinispan-issues] [JBoss JIRA] (ISPN-11033) Cluster fails while inserting data for a while

[Jens Reimann (Jira)]

Jens Reimann created ISPN-11033:
-----------------------------------

             Summary: Cluster fails while inserting data for a while
                 Key: ISPN-11033
                 URL: https://issues.jboss.org/browse/ISPN-11033
             Project: Infinispan
          Issue Type: Bug
          Components: Server
    Affects Versions: 10.0.1.Final
         Environment: 12 node Infinispan cluster, OpenShift 4.2
            Reporter: Jens Reimann
         Attachments: deviceManagement.proto, infinispan.xml

Inserting data into an Infinispan cluster works for a while, and then the cluster fails. Showing the following log messages in one pod:


{code}
14:20:34,432 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p4-t1) ISPN000136: Error executing command ReplaceCommand on Cache 'devices', writing keys [WrappedByteArray{bytes=8201\*\i\o\.\e\n\m\a\s\s\e\.\i\o\t\.\i\n\f\i\n\i\s\p\a\n\.\d\e\v\i\c\e\.\D\e\v\i\c\e\K\e\y8A01\<0A1F\j\b\t\e\s\t\.\i\o\t\/\2\0\1\9\-\1\2\-\0\4\T\0\8\:\2\5\:\3\4\Z1219\h\t\t\p\-\i\n\s\e\r\t\e\r\-\f\r\8\l\m\1\5\2\2\4\7, hashCode=-381217399}]: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key WrappedByteArray{bytes=8201\*\i\o\.\e\n\m\a\s\s\e\.\i\o\t\.\i\n\f\i\n\i\s\p\a\n\.\d\e\v\i\c\e\.\D\e\v\i\c\e\K\e\y8A01\<0A1F\j\b\t\e\s\t\.\i\o\t\/\2\0\1\9\-\1\2\-\0\4\T\0\8\:\2\5\:\3\4\Z1219\h\t\t\p\-\i\n\s\e\r\t\e\r\-\f\r\8\l\m\1\5\2\2\4\7, hashCode=-381217399} and requestor GlobalTx:infinispan-8-8720:1383960. Lock is held by GlobalTx:infinispan-8-8720:33804
	at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.get(DefaultLockManager.java:292)
	at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.get(DefaultLockManager.java:222)
	at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.checkState(InfinispanLock.java:440)
	at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.lambda$toInvocationStage$3(InfinispanLock.java:416)
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642)
	at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
	at org.infinispan.commons.util.concurrent.CallerRunsRejectOnShutdownPolicy.rejectedExecution(CallerRunsRejectOnShutdownPolicy.java:19)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
	at org.infinispan.executors.LazyInitializingExecutorService.execute(LazyInitializingExecutorService.java:138)
	at java.base/java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:568)
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:638)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
	at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.notifyListeners(InfinispanLock.java:527)
	at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.cancel(InfinispanLock.java:382)
	at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.call(DefaultLockManager.java:286)
	at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.call(DefaultLockManager.java:222)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

{code}

While showing the following message in the other nodes log:

{code}
14:44:26,310 ERROR [org.jgroups.protocols.TCP] (jgroups-133,infinispan-3-50867) JGRP000034: infinispan-3-50867: failure sending message to infinispan-8-17029: java.net.SocketTimeoutException: connect timed out
14:44:28,611 ERROR [org.jgroups.protocols.TCP] (jgroups-133,infinispan-3-50867) JGRP000034: infinispan-3-50867: failure sending message to infinispan-8-17029: java.net.SocketTimeoutException: connect timed out
14:44:30,912 ERROR [org.jgroups.protocols.TCP] (jgroups-126,infinispan-3-50867) JGRP000034: infinispan-3-50867: failure sending message to infinispan-8-17029: java.net.SocketTimeoutException: connect timed out
{code}

The node showing the exception gets killed after a while by Kubernetes:

{code}
NAME            READY   STATUS                 RESTARTS   AGE
infinispan-0    1/1     Running                0          83m
infinispan-1    1/1     Running                0          83m
infinispan-10   1/1     Running                0          83m
infinispan-11   1/1     Running                0          83m
infinispan-2    1/1     Running                0          83m
infinispan-3    1/1     Running                0          83m
infinispan-4    1/1     Running                0          83m
infinispan-5    1/1     Running                0          83m
infinispan-6    1/1     Running                0          83m
infinispan-7    1/1     Running                0          83m
infinispan-8    0/1     CreateContainerError   3          83m
infinispan-9    1/1     Running                0          83m
{code}

But it never becomes ready again.



--
This message was sent by Atlassian Jira
(v7.13.8#713008)