[infinispan-issues] [JBoss JIRA] (ISPN-11017) Cluster fails and doesn't recover under load

[Jens Reimann (Jira)]

Jens Reimann created ISPN-11017:
-----------------------------------

             Summary: Cluster fails and doesn't recover under load
                 Key: ISPN-11017
                 URL: https://issues.jboss.org/browse/ISPN-11017
             Project: Infinispan
          Issue Type: Bug
          Components: Server
    Affects Versions: 10.0.1.Final
         Environment: Running in OpenShift, with a stateful set of 12 nodes, a distributed cache with 3 owners, async indexing enabled, persistence with rocksdb.
            Reporter: Jens Reimann


After running the load test for a few seconds the inifinispan cluster stops accepting requests and the nodes start to split off from the cluster. In the server's log you can find tons of exceptions like:

{code:java}
10:42:26,939 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p4-t1) ISPN000136: Error executing command PutKeyValueCommand on Cache '___protobuf_metadata', writing keys [deviceRegistry.proto]: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 10 seconds for key deviceRegistry.proto and requestor GlobalTx:infinispan-2-61958:249. Lock is held by GlobalTx:infinispan-2-61958:248
{code}

Stopping the load test doesn't let the cluster recover. Most (not all) of the liveness checks fail and pods get restarted. But even after 1 hour, the cluster is still in a non-working state.




--
This message was sent by Atlassian Jira
(v7.13.8#713008)