[infinispan-issues] [JBoss JIRA] (ISPN-11017) Cluster fails and doesn't recover under load

Cluster fails and doesn't recover under load -------------------------------------------- Key: ISPN-11017 URL: https://issues.jboss.org/browse/ISPN-11017 Project: Infinispan Issue Type: Bug Components: Server Affects Versions: 10.0.1.Final Environment: Running in OpenShift, with a stateful set of 12 nodes, a distributed cache with 3 owners, async indexing enabled, persistence with rocksdb. Reporter: Jens Reimann Priority: Blocker Attachments: infinispan.xml After running the load test for a few seconds the inifinispan cluster stops accepting requests and the nodes start to split off from the cluster. In the server's log you can find tons of exceptions like: {code:java} 10:42:26,939 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p4-t1) ISPN000136: Error executing command PutKeyValueCommand on Cache '___protobuf_metadata', writing keys [deviceRegistry.proto]: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 10 seconds for key deviceRegistry.proto and requestor GlobalTx:infinispan-2-61958:249. Lock is held by GlobalTx:infinispan-2-61958:248 {code} Stopping the load test doesn't let the cluster recover. Most (not all) of the liveness checks fail and pods get restarted. But even after 1 hour, the cluster is still in a non-working state.