[infinispan-issues] [JBoss JIRA] (ISPN-11017) Cluster fails and doesn't recover under load

[Pedro Zapata Fernandez (Jira)]

     [ https://issues.redhat.com/browse/ISPN-11017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pedro Zapata Fernandez reassigned ISPN-11017:
---------------------------------------------

    Assignee: Ryan Emerson


> Cluster fails and doesn't recover under load
> --------------------------------------------
>
>                 Key: ISPN-11017
>                 URL: https://issues.redhat.com/browse/ISPN-11017
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Server
>    Affects Versions: 10.0.1.Final
>         Environment: Running in OpenShift, with a stateful set of 12 nodes, a distributed cache with 3 owners, async indexing enabled, persistence with rocksdb.
>            Reporter: Jens Reimann
>            Assignee: Ryan Emerson
>            Priority: Blocker
>         Attachments: infinispan.xml
>
>
> After running the load test for a few seconds the inifinispan cluster stops accepting requests and the nodes start to split off from the cluster. In the server's log you can find tons of exceptions like:
> {code:java}
> 10:42:26,939 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p4-t1) ISPN000136: Error executing command PutKeyValueCommand on Cache '___protobuf_metadata', writing keys [deviceRegistry.proto]: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 10 seconds for key deviceRegistry.proto and requestor GlobalTx:infinispan-2-61958:249. Lock is held by GlobalTx:infinispan-2-61958:248
> {code}
> Stopping the load test doesn't let the cluster recover. Most (not all) of the liveness checks fail and pods get restarted. But even after 1 hour, the cluster is still in a non-working state.



--
This message was sent by Atlassian Jira
(v7.13.8#713008)