[jboss-jira] [JBoss JIRA] (WFLY-10273) ArrayIndexOutOfBoundsException on server using scattered cache

[Michal Vinkler (JIRA)]

     [ https://issues.jboss.org/browse/WFLY-10273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michal Vinkler updated WFLY-10273:
----------------------------------
    Description: 
We hit ArrayIndexOutOfBoundsException when running tests for RFE EAP7-867.

EAP distribution was built from {{https://github.com/pferraro/wildfly/tree/scattered}} .
Test description: Positive stress test (no failover), 4-node EAP cluster, clients: starting with 400 clients in the beginning, raising the number of clients to 6000 in the end of the test.

Error occured on server dev215 around 7th iteration (can be seen in the performance report, link below):
{code}
[JBossINF] [0m[31m04:26:11,708 ERROR [stderr] (transport-thread--p15-t25) Exception in thread "transport-thread--p15-t25" java.lang.ArrayIndexOutOfBoundsException: 129
[JBossINF] [0m[31m04:26:11,708 ERROR [stderr] (transport-thread--p15-t25) 	at org.infinispan.scattered.impl.ScatteredVersionManagerImpl.lambda$tryRegularInvalidations$4(ScatteredVersionManagerImpl.java:413)
[JBossINF] [0m[31m04:26:11,708 ERROR [stderr] (transport-thread--p15-t25) 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] [0m[31m04:26:11,708 ERROR [stderr] (transport-thread--p15-t25) 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] [0m[31m04:26:11,708 ERROR [stderr] (transport-thread--p15-t25) 	at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
[JBossINF] [0m[31m04:26:11,708 ERROR [stderr] (transport-thread--p15-t25) 	at java.lang.Thread.run(Thread.java:748)
{code} 

Clients were getting "SocketTimeoutException: Read timed out" exceptions even before the ArrayIndexOutOfBoundsException ocurred, but also after. 

Performance report (accessible only when connected to VPN): 
http://download.eng.brq.redhat.com/scratch/mvinkler/reports/2018-04-19_15-10-14/stress.html

One can observe that dev215 CPU usage and network usage dropped after 7th iteration. 

dev215 server log link:
https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-stress-session-scattered-sync/13/console-dev215/

  was:
We hit ArrayIndexOutOfBoundsException when running tests for RFE EAP7-867.

EAP distribution was built from {{https://github.com/pferraro/wildfly/tree/scattered}}.
Test description: Positive stress test (no failover), 4-node EAP cluster, clients: starting with 400 clients in the beginning, raising the number of clients to 6000 in the end of the test.

Error occured on server dev215 around 7th iteration (can be seen in the performance report, link below):
{code}
[JBossINF] [0m[31m04:26:11,708 ERROR [stderr] (transport-thread--p15-t25) Exception in thread "transport-thread--p15-t25" java.lang.ArrayIndexOutOfBoundsException: 129
[JBossINF] [0m[31m04:26:11,708 ERROR [stderr] (transport-thread--p15-t25) 	at org.infinispan.scattered.impl.ScatteredVersionManagerImpl.lambda$tryRegularInvalidations$4(ScatteredVersionManagerImpl.java:413)
[JBossINF] [0m[31m04:26:11,708 ERROR [stderr] (transport-thread--p15-t25) 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] [0m[31m04:26:11,708 ERROR [stderr] (transport-thread--p15-t25) 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] [0m[31m04:26:11,708 ERROR [stderr] (transport-thread--p15-t25) 	at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
[JBossINF] [0m[31m04:26:11,708 ERROR [stderr] (transport-thread--p15-t25) 	at java.lang.Thread.run(Thread.java:748)
{code} 

Clients were getting "SocketTimeoutException: Read timed out" exceptions even before the ArrayIndexOutOfBoundsException ocurred, but also after. 

Performance report (accessible only when connected to VPN): 
http://download.eng.brq.redhat.com/scratch/mvinkler/reports/2018-04-19_15-10-14/stress.html

One can observe that dev215 CPU usage and network usage dropped after 7th iteration. 

dev215 server log link:
https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-stress-session-scattered-sync/13/console-dev215/



> ArrayIndexOutOfBoundsException on server using scattered cache 
> ---------------------------------------------------------------
>
>                 Key: WFLY-10273
>                 URL: https://issues.jboss.org/browse/WFLY-10273
>             Project: WildFly
>          Issue Type: Bug
>          Components: Clustering
>    Affects Versions: 13.0.0.Beta1
>            Reporter: Michal Vinkler
>            Assignee: Paul Ferraro
>
> We hit ArrayIndexOutOfBoundsException when running tests for RFE EAP7-867.
> EAP distribution was built from {{https://github.com/pferraro/wildfly/tree/scattered}} .
> Test description: Positive stress test (no failover), 4-node EAP cluster, clients: starting with 400 clients in the beginning, raising the number of clients to 6000 in the end of the test.
> Error occured on server dev215 around 7th iteration (can be seen in the performance report, link below):
> {code}
> [JBossINF] [0m[31m04:26:11,708 ERROR [stderr] (transport-thread--p15-t25) Exception in thread "transport-thread--p15-t25" java.lang.ArrayIndexOutOfBoundsException: 129
> [JBossINF] [0m[31m04:26:11,708 ERROR [stderr] (transport-thread--p15-t25) 	at org.infinispan.scattered.impl.ScatteredVersionManagerImpl.lambda$tryRegularInvalidations$4(ScatteredVersionManagerImpl.java:413)
> [JBossINF] [0m[31m04:26:11,708 ERROR [stderr] (transport-thread--p15-t25) 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] [0m[31m04:26:11,708 ERROR [stderr] (transport-thread--p15-t25) 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] [0m[31m04:26:11,708 ERROR [stderr] (transport-thread--p15-t25) 	at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> [JBossINF] [0m[31m04:26:11,708 ERROR [stderr] (transport-thread--p15-t25) 	at java.lang.Thread.run(Thread.java:748)
> {code} 
> Clients were getting "SocketTimeoutException: Read timed out" exceptions even before the ArrayIndexOutOfBoundsException ocurred, but also after. 
> Performance report (accessible only when connected to VPN): 
> http://download.eng.brq.redhat.com/scratch/mvinkler/reports/2018-04-19_15-10-14/stress.html
> One can observe that dev215 CPU usage and network usage dropped after 7th iteration. 
> dev215 server log link:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-stress-session-scattered-sync/13/console-dev215/



--
This message was sent by Atlassian JIRA
(v7.5.0#75005)