[JBoss JIRA] (ISPN-11035) XSiteResourceTest.testPushAllCaches random failures
by Pedro Zapata Fernandez (Jira)
[ https://issues.redhat.com/browse/ISPN-11035?page=com.atlassian.jira.plugi... ]
Pedro Zapata Fernandez updated ISPN-11035:
------------------------------------------
Sprint: DataGrid Sprint #37, DataGrid Sprint #38, DataGrid Sprint #39 (was: DataGrid Sprint #37, DataGrid Sprint #38)
> XSiteResourceTest.testPushAllCaches random failures
> ---------------------------------------------------
>
> Key: ISPN-11035
> URL: https://issues.redhat.com/browse/ISPN-11035
> Project: Infinispan
> Issue Type: Bug
> Components: Cross-Site Replication, REST, Test Suite
> Affects Versions: 10.1.0.Beta1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.1.0.CR1
>
>
> The test doesn't seem to wait enough for the xsite state transfer to finish. When it fails, there is only one request to `/rest/v2/cache-managers/default/x-site/backups`, before the remote site receives the state:
> {noformat}
> 19:32:35,797 TRACE (REST-Test-Test-NodeA-48205-ServerIO-15-9:[]) [REST_ACCESS_LOG] /rest/v2/cache-managers/default/x-site/backups
> 19:32:35,807 TRACE (jgroups-4,bridge-org.infinispan.rest.resources.Test,_Test-NodeC-47561:SFO-3:[]) [JGroupsTransport] Test-NodeC-47561 received request 628 from Test-NodeA-48205:LON-1: XSiteStatePushCommand{cacheName=CACHE_2, timeout=1200000 (10 keys)}
> 19:32:35,809 TRACE (REST-Test-Test-NodeC-47561-ServerIO-21-1:[]) [InvocationContextInterceptor] Invoked with command SizeCommand{} and InvocationContext [org.infinispan.context.impl.NonTxInvocationContext@30b93450]
> 19:32:35,813 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.rest.resources.XSiteResourceTest.testPushAllCaches
> java.lang.AssertionError: expected:<10> but was:<9>
> 19:32:35,813 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.rest.resources.XSiteResourceTest.testPushAllCaches
> java.lang.AssertionError: expected:<10> but was:<9>
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59) ~[testng-6.14.3.jar:?]
> at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364) ~[testng-6.14.3.jar:?]
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80) ~[testng-6.14.3.jar:?]
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:245) ~[testng-6.14.3.jar:?]
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:252) ~[testng-6.14.3.jar:?]
> at org.infinispan.rest.resources.XSiteResourceTest.testPushAllCaches(XSiteResourceTest.java:330) ~[test-classes/:?]
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 11 months
[JBoss JIRA] (ISPN-10972) REST caches list order should be stable
by Pedro Zapata Fernandez (Jira)
[ https://issues.redhat.com/browse/ISPN-10972?page=com.atlassian.jira.plugi... ]
Pedro Zapata Fernandez updated ISPN-10972:
------------------------------------------
Sprint: DataGrid Sprint #36, DataGrid Sprint #37, DataGrid Sprint #38, DataGrid Sprint #39 (was: DataGrid Sprint #36, DataGrid Sprint #37, DataGrid Sprint #38)
> REST caches list order should be stable
> ---------------------------------------
>
> Key: ISPN-10972
> URL: https://issues.redhat.com/browse/ISPN-10972
> Project: Infinispan
> Issue Type: Enhancement
> Components: REST
> Affects Versions: 10.1.0.Beta1
> Reporter: Katia Aresti
> Assignee: Katia Aresti
> Priority: Major
> Labels: console, rest
> Fix For: 10.1.0.CR1, 10.1.0.Final
>
>
> Today the caches resource displays the list of caches and the result is not ordered so it's not stable. Display ordered by name
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 11 months
[JBoss JIRA] (ISPN-11000) LocalTopologyManager should not wait for view if the local node is not a member
by Pedro Zapata Fernandez (Jira)
[ https://issues.redhat.com/browse/ISPN-11000?page=com.atlassian.jira.plugi... ]
Pedro Zapata Fernandez updated ISPN-11000:
------------------------------------------
Sprint: DataGrid Sprint #37, DataGrid Sprint #38, DataGrid Sprint #39 (was: DataGrid Sprint #37, DataGrid Sprint #38)
> LocalTopologyManager should not wait for view if the local node is not a member
> -------------------------------------------------------------------------------
>
> Key: ISPN-11000
> URL: https://issues.redhat.com/browse/ISPN-11000
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.4.16.Final, 10.1.0.Beta1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Fix For: 10.1.0.Final
>
>
> Sometimes a node is excluded from the cluster view but it can still receive multicast messages like FD_ALL heartbeats and topology updates from the coordinator.
> Because it is still receiving heartbeats, the excluded is not becoming coordinator itself and installing a new view. If MERGE3 doesn't merge the partitions, it could keep the outdated view for a long time, and {{LocalTopologyManagerImpl}} will block many transport threads waiting for the right view to process the topology updates that keep coming from the coordinator:
> {noformat}
> 11:31:01,052 INFO [org.radargun.service.InfinispanRestAPI] (pool-2-thread-1) CacheManagerInfo{clusterMembers=[edg-perf01-21541, edg-perf02-54831, edg-perf05-28640, edg-perf03-47882, edg-perf06-47720, edg-perf04-19840, edg-perf07-34498, edg-perf08-52975], clusterSize=8}
> [33m11:31:05,281 WARN [org.jgroups.protocols.pbcast.GMS] (jgroups-79,edg-perf03-47882) edg-perf03-47882: not member of view [edg-perf01-21541|6]; discarding it
> 11:31:11,041 INFO [org.radargun.service.InfinispanRestAPI] (pool-2-thread-1) CacheManagerInfo{clusterMembers=[edg-perf01-21541, edg-perf02-54831, edg-perf05-28640, edg-perf03-47882, edg-perf06-47720, edg-perf04-19840, edg-perf07-34498, edg-perf08-52975], clusterSize=8}
> [0m[33m11:31:16,267 WARN [org.jgroups.protocols.pbcast.GMS] (jgroups-80,edg-perf03-47882) edg-perf03-47882: failed to create view from delta-view; dropping view: java.lang.IllegalStateException: the view-id of the delta view ([edg-perf01-21541|6]) doesn't match the current view-id ([edg-perf01-21541|5]); discarding delta view [edg-perf01-21541|7], ref-view=[edg-perf01-21541|6], left=[edg-perf06-47720]
> [0m[33m11:31:16,274 WARN [org.jgroups.protocols.pbcast.GMS] (jgroups-80,edg-perf03-47882) edg-perf03-47882: not member of view [edg-perf01-21541|7]; discarding it
> 11:31:21,035 INFO [org.radargun.service.InfinispanRestAPI] (pool-2-thread-1) CacheManagerInfo{clusterMembers=[edg-perf01-21541, edg-perf02-54831, edg-perf05-28640, edg-perf03-47882, edg-perf06-47720, edg-perf04-19840, edg-perf07-34498, edg-perf08-52975], clusterSize=8}
> 11:31:31,040 INFO [org.radargun.service.InfinispanRestAPI] (pool-2-thread-1) CacheManagerInfo{clusterMembers=[edg-perf01-21541, edg-perf02-54831, edg-perf05-28640, edg-perf03-47882, edg-perf06-47720, edg-perf04-19840, edg-perf07-34498, edg-perf08-52975], clusterSize=8}
> 11:31:41,047 INFO [org.radargun.service.InfinispanRestAPI] (pool-2-thread-1) CacheManagerInfo{clusterMembers=[edg-perf01-21541, edg-perf02-54831, edg-perf05-28640, edg-perf03-47882, edg-perf06-47720, edg-perf04-19840, edg-perf07-34498, edg-perf08-52975], clusterSize=8}
> 11:31:51,033 INFO [org.radargun.service.InfinispanRestAPI] (pool-2-thread-1) CacheManagerInfo{clusterMembers=[edg-perf01-21541, edg-perf02-54831, edg-perf05-28640, edg-perf03-47882, edg-perf06-47720, edg-perf04-19840, edg-perf07-34498, edg-perf08-52975], clusterSize=8}
> 11:32:01,035 INFO [org.radargun.service.InfinispanRestAPI] (pool-2-thread-1) CacheManagerInfo{clusterMembers=[edg-perf01-21541, edg-perf02-54831, edg-perf05-28640, edg-perf03-47882, edg-perf06-47720, edg-perf04-19840, edg-perf07-34498, edg-perf08-52975], clusterSize=8}
> [0m[33m11:32:03,051 WARN [org.jgroups.protocols.pbcast.GMS] (jgroups-80,edg-perf03-47882) edg-perf03-47882: failed to create view from delta-view; dropping view: java.lang.IllegalStateException: the view-id of the delta view ([edg-perf01-21541|7]) doesn't match the current view-id ([edg-perf01-21541|5]); discarding delta view [edg-perf01-21541|8], ref-view=[edg-perf01-21541|7], left=[edg-perf04-19840]
> [0m[33m11:32:03,063 WARN [org.jgroups.protocols.pbcast.GMS] (jgroups-80,edg-perf03-47882) edg-perf03-47882: not member of view [edg-perf01-21541|8]; discarding it
> [0m[31m11:32:05,321 ERROR [org.infinispan.topology.LocalTopologyManagerImpl] (transport-thread--p5-t5) ISPN000452: Failed to update topology for cache memcachedCache: org.infinispan.util.concurrent.TimeoutException: ISPN000451: Timed out waiting for view 6, current view is 5
> at org.infinispan.topology.LocalTopologyManagerImpl.waitForView(LocalTopologyManagerImpl.java:571)
> at org.infinispan.topology.LocalTopologyManagerImpl.doHandleTopologyUpdate(LocalTopologyManagerImpl.java:302)
> at org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleTopologyUpdate$1(LocalTopologyManagerImpl.java:286)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {noformat}
> {{LocalTopologyManagerImpl.doHandleTopologyUpdate()}} could check if the local node is a member of the new topology first, avoid blocking, and avoid logging an error message.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 11 months
[JBoss JIRA] (ISPN-10984) StackOverflowError following restart of scattered-cache with state-transfer awaitInitialTransfer disabled
by Pedro Zapata Fernandez (Jira)
[ https://issues.redhat.com/browse/ISPN-10984?page=com.atlassian.jira.plugi... ]
Pedro Zapata Fernandez updated ISPN-10984:
------------------------------------------
Sprint: DataGrid Sprint #37, DataGrid Sprint #38, DataGrid Sprint #39 (was: DataGrid Sprint #37, DataGrid Sprint #38)
> StackOverflowError following restart of scattered-cache with state-transfer awaitInitialTransfer disabled
> ---------------------------------------------------------------------------------------------------------
>
> Key: ISPN-10984
> URL: https://issues.redhat.com/browse/ISPN-10984
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.4.16.Final
> Reporter: Paul Ferraro
> Assignee: Dan Berindei
> Priority: Critical
> Fix For: 10.1.0.Final, 9.4.18.Final
>
>
> {noformat}
> 2019-11-24 18:30:00,837 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-3) MSC000001: Failed to start service jboss.clustering.web."clusterbench-ee8.ear.clusterbench-ee8-web.war": org.jboss.msc.service.StartException in service jboss.clustering.web."clusterbench-ee8.ear.clusterbench-ee8-web.war": org.infinispan.commons.CacheException: java.util.concurrent.ExecutionException: java.lang.StackOverflowError
> at org.wildfly.clustering.service@18.0.1.Final//org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:70)
> at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739)
> at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701)
> at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559)
> at org.jboss.threads@2.3.3.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
> at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
> at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: org.infinispan.commons.CacheException: java.util.concurrent.ExecutionException: java.lang.StackOverflowError
> at org.infinispan@9.4.16.Final//org.infinispan.interceptors.impl.PrefetchInterceptor$BackingIterator.hasNext(PrefetchInterceptor.java:651)
> at org.infinispan.commons@9.4.16.Final//org.infinispan.commons.util.IteratorMapper.hasNext(IteratorMapper.java:27)
> at org.wildfly.clustering.web.infinispan@18.0.1.Final//org.wildfly.clustering.web.infinispan.session.InfinispanSessionManagerFactory.schedule(InfinispanSessionManagerFactory.java:232)
> at org.wildfly.clustering.web.infinispan(a)18.0.1.Final//org.wildfly.clustering.web.infinispan.session.InfinispanSessionManagerFactory.<init>(InfinispanSessionManagerFactory.java:120)
> at org.wildfly.clustering.web.infinispan@18.0.1.Final//org.wildfly.clustering.web.infinispan.session.InfinispanSessionManagerFactoryServiceConfigurator.get(InfinispanSessionManagerFactoryServiceConfigurator.java:92)
> at org.wildfly.clustering.web.infinispan@18.0.1.Final//org.wildfly.clustering.web.infinispan.session.InfinispanSessionManagerFactoryServiceConfigurator.get(InfinispanSessionManagerFactoryServiceConfigurator.java:69)
> at org.wildfly.clustering.service@18.0.1.Final//org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:67)
> ... 8 more
> Caused by: java.util.concurrent.ExecutionException: java.lang.StackOverflowError
> at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
> at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2022)
> at org.infinispan@9.4.16.Final//org.infinispan.interceptors.impl.PrefetchInterceptor$BackingIterator.hasNext(PrefetchInterceptor.java:649)
> ... 14 more
> Caused by: java.lang.StackOverflowError
> at java.base/java.lang.Throwable.getMessage(Throwable.java:382)
> at java.base/java.lang.Throwable.getLocalizedMessage(Throwable.java:396)
> at java.base/java.lang.Throwable.toString(Throwable.java:485)
> at java.base/java.lang.Throwable.<init>(Throwable.java:316)
> at java.base/java.lang.Exception.<init>(Exception.java:102)
> at java.base/java.lang.RuntimeException.<init>(RuntimeException.java:96)
> at java.base/java.util.concurrent.CompletionException.<init>(CompletionException.java:88)
> at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
> at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1113)
> at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
> at org.infinispan@9.4.16.Final//org.infinispan.scattered.impl.ScatteredVersionManagerImpl.valuesFuture(ScatteredVersionManagerImpl.java:348)
> at org.infinispan@9.4.16.Final//org.infinispan.scattered.impl.ScatteredVersionManagerImpl.lambda$valuesFuture$3(ScatteredVersionManagerImpl.java:348)
> at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
> at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
> at org.infinispan@9.4.16.Final//org.infinispan.scattered.impl.ScatteredVersionManagerImpl.valuesFuture(ScatteredVersionManagerImpl.java:348)
> at org.infinispan@9.4.16.Final//org.infinispan.scattered.impl.ScatteredVersionManagerImpl.lambda$valuesFuture$3(ScatteredVersionManagerImpl.java:348)
> at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
> at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
> at org.infinispan@9.4.16.Final//org.infinispan.scattered.impl.ScatteredVersionManagerImpl.valuesFuture(ScatteredVersionManagerImpl.java:348)
> at org.infinispan@9.4.16.Final//org.infinispan.scattered.impl.ScatteredVersionManagerImpl.lambda$valuesFuture$3(ScatteredVersionManagerImpl.java:348)
> at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
> at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
> ...etc...
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 11 months