[jboss-jira] [JBoss JIRA] (WFLY-12414) Merged/Killed member handling in CacheRegistry triggers intermittent failures during server startup
Paul Ferraro (Jira)
issues at jboss.org
Thu Aug 22 12:35:01 EDT 2019
[ https://issues.jboss.org/browse/WFLY-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Ferraro moved JBEAP-17470 to WFLY-12414:
---------------------------------------------
Project: WildFly (was: JBoss Enterprise Application Platform)
Key: WFLY-12414 (was: JBEAP-17470)
Workflow: GIT Pull Request workflow (was: CDW with loose statuses v1)
Component/s: Clustering
(was: Clustering)
(was: OpenShift)
Affects Version/s: 17.0.1.Final
(was: 7.3.0.CD16)
> Merged/Killed member handling in CacheRegistry triggers intermittent failures during server startup
> ---------------------------------------------------------------------------------------------------
>
> Key: WFLY-12414
> URL: https://issues.jboss.org/browse/WFLY-12414
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 17.0.1.Final
> Reporter: Paul Ferraro
> Assignee: Paul Ferraro
> Priority: Critical
>
> Sometimes (1:100) one of EAPs does not start .
> So far we are seeing this only on OCP 4 jobs. Thus filling as OCP 4 bug.
> Currently automatic restart of not "Ready" pods does not work on OCP 4, so maybe that is why we see only on OCP .
> So far we hit mainly on Upstream Kube ping, but once we saw on Upstream DNS Ping.
> {code}
> [0m[31m11:24:39,946 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 77) MSC000001: Failed to start service org.wildfly.clustering.infinispan.cache.web.default-server: org.jboss.msc.service.StartException in service org.wildfly.clustering.infinispan.cache.web.default-server: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:70)
> at org.wildfly.clustering.service.AsyncServiceConfigurator$AsyncService.lambda$start$0(AsyncServiceConfigurator.java:117)
> at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> at java.lang.Thread.run(Thread.java:748)
> at org.jboss.threads.JBossThread.run(JBossThread.java:485)
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> at org.infinispan.security.Security.doPrivileged(Security.java:44)
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:244)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:646)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:591)
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:477)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:463)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:449)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:86)
> at org.wildfly.clustering.infinispan.spi.service.CacheServiceConfigurator.get(CacheServiceConfigurator.java:77)
> at org.wildfly.clustering.infinispan.spi.service.CacheServiceConfigurator.get(CacheServiceConfigurator.java:55)
> at org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:67)
> ... 7 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000451: Timed out waiting for view 2, current view is 1
> at org.infinispan.topology.LocalTopologyManagerImpl.waitForView(LocalTopologyManagerImpl.java:556)
> at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:151)
> at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> ... 30 more
> [0m[31m11:24:39,947 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 83) MSC000001: Failed to start service org.wildfly.clustering.infinispan.cache.web.ROOT.war: org.jboss.msc.service.StartException in service org.wildfly.clustering.infinispan.cache.web.ROOT.war: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:70)
> at org.wildfly.clustering.service.AsyncServiceConfigurator$AsyncService.lambda$start$0(AsyncServiceConfigurator.java:117)
> at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
> at java.lang.Thread.run(Thread.java:748)
> at org.jboss.threads.JBossThread.run(JBossThread.java:485)
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> at org.infinispan.security.Security.doPrivileged(Security.java:44)
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:244)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:646)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:591)
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:477)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:463)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:449)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:86)
> at org.wildfly.clustering.infinispan.spi.service.CacheServiceConfigurator.get(CacheServiceConfigurator.java:77)
> at org.wildfly.clustering.infinispan.spi.service.CacheServiceConfigurator.get(CacheServiceConfigurator.java:55)
> at org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:67)
> ... 7 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000451: Timed out waiting for view 2, current view is 1
> at org.infinispan.topology.LocalTopologyManagerImpl.waitForView(LocalTopologyManagerImpl.java:556)
> at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:151)
> at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> ... 30 more
> [0m[31m11:24:39,954 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "ROOT.war")]) - failure description: {"WFLYCTL0080: Failed services" => {"org.wildfly.clustering.infinispan.cache.web.ROOT.war" => "org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000451: Timed out waiting for view 2, current view is 1"}}
> [0m[31m11:24:39,956 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
> ("subsystem" => "undertow"),
> ("server" => "default-server")
> ]) - failure description: {"WFLYCTL0080: Failed services" => {"org.wildfly.clustering.infinispan.cache.web.default-server" => "org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000451: Timed out waiting for view 2, current view is 1"}}
> [0m[0m11:24:39,963 INFO [org.jboss.as.server] (ServerService Thread Pool -- 43) WFLYSRV0010: Deployed "ROOT.war" (runtime-name : "ROOT.war")
> [0m[0m11:24:39,967 INFO [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report
> WFLYCTL0186: Services which failed to start: service org.wildfly.clustering.infinispan.cache.web.ROOT.war: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> service org.wildfly.clustering.infinispan.cache.web.default-server: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> WFLYCTL0448: 9 additional services are down due to their dependencies being missing or failed
> [0m[0m11:24:40,047 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
> [0m[0m11:24:40,050 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:9990/management
> [0m[0m11:24:40,050 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0054: Admin console is not enabled
> [0m[31m11:24:40,050 ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: JBoss EAP 7.2.2.GA (WildFly Core 6.0.14.Final-redhat-00001) started (with errors) in 265662ms - Started 540 of 784 services (12 services failed or missing dependencies, 484 services are lazy, passive or on-demand)
> [0m[0m11:24:42,650 INFO [org.apache.activemq.artemis.core.server] (Thread-0 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5 at 554b59fa)) AMQ221056: Reloading configuration: logging
> [0m[33m11:24:42,650 WARN [org.apache.activemq.artemis.core.server] (Thread-0 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5 at 554b59fa)) AMQ222276: Failed to process changes to the logging configuration file: file:/opt/eap/standalone/configuration/logging.properties
> [0m[31m11:28:39,946 ERROR [org.infinispan.topology.LocalTopologyManagerImpl] (transport-thread--p19-t4) ISPN000230: Failed to start rebalance for cache default-server: org.infinispan.util.concurrent.TimeoutException: ISPN000451: Timed out waiting for view 2, current view is 1
> at org.infinispan.topology.LocalTopologyManagerImpl.waitForView(LocalTopologyManagerImpl.java:556)
> at org.infinispan.topology.LocalTopologyManagerImpl.doHandleRebalance(LocalTopologyManagerImpl.java:487)
> at org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleRebalance$3(LocalTopologyManagerImpl.java:475)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> at java.lang.Thread.run(Thread.java:748)
> [0m[31m11:28:39,948 ERROR [org.infinispan.topology.LocalTopologyManagerImpl] (transport-thread--p19-t5) ISPN000230: Failed to start rebalance for cache ROOT.war: org.infinispan.util.concurrent.TimeoutException: ISPN000451: Timed out waiting for view 2, current view is 1
> at org.infinispan.topology.LocalTopologyManagerImpl.waitForView(LocalTopologyManagerImpl.java:556)
> at org.infinispan.topology.LocalTopologyManagerImpl.doHandleRebalance(LocalTopologyManagerImpl.java:487)
> at org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleRebalance$3(LocalTopologyManagerImpl.java:475)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
More information about the jboss-jira
mailing list