[JBoss JIRA] (SWSQE-385) Kiali-Test-Meshes-Ansible Failure
by Hayk Hovsepyan (JIRA)
Hayk Hovsepyan created SWSQE-385:
------------------------------------
Summary: Kiali-Test-Meshes-Ansible Failure
Key: SWSQE-385
URL: https://issues.jboss.org/browse/SWSQE-385
Project: Kiali QE
Issue Type: Bug
Reporter: Hayk Hovsepyan
Assignee: Guilherme Baufaker Rêgo
Hi Guilherme,
Please take a look into this failure, it started happening today:
http://jenkins2.bc.jonqe.lab.eng.bos.redhat.com:8080/job/Kiali-Test-Meshe...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (WFLY-10837) IIOP subsystem requires port binding to be defined which was not necessary in prior WFLY versions
by Tomasz Adamski (JIRA)
[ https://issues.jboss.org/browse/WFLY-10837?page=com.atlassian.jira.plugin... ]
Tomasz Adamski commented on WFLY-10837:
---------------------------------------
[~baranowb] https://issues.jboss.org/browse/WFLY-10735 was related to the same refactor but it deals with different problem: POAs were using cleartext connection even in SSL-only configuration - that is already fixed
JacORB migration indeed has to be fixed - will do
> IIOP subsystem requires port binding to be defined which was not necessary in prior WFLY versions
> -------------------------------------------------------------------------------------------------
>
> Key: WFLY-10837
> URL: https://issues.jboss.org/browse/WFLY-10837
> Project: WildFly
> Issue Type: Bug
> Components: IIOP
> Reporter: Ondra Chaloupka
> Assignee: Tomasz Adamski
>
> If the {{standalone-*.xml}} configuration defines to use IIOP subsystem but it does not defines the port binding element
> {code}
> <orb socket-binding="iiop"/>
> {code}
> The server starts with error
> {code}
> ERROR [org.jboss.msc.service.fail] (MSC service thread 1-5) MSC000001: Failed to start service jboss.iiop-openjdk.orb-service: org.jboss.msc.service.StartException in service jboss.iiop-openjdk.orb-service: java.lang.IllegalStateException: WFLYIIOP0115: No IIOP socket bindings have been configured
> at org.wildfly.iiop.openjdk.service.CorbaORBService.start(CorbaORBService.java:150)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1736)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1698)
> at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1556)
> at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalStateException: WFLYIIOP0115: No IIOP socket bindings have been configured
> at org.wildfly.iiop.openjdk.service.CorbaORBService.start(CorbaORBService.java:109)
> ... 8 more
> {code}
> The attribute of the {{socket-binding}} in the model is not defined as {{required}}.
> {code}
> "socket-binding" => {
> "type" => STRING,
> "description" => "The name of the socket binding configuration that specifies the ORB port.",
> "attribute-group" => "orb",
> "expressions-allowed" => false,
> "required" => false,
> "nillable" => true,
> "min-length" => 1L,
> "max-length" => 2147483647L,
> "access-constraints" => {"sensitive" => {"socket-binding-ref" => {"type" => "core"}}},
> "access-type" => "read-write",
> "storage" => "configuration",
> "restart-required" => "all-services"
> }
> {code}
> Up to that declaring the iiop socket binding was not necessary in WildFly 13.0.0.Final. Could that be a backward compatibility problem too?
> Up to that
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (WFLY-10906) Tests regressions after WFLY-10480
by Darran Lofthouse (JIRA)
[ https://issues.jboss.org/browse/WFLY-10906?page=com.atlassian.jira.plugin... ]
Darran Lofthouse commented on WFLY-10906:
-----------------------------------------
Hi [~jim.ma] could you please confirm which piece of code it is you are referring to? If you believe it triggered a bug here we may need to apply the same fix elsewhere.
> Tests regressions after WFLY-10480
> ----------------------------------
>
> Key: WFLY-10906
> URL: https://issues.jboss.org/browse/WFLY-10906
> Project: WildFly
> Issue Type: Bug
> Components: Security, Web Services
> Affects Versions: 14.0.0.Beta2
> Reporter: Jan Blizňák
> Assignee: Jim Ma
> Priority: Blocker
>
> There are tests regressions after WFLY-10480 was merged which modified org.jboss.as.webservices.security.ElytronSecurityDomainContextImpl.
> Namely these tests in JBWS testsuite fail when running with elytron profile:
> UsernameAuthorizationDigestEjbTestCase.test
> UsernameAuthorizationDigestEjbTestCase.testConcurrent
> UsernameAuthorizationEJBTestCase.test
> All fail with the same message and stacktrace:
> {code:java}
> WFLYEJB0364: Invocation on method: public java.lang.String org.jboss.test.ws.jaxws.samples.wsse.policy.jaas.EJBDigestServiceImpl.sayHello() of bean: EjbEndpoint is not allowed
> {code}
> {code:java}
> 13:13:09,199 ERROR [org.jboss.as.ejb3.invocation] (default task-3) WFLYEJB0034: EJB Invocation failed on component EjbEndpoint for method public java.lang.String org.jboss.test.ws.jaxws.samples.wsse.policy.jaas.EJBDigestServiceImpl.sayHello(): javax.ejb.EJBAccessException: WFLYEJB0364: Invocation on method: public java.lang.String org.jboss.test.ws.jaxws.samples.wsse.policy.jaas.EJBDigestServiceImpl.sayHello() of bean: EjbEndpoint is not allowed
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.security.RolesAllowedInterceptor.processInvocation(RolesAllowedInterceptor.java:67)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.security.SecurityDomainInterceptor.processInvocation(SecurityDomainInterceptor.java:44)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee@14.0.0.CR1-SNAPSHOT//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
> at org.jboss.as.ee@14.0.0.CR1-SNAPSHOT//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.invokeInternal(AbstractInvocationHandler.java:161)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.lambda$invoke$0(AbstractInvocationHandler.java:116)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:382)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.Scoped.runAsFunctionEx(Scoped.java:161)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.Scoped.runAs(Scoped.java:73)
> at org.jboss.as.webservices//org.jboss.as.webservices.security.ElytronSecurityDomainContextImpl.runAs(ElytronSecurityDomainContextImpl.java:88)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:115)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.JBossWSInvoker.performInvocation(JBossWSInvoker.java:170)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
> at org.apache.cxf.impl//org.apache.cxf.jaxws.AbstractJAXWSMethodInvoker.invoke(AbstractJAXWSMethodInvoker.java:232)
> at org.apache.cxf.impl//org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:85)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.JBossWSInvoker.invoke(JBossWSInvoker.java:146)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor$2.run(ServiceInvokerInterceptor.java:126)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:131)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
> at org.apache.cxf.impl//org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.RequestHandlerImpl.handleHttpRequest(RequestHandlerImpl.java:110)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.transport.ServletHelper.callRequestHandler(ServletHelper.java:134)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.CXFServletExt.invoke(CXFServletExt.java:88)
> at org.apache.cxf.impl//org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:301)
> at org.apache.cxf.impl//org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:220)
> at javax.servlet.api@1.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:706)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.CXFServletExt.service(CXFServletExt.java:136)
> at org.jboss.ws.spi@3.2.2.Final//org.jboss.wsf.spi.deployment.WSFServlet.service(WSFServlet.java:140)
> at javax.servlet.api@1.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> at io.undertow.core@2.0.13.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> at io.undertow.core@2.0.13.Final//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> at java.base/java.lang.Thread.run(Thread.java:844)
> &#27;[0m&#27;[33m13:13:09,199 WARNING [org.apache.cxf.phase.PhaseInterceptorChain] (default task-3) Application {http://www.jboss.org/jbossws/ws-extensions/wssecuritypolicy}SecurityService#{http://www.jboss.org/jbossws/ws-extensions/wssecuritypolicy}sayHello has thrown exception, unwinding now: org.apache.cxf.interceptor.Fault: WFLYEJB0364: Invocation on method: public java.lang.String org.jboss.test.ws.jaxws.samples.wsse.policy.jaas.EJBDigestServiceImpl.sayHello() of bean: EjbEndpoint is not allowed
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162)
> at org.apache.cxf.impl//org.apache.cxf.jaxws.AbstractJAXWSMethodInvoker.createFault(AbstractJAXWSMethodInvoker.java:267)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:128)
> at org.apache.cxf.impl//org.apache.cxf.jaxws.AbstractJAXWSMethodInvoker.invoke(AbstractJAXWSMethodInvoker.java:232)
> at org.apache.cxf.impl//org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:85)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.JBossWSInvoker.invoke(JBossWSInvoker.java:146)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor$2.run(ServiceInvokerInterceptor.java:126)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:131)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
> at org.apache.cxf.impl//org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.RequestHandlerImpl.handleHttpRequest(RequestHandlerImpl.java:110)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.transport.ServletHelper.callRequestHandler(ServletHelper.java:134)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.CXFServletExt.invoke(CXFServletExt.java:88)
> at org.apache.cxf.impl//org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:301)
> at org.apache.cxf.impl//org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:220)
> at javax.servlet.api@1.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:706)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.CXFServletExt.service(CXFServletExt.java:136)
> at org.jboss.ws.spi@3.2.2.Final//org.jboss.wsf.spi.deployment.WSFServlet.service(WSFServlet.java:140)
> at javax.servlet.api@1.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> at io.undertow.core@2.0.13.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> at io.undertow.core@2.0.13.Final//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> at java.base/java.lang.Thread.run(Thread.java:844)
> Caused by: javax.ejb.EJBAccessException: WFLYEJB0364: Invocation on method: public java.lang.String org.jboss.test.ws.jaxws.samples.wsse.policy.jaas.EJBDigestServiceImpl.sayHello() of bean: EjbEndpoint is not allowed
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.security.RolesAllowedInterceptor.processInvocation(RolesAllowedInterceptor.java:67)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.security.SecurityDomainInterceptor.processInvocation(SecurityDomainInterceptor.java:44)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee@14.0.0.CR1-SNAPSHOT//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
> at org.jboss.as.ee@14.0.0.CR1-SNAPSHOT//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.invokeInternal(AbstractInvocationHandler.java:161)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.lambda$invoke$0(AbstractInvocationHandler.java:116)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:382)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.Scoped.runAsFunctionEx(Scoped.java:161)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.Scoped.runAs(Scoped.java:73)
> at org.jboss.as.webservices//org.jboss.as.webservices.security.ElytronSecurityDomainContextImpl.runAs(ElytronSecurityDomainContextImpl.java:88)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:115)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.JBossWSInvoker.performInvocation(JBossWSInvoker.java:170)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
> ... 62 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (WFWIP-119) [Artemis 2.x upgrade] Lost message in replicated HA scenario with MDB
by Erich Duda (JIRA)
[ https://issues.jboss.org/browse/WFWIP-119?page=com.atlassian.jira.plugin.... ]
Erich Duda edited comment on WFWIP-119 at 8/23/18 5:26 AM:
-----------------------------------------------------------
[~clebert.suconic] I've looked at \[1\] and \[2\].
\[1\] failed on "getCountOfMessagesOnQueue() failed for queue:OutQueue and container: node-1. Number of tries: 10". Boot of node-1 failed with an following exception. Do you know what could happen?
{code}
21:57:16,375 ERROR [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ224000: Failure in initialisation: java.lang.NullPointerException
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1745) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1181) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2620) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2375) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:376) [artemis-jms-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:206) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:65) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:100) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
at org.jboss.threads.JBossThread.run(JBossThread.java:485) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
21:57:16,375 ERROR [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ224000: Failure in initialisation: java.lang.NullPointerException
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1745) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPS
HOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1181) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2620) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2375) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:376) [artemis-jms-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:206) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:65) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:100) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
at org.jboss.threads.JBossThread.run(JBossThread.java:485) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
21:57:16,377 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 80) MSC000001: Failed to start service jboss.messaging-activemq.default.jms.manager: org.jboss.msc.service.StartException in service jboss.messaging-activemq.default.jms.manager: java.lang.NullPointerException
at org.wildfly.extension.messaging.activemq.jms.JMSService.lambda$doStart$0(JMSService.java:142) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.callActivationFailureListeners(ActiveMQServerImpl.java:1919) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:127) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:376) [artemis-jms-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:206) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:65) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:100) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
at org.jboss.threads.JBossThread.run(JBossThread.java:485) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
Caused by: java.lang.NullPointerException
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1745) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1181) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2620) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2375) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
... 14 more
{code}
The \[2\] failed because of the issue you mentioned that receiver didn't receive messages from node-4. However it is not test issue but it is caused by bug in JGroups. I saw in different test that if JGroups are used, Artemis cluster is not formed sometimes. It is reported in WFWIP-90. I will re-run the job with Netty discovery. Here is the new run \[3\]. If you will run the test by yourself be sure that there is {{-Dprepare.param.CLUSTER_TYPE=NETTY_DISCOVERY}} in {{ADDITIONAL_PARAMS}} field.
\[1\] https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messag...
\[2\] https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messag...
\[3\] https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messag...
was (Author: eduda):
[~clebert.suconic] I've looked at \[1\] and \[2\].
\[1\] failed on "getCountOfMessagesOnQueue() failed for queue:OutQueue and container: node-1. Number of tries: 10". Boot of node-1 failed with an following exception. Do you know what could happen?
{code}
21:57:16,375 ERROR [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ224000: Failure in initialisation: java.lang.NullPointerException
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1745) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1181) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2620) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2375) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:376) [artemis-jms-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:206) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:65) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:100) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
at org.jboss.threads.JBossThread.run(JBossThread.java:485) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
21:57:16,375 ERROR [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ224000: Failure in initialisation: java.lang.NullPointerException
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1745) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPS
HOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1181) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2620) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2375) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:376) [artemis-jms-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:206) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:65) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:100) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
at org.jboss.threads.JBossThread.run(JBossThread.java:485) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
21:57:16,377 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 80) MSC000001: Failed to start service jboss.messaging-activemq.default.jms.manager: org.jboss.msc.service.StartException in service jboss.messaging-activemq.default.jms.manager: java.lang.NullPointerException
at org.wildfly.extension.messaging.activemq.jms.JMSService.lambda$doStart$0(JMSService.java:142) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.callActivationFailureListeners(ActiveMQServerImpl.java:1919) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:127) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:376) [artemis-jms-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:206) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:65) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:100) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
at org.jboss.threads.JBossThread.run(JBossThread.java:485) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
Caused by: java.lang.NullPointerException
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1745) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1181) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2620) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2375) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
... 14 more
{code}
The \[2\] failed because of the issue you mentioned that receiver didn't receive messages from node-4. However it is not test issue but it is caused by bug in JGroups. I saw in different test that if JGroups are used, Artemis cluster is not formed sometimes. It is reported in WFWIP-90. I will re-run the job with Netty discovery. Here is the new run \[3\]. If you will run the test by yourself be sure that there is that there is {{-Dprepare.param.CLUSTER_TYPE=NETTY_DISCOVERY}} in {{ADDITIONAL_PARAMS}} field.
\[1\] https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messag...
\[2\] https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messag...
\[3\] https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messag...
> [Artemis 2.x upgrade] Lost message in replicated HA scenario with MDB
> ---------------------------------------------------------------------
>
> Key: WFWIP-119
> URL: https://issues.jboss.org/browse/WFWIP-119
> Project: WildFly WIP
> Issue Type: Bug
> Components: Artemis
> Reporter: Erich Duda
> Assignee: Martyn Taylor
> Priority: Blocker
>
> *Scenario*
> * Start live/backup pair in replicated dedicated topology and start one more server which is in Artemis cluster with live.
> * Send messages to queue InQueue.
> * Start another server with MDB connected to cluster with live server which resend messages from InQueue to OutQueue.
> * Stop backup server and then live server.
> * Stop server which was in cluster with live server.
> * Start live server and server in cluster.
> * Start backup server.
> The test fails because one message get lost. It wasn't sent to OutQueue.
> Logs from failed test including traces you can find in \[1\].
> \[1\] https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messag...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (WFWIP-119) [Artemis 2.x upgrade] Lost message in replicated HA scenario with MDB
by Erich Duda (JIRA)
[ https://issues.jboss.org/browse/WFWIP-119?page=com.atlassian.jira.plugin.... ]
Erich Duda commented on WFWIP-119:
----------------------------------
The run \[1\] finished and failed on issue with one lost message.
\[1\] https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messag...
> [Artemis 2.x upgrade] Lost message in replicated HA scenario with MDB
> ---------------------------------------------------------------------
>
> Key: WFWIP-119
> URL: https://issues.jboss.org/browse/WFWIP-119
> Project: WildFly WIP
> Issue Type: Bug
> Components: Artemis
> Reporter: Erich Duda
> Assignee: Martyn Taylor
> Priority: Blocker
>
> *Scenario*
> * Start live/backup pair in replicated dedicated topology and start one more server which is in Artemis cluster with live.
> * Send messages to queue InQueue.
> * Start another server with MDB connected to cluster with live server which resend messages from InQueue to OutQueue.
> * Stop backup server and then live server.
> * Stop server which was in cluster with live server.
> * Start live server and server in cluster.
> * Start backup server.
> The test fails because one message get lost. It wasn't sent to OutQueue.
> Logs from failed test including traces you can find in \[1\].
> \[1\] https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messag...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (WFWIP-119) [Artemis 2.x upgrade] Lost message in replicated HA scenario with MDB
by Erich Duda (JIRA)
[ https://issues.jboss.org/browse/WFWIP-119?page=com.atlassian.jira.plugin.... ]
Erich Duda edited comment on WFWIP-119 at 8/23/18 5:19 AM:
-----------------------------------------------------------
[~clebert.suconic] I've looked at \[1\] and \[2\].
\[1\] failed on "getCountOfMessagesOnQueue() failed for queue:OutQueue and container: node-1. Number of tries: 10". Boot of node-1 failed with an following exception. Do you know what could happen?
{code}
21:57:16,375 ERROR [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ224000: Failure in initialisation: java.lang.NullPointerException
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1745) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1181) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2620) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2375) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:376) [artemis-jms-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:206) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:65) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:100) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
at org.jboss.threads.JBossThread.run(JBossThread.java:485) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
21:57:16,375 ERROR [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ224000: Failure in initialisation: java.lang.NullPointerException
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1745) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPS
HOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1181) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2620) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2375) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:376) [artemis-jms-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:206) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:65) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:100) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
at org.jboss.threads.JBossThread.run(JBossThread.java:485) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
21:57:16,377 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 80) MSC000001: Failed to start service jboss.messaging-activemq.default.jms.manager: org.jboss.msc.service.StartException in service jboss.messaging-activemq.default.jms.manager: java.lang.NullPointerException
at org.wildfly.extension.messaging.activemq.jms.JMSService.lambda$doStart$0(JMSService.java:142) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.callActivationFailureListeners(ActiveMQServerImpl.java:1919) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:127) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:376) [artemis-jms-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:206) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:65) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:100) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
at org.jboss.threads.JBossThread.run(JBossThread.java:485) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
Caused by: java.lang.NullPointerException
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1745) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1181) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2620) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2375) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
... 14 more
{code}
The \[2\] failed because of the issue you mentioned that receiver didn't receive messages from node-4. However it is not test issue but it is caused by bug in JGroups. I saw in different test that if JGroups are used, Artemis cluster is not formed sometimes. It is reported in WFWIP-90. I will re-run the job with Netty discovery. Here is the new run \[3\]. If you will run the test by yourself be sure that there is that there is {{-Dprepare.param.CLUSTER_TYPE=NETTY_DISCOVERY}} in {{ADDITIONAL_PARAMS}} field.
\[1\] https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messag...
\[2\] https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messag...
\[3\] https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messag...
was (Author: eduda):
[~clebert.suconic] I've looked at \[1\] and \[2\].
\[1\] failed on "getCountOfMessagesOnQueue() failed for queue:OutQueue and container: node-1. Number of tries: 10". Boot of node-1 failed with an following exception. Do you know what could happen?
{code}
21:57:16,375 ERROR [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ224000: Failure in initialisation: java.lang.NullPointerException
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1745) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1181) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2620) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2375) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:376) [artemis-jms-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:206) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:65) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:100) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
at org.jboss.threads.JBossThread.run(JBossThread.java:485) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
21:57:16,375 ERROR [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ224000: Failure in initialisation: java.lang.NullPointerException
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1745) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPS
HOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1181) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2620) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2375) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:376) [artemis-jms-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:206) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:65) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:100) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
at org.jboss.threads.JBossThread.run(JBossThread.java:485) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
21:57:16,377 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 80) MSC000001: Failed to start service jboss.messaging-activemq.default.jms.manager: org.jboss.msc.service.StartException in service jboss.messaging-activemq.default.jms.manager: java.lang.NullPointerException
at org.wildfly.extension.messaging.activemq.jms.JMSService.lambda$doStart$0(JMSService.java:142) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.callActivationFailureListeners(ActiveMQServerImpl.java:1919) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:127) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:376) [artemis-jms-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:206) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:65) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:100) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
at org.jboss.threads.JBossThread.run(JBossThread.java:485) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
Caused by: java.lang.NullPointerException
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1745) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1181) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2620) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2375) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
... 14 more
{code}
The \[2\] failed because of the issue you mentioned that receiver didn't receive messages from node-4. However it is not test issue but it is caused by bug in JGroups. I saw in different test that if JGroups are used, Artemis cluster is not formed sometimes. It is reported in WFWIP-90. I will re-run the job with Netty discovery. Here is the new run \[1\]. If you will run the test by yourself be sure that there is that there is {{-Dprepare.param.CLUSTER_TYPE=NETTY_DISCOVERY}} in {{ADDITIONAL_PARAMS}} field.
\[1\] https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messag...
\[2\] https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messag...
\[3\] https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messag...
> [Artemis 2.x upgrade] Lost message in replicated HA scenario with MDB
> ---------------------------------------------------------------------
>
> Key: WFWIP-119
> URL: https://issues.jboss.org/browse/WFWIP-119
> Project: WildFly WIP
> Issue Type: Bug
> Components: Artemis
> Reporter: Erich Duda
> Assignee: Martyn Taylor
> Priority: Blocker
>
> *Scenario*
> * Start live/backup pair in replicated dedicated topology and start one more server which is in Artemis cluster with live.
> * Send messages to queue InQueue.
> * Start another server with MDB connected to cluster with live server which resend messages from InQueue to OutQueue.
> * Stop backup server and then live server.
> * Stop server which was in cluster with live server.
> * Start live server and server in cluster.
> * Start backup server.
> The test fails because one message get lost. It wasn't sent to OutQueue.
> Logs from failed test including traces you can find in \[1\].
> \[1\] https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messag...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (WFLY-10876) Align jar file provided for MP OpenTracing into right modules
by Jan Stourac (JIRA)
[ https://issues.jboss.org/browse/WFLY-10876?page=com.atlassian.jira.plugin... ]
Jan Stourac commented on WFLY-10876:
------------------------------------
Note that {{gson}} dependency is moved to its own module by [this MR|https://github.com/wildfly/wildfly/pull/11556].
> Align jar file provided for MP OpenTracing into right modules
> -------------------------------------------------------------
>
> Key: WFLY-10876
> URL: https://issues.jboss.org/browse/WFLY-10876
> Project: WildFly
> Issue Type: Bug
> Components: MP OpenTracing
> Affects Versions: 14.0.0.Beta2
> Reporter: Jan Stourac
> Assignee: Juraci Paixão Kröhling
> Priority: Critical
>
> With recent addition of MicroProfile OpenTracing implementation we currently have following set of jar files that are in the build to provide various MicroProfile features:
> {code}
> $ find modules | grep smallrye | grep jar$ | xargs ls -hl
> -rw-rw-r--. 1 jstourac jstourac 39K Aug 16 15:13 modules/system/layers/base/io/smallrye/config/main/smallrye-config-1.3.3.jar
> -rw-rw-r--. 1 jstourac jstourac 8.9K Aug 16 15:13 modules/system/layers/base/io/smallrye/health/main/smallrye-health-1.0.2.jar
> -rw-rw-r--. 1 jstourac jstourac 38K Aug 16 15:13 modules/system/layers/base/org/wildfly/extension/microprofile/config-smallrye/main/wildfly-microprofile-config-smallrye-14.0.0.CR1-SNAPSHOT.jar
> -rw-rw-r--. 1 jstourac jstourac 27K Aug 16 15:13 modules/system/layers/base/org/wildfly/extension/microprofile/health-smallrye/main/wildfly-microprofile-health-smallrye-14.0.0.CR1-SNAPSHOT.jar
> -rw-rw-r--. 1 jstourac jstourac 228K Aug 16 15:13 modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/gson-2.8.2.jar
> -rw-rw-r--. 1 jstourac jstourac 123K Aug 16 15:13 modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/jaeger-core-0.30.4.jar
> -rw-rw-r--. 1 jstourac jstourac 2.8M Aug 16 15:13 modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/jaeger-thrift-0.30.4.jar
> -rw-rw-r--. 1 jstourac jstourac 14K Aug 16 15:13 modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/opentracing-api-0.31.0.jar
> -rw-rw-r--. 1 jstourac jstourac 7.5K Aug 16 15:13 modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/opentracing-concurrent-0.1.0.jar
> -rw-rw-r--. 1 jstourac jstourac 41K Aug 16 15:13 modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/opentracing-jaxrs2-0.1.6.jar
> -rw-rw-r--. 1 jstourac jstourac 11K Aug 16 15:13 modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/opentracing-noop-0.31.0.jar
> -rw-rw-r--. 1 jstourac jstourac 7.4K Aug 16 15:13 modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/opentracing-tracerresolver-0.1.5.jar
> -rw-rw-r--. 1 jstourac jstourac 7.9K Aug 16 15:13 modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/opentracing-util-0.31.0.jar
> -rw-rw-r--. 1 jstourac jstourac 13K Aug 16 15:13 modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/opentracing-web-servlet-filter-0.1.0.jar
> -rw-rw-r--. 1 jstourac jstourac 5.4K Aug 16 15:13 modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/smallrye-opentracing-1.1.1.jar
> -rw-rw-r--. 1 jstourac jstourac 11K Aug 16 15:13 modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/wildfly-microprofile-opentracing-smallrye-14.0.0.CR1-SNAPSHOT.jar
> {code}
> It looks like the opentracing-smallrye module could be split up into a few separate modules. There at least two motivations:
> # particular implementation into its own module separate from the e.g. microprofile specification
> # because of the way how current patching mechanism works in WildFly, if we leave current {{opentracing-smallrye}} module as is, in case of a one-off patch it would mean to replace all files in the module - currently 3.3MB:
> {code}
> $ du -h modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/
> 3.3M modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/
> {code}
> My proposal how to align the jar files is following:
> # jaeger specific files should go into separate jaeger module
> # {{wildfly-microprofile-opentracing-smallrye-14.0.0.CR1-SNAPSHOT.jar}} should be moved to new {{modules/system/layers/base/org/wildfly/extension/microprofile/opentracing-smallrye}} module so it matches what is done for {{health-smallrye}} and {{config-smallrye}} in extension subdir
> # {{smallrye-opentracing-1.1.1.jar}} should be moved to {{base/io}} as is done for {{config}} and {{health}} - {{modules/system/layers/base/io/smallrye/opentracing/main/smallrye-opentracing-1.1.1.jar}}
> # what about that {{gson-2.8.2.jar}} dependency? Shall it be moved to some more generic place?
> This would give us following structure:
> {code}
> modules/system/layers/base/io/smallrye/config/main/smallrye-config-1.3.3.jar
> modules/system/layers/base/io/smallrye/health/main/smallrye-health-1.0.2.jar
> modules/system/layers/base/io/smallrye/opentracing/main/smallrye-opentracing-1.1.1.jar
> modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/opentracing-api-0.31.0.jar
> modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/opentracing-jaxrs2-0.1.6.jar
> modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/opentracing-concurrent-0.1.0.jar
> modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/opentracing-web-servlet-filter-0.1.0.jar
> modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/opentracing-noop-0.31.0.jar
> modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/opentracing-tracerresolver-0.1.5.jar
> modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/opentracing-util-0.31.0.jar
> modules/system/layers/base/org/wildfly/extension/microprofile/health-smallrye/main/wildfly-microprofile-health-smallrye-14.0.0.CR1-SNAPSHOT.jar
> modules/system/layers/base/org/wildfly/extension/microprofile/config-smallrye/main/wildfly-microprofile-config-smallrye-14.0.0.CR1-SNAPSHOT.jar
> modules/system/layers/base/org/wildfly/extension/microprofile/opentracing-smallrye/main/wildfly-microprofile-opentracing-smallrye-14.0.0.CR1-SNAPSHOT.jar
> ---- I have no idea how this module for jaeger should be named...
> modules/system/layers/base/org/wildfly/microprofile/jaeger/opentracing-smallrye/main/jaeger-thrift-0.30.4.jar
> modules/system/layers/base/org/wildfly/microprofile/jaeger/opentracing-smallrye/main/jaeger-core-0.30.4.jar
> ---- I have no idea how this module for gson should be named...
> modules/system/layers/base/org/wildfly/gson/main/gson-2.8.2.jar
> {code}
> One additional piece for consideration - maybe we should also split up jar files that are just the microprofile-opentracing specification (api at least I guess) apart from the others in the {{modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/}}
> What do you think?
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (WFLY-10877) Jaeger-thrift jar dependency is quite chubby
by Jan Stourac (JIRA)
[ https://issues.jboss.org/browse/WFLY-10877?page=com.atlassian.jira.plugin... ]
Jan Stourac commented on WFLY-10877:
------------------------------------
Note - size of the artifacts after build from the MR branch:
{code}
$ ll -h ./system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/
total 952K
-rw-rw-r--. 1 jstourac jstourac 125K Aug 23 10:55 jaeger-core-0.30.5.jar
-rw-rw-r--. 1 jstourac jstourac 682K Aug 23 10:55 jaeger-thrift-0.30.5.jar
-rw-rw-r--. 1 jstourac jstourac 2.1K Aug 23 10:55 module.xml
-rw-rw-r--. 1 jstourac jstourac 14K Aug 23 10:55 opentracing-api-0.31.0.jar
-rw-rw-r--. 1 jstourac jstourac 7.5K Aug 23 10:55 opentracing-concurrent-0.1.0.jar
-rw-rw-r--. 1 jstourac jstourac 41K Aug 23 10:55 opentracing-jaxrs2-0.1.6.jar
-rw-rw-r--. 1 jstourac jstourac 11K Aug 23 10:55 opentracing-noop-0.31.0.jar
-rw-rw-r--. 1 jstourac jstourac 7.4K Aug 23 10:55 opentracing-tracerresolver-0.1.5.jar
-rw-rw-r--. 1 jstourac jstourac 7.9K Aug 23 10:55 opentracing-util-0.31.0.jar
-rw-rw-r--. 1 jstourac jstourac 13K Aug 23 10:55 opentracing-web-servlet-filter-0.1.0.jar
-rw-rw-r--. 1 jstourac jstourac 5.4K Aug 23 10:55 smallrye-opentracing-1.1.1.jar
-rw-rw-r--. 1 jstourac jstourac 13K Aug 23 10:55 wildfly-microprofile-opentracing-smallrye-14.0.0.CR1-SNAPSHOT.jar
{code}
> Jaeger-thrift jar dependency is quite chubby
> --------------------------------------------
>
> Key: WFLY-10877
> URL: https://issues.jboss.org/browse/WFLY-10877
> Project: WildFly
> Issue Type: Bug
> Components: MP OpenTracing
> Reporter: Jan Stourac
> Assignee: Juraci Paixão Kröhling
>
> Jaeger-thrift dependency file is quite chubby:
> {code}
> -rw-rw-r--. 1 jstourac jstourac 123K Aug 16 15:13 modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/jaeger-core-0.30.4.jar
> -rw-rw-r--. 1 jstourac jstourac 2.8M Aug 16 15:13 modules/system/layers/base/org/wildfly/microprofile/opentracing-smallrye/main/jaeger-thrift-0.30.4.jar
> {code}
> Is there any possibility to shrink it's size somehow?
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (WFLY-10906) Tests regressions after WFLY-10480
by Jim Ma (JIRA)
[ https://issues.jboss.org/browse/WFLY-10906?page=com.atlassian.jira.plugin... ]
Jim Ma edited comment on WFLY-10906 at 8/23/18 5:03 AM:
--------------------------------------------------------
There is small issue in elytron team provided code which converts subject to securityIdentity : identity.withRoleMapper(roleCategory, (rolesToMap) -> Roles.fromSet(roles)) doesn't add roleMapper to this "identity". To fix this this roleMapper loss , simply returns the identity.withRoleMapper()'s result when roles is not empty : https://github.com/wildfly/wildfly/pull/11565
was (Author: jim.ma):
There is small issue in elytron team provided code which converts subject to securityIdentity : identity.withRoleMapper(roleCategory, (rolesToMap) -> Roles.fromSet(roles)) doesn't add roleMapper to this "identity". To fix this this roleMapper loss , simply returns the identity.withRoleMapper()'s result when roles is not empty: https://github.com/jimma/wildfly/commit/6014fff1d28284d56d768514737885de4.... PR is sent : https://github.com/wildfly/wildfly/pull/11565
> Tests regressions after WFLY-10480
> ----------------------------------
>
> Key: WFLY-10906
> URL: https://issues.jboss.org/browse/WFLY-10906
> Project: WildFly
> Issue Type: Bug
> Components: Security, Web Services
> Affects Versions: 14.0.0.Beta2
> Reporter: Jan Blizňák
> Assignee: Jim Ma
> Priority: Blocker
>
> There are tests regressions after WFLY-10480 was merged which modified org.jboss.as.webservices.security.ElytronSecurityDomainContextImpl.
> Namely these tests in JBWS testsuite fail when running with elytron profile:
> UsernameAuthorizationDigestEjbTestCase.test
> UsernameAuthorizationDigestEjbTestCase.testConcurrent
> UsernameAuthorizationEJBTestCase.test
> All fail with the same message and stacktrace:
> {code:java}
> WFLYEJB0364: Invocation on method: public java.lang.String org.jboss.test.ws.jaxws.samples.wsse.policy.jaas.EJBDigestServiceImpl.sayHello() of bean: EjbEndpoint is not allowed
> {code}
> {code:java}
> 13:13:09,199 ERROR [org.jboss.as.ejb3.invocation] (default task-3) WFLYEJB0034: EJB Invocation failed on component EjbEndpoint for method public java.lang.String org.jboss.test.ws.jaxws.samples.wsse.policy.jaas.EJBDigestServiceImpl.sayHello(): javax.ejb.EJBAccessException: WFLYEJB0364: Invocation on method: public java.lang.String org.jboss.test.ws.jaxws.samples.wsse.policy.jaas.EJBDigestServiceImpl.sayHello() of bean: EjbEndpoint is not allowed
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.security.RolesAllowedInterceptor.processInvocation(RolesAllowedInterceptor.java:67)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.security.SecurityDomainInterceptor.processInvocation(SecurityDomainInterceptor.java:44)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee@14.0.0.CR1-SNAPSHOT//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
> at org.jboss.as.ee@14.0.0.CR1-SNAPSHOT//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.invokeInternal(AbstractInvocationHandler.java:161)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.lambda$invoke$0(AbstractInvocationHandler.java:116)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:382)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.Scoped.runAsFunctionEx(Scoped.java:161)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.Scoped.runAs(Scoped.java:73)
> at org.jboss.as.webservices//org.jboss.as.webservices.security.ElytronSecurityDomainContextImpl.runAs(ElytronSecurityDomainContextImpl.java:88)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:115)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.JBossWSInvoker.performInvocation(JBossWSInvoker.java:170)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
> at org.apache.cxf.impl//org.apache.cxf.jaxws.AbstractJAXWSMethodInvoker.invoke(AbstractJAXWSMethodInvoker.java:232)
> at org.apache.cxf.impl//org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:85)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.JBossWSInvoker.invoke(JBossWSInvoker.java:146)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor$2.run(ServiceInvokerInterceptor.java:126)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:131)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
> at org.apache.cxf.impl//org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.RequestHandlerImpl.handleHttpRequest(RequestHandlerImpl.java:110)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.transport.ServletHelper.callRequestHandler(ServletHelper.java:134)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.CXFServletExt.invoke(CXFServletExt.java:88)
> at org.apache.cxf.impl//org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:301)
> at org.apache.cxf.impl//org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:220)
> at javax.servlet.api@1.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:706)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.CXFServletExt.service(CXFServletExt.java:136)
> at org.jboss.ws.spi@3.2.2.Final//org.jboss.wsf.spi.deployment.WSFServlet.service(WSFServlet.java:140)
> at javax.servlet.api@1.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> at io.undertow.core@2.0.13.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> at io.undertow.core@2.0.13.Final//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> at java.base/java.lang.Thread.run(Thread.java:844)
> &#27;[0m&#27;[33m13:13:09,199 WARNING [org.apache.cxf.phase.PhaseInterceptorChain] (default task-3) Application {http://www.jboss.org/jbossws/ws-extensions/wssecuritypolicy}SecurityService#{http://www.jboss.org/jbossws/ws-extensions/wssecuritypolicy}sayHello has thrown exception, unwinding now: org.apache.cxf.interceptor.Fault: WFLYEJB0364: Invocation on method: public java.lang.String org.jboss.test.ws.jaxws.samples.wsse.policy.jaas.EJBDigestServiceImpl.sayHello() of bean: EjbEndpoint is not allowed
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162)
> at org.apache.cxf.impl//org.apache.cxf.jaxws.AbstractJAXWSMethodInvoker.createFault(AbstractJAXWSMethodInvoker.java:267)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:128)
> at org.apache.cxf.impl//org.apache.cxf.jaxws.AbstractJAXWSMethodInvoker.invoke(AbstractJAXWSMethodInvoker.java:232)
> at org.apache.cxf.impl//org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:85)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.JBossWSInvoker.invoke(JBossWSInvoker.java:146)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor$2.run(ServiceInvokerInterceptor.java:126)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:131)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
> at org.apache.cxf.impl//org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.RequestHandlerImpl.handleHttpRequest(RequestHandlerImpl.java:110)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.transport.ServletHelper.callRequestHandler(ServletHelper.java:134)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.CXFServletExt.invoke(CXFServletExt.java:88)
> at org.apache.cxf.impl//org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:301)
> at org.apache.cxf.impl//org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:220)
> at javax.servlet.api@1.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:706)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.CXFServletExt.service(CXFServletExt.java:136)
> at org.jboss.ws.spi@3.2.2.Final//org.jboss.wsf.spi.deployment.WSFServlet.service(WSFServlet.java:140)
> at javax.servlet.api@1.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> at io.undertow.core@2.0.13.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> at io.undertow.core@2.0.13.Final//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> at java.base/java.lang.Thread.run(Thread.java:844)
> Caused by: javax.ejb.EJBAccessException: WFLYEJB0364: Invocation on method: public java.lang.String org.jboss.test.ws.jaxws.samples.wsse.policy.jaas.EJBDigestServiceImpl.sayHello() of bean: EjbEndpoint is not allowed
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.security.RolesAllowedInterceptor.processInvocation(RolesAllowedInterceptor.java:67)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.security.SecurityDomainInterceptor.processInvocation(SecurityDomainInterceptor.java:44)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee@14.0.0.CR1-SNAPSHOT//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
> at org.jboss.as.ee@14.0.0.CR1-SNAPSHOT//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.invokeInternal(AbstractInvocationHandler.java:161)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.lambda$invoke$0(AbstractInvocationHandler.java:116)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:382)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.Scoped.runAsFunctionEx(Scoped.java:161)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.Scoped.runAs(Scoped.java:73)
> at org.jboss.as.webservices//org.jboss.as.webservices.security.ElytronSecurityDomainContextImpl.runAs(ElytronSecurityDomainContextImpl.java:88)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:115)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.JBossWSInvoker.performInvocation(JBossWSInvoker.java:170)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
> ... 62 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (WFLY-10736) Server in cluster hangs during start after previous kill
by tommaso borgato (JIRA)
[ https://issues.jboss.org/browse/WFLY-10736?page=com.atlassian.jira.plugin... ]
tommaso borgato edited comment on WFLY-10736 at 8/23/18 5:03 AM:
-----------------------------------------------------------------
[~pferraro] With both distributions the server often fails to shutdown within 60 seconds and the jvm is killed: "Server shutdown timed out. Terminating forcefully".
In both cases we observe the WARN log message: "ISPN000197: Error updating cluster member list".
In the test performed on distribution eap-7.2.0.CD14-20180815, the server happend to fail to stop both during failover and at the end of the test.
In the test performed on wildfly:master build 22 Aug 2018, we always observe the WARN log message "ISPN000071: Caught exception when handling command LockControlCommand{...}" as the last log from the server before being forcefully terminated.
In the test performed on wildfly:master build 22 Aug 2018, the server fails to stop just during failover but stop normally at the end of the test.
h2. Distribution eap-7.2.0.CD14-20180815
h3. [perf18|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
failover:
{noformat}
[JBossINF] [0m[0m16:13:04,626 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel ejb
[JBossINF] [0m[0m16:13:04,660 INFO [org.jboss.as] (MSC service thread 1-7) WFLYSRV0050: JBoss EAP CD 7.2.0.CD14 (WildFly Core 6.0.0.Final-redhat-20180815) stopped in 10153ms
{noformat}
end of test:
{noformat}
[JBossINF] [0m[0m16:28:20,518 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel ejb
2018/08/21 16:30:10:437 EDT [DEBUG][RMI TCP Connection(61)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
h3. [perf19|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
failover:
{noformat}
[JBossINF] [0m[0m16:15:20,382 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000080: Disconnecting JGroups channel ejb
...
2018/08/21 16:17:20:263 EDT [DEBUG][RMI TCP Connection(44)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
end of test:
{noformat}
[JBossINF] [0m[0m16:30:23,976 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel ejb
2018/08/21 16:32:13:927 EDT [DEBUG][RMI TCP Connection(53)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
h3. [perf20|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
failover:
{noformat}
[JBossINF] [0m[0m16:19:34,766 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000080: Disconnecting JGroups channel ejb
...
2018/08/21 16:21:34:692 EDT [DEBUG][RMI TCP Connection(31)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
end of test:
{noformat}
[JBossINF] [0m[0m16:35:37,668 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel ejb
2018/08/21 16:37:27:615 EDT [DEBUG][RMI TCP Connection(42)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
h3. [perf21|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
failover:
{noformat}
[JBossINF] [0m[0m16:23:49,076 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000080: Disconnecting JGroups channel ejb
...
2018/08/21 16:25:49:018 EDT [DEBUG][RMI TCP Connection(22)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
h2. Distribution https://github.com/wildfly/wildfly refs/heads/master (22 Aug 2018)
h3. [perf18|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
OK
h3. [perf19|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
failover:
{noformat}
[JBossINF] [0m[33m11:42:26,122 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p7-t4) ISPN000071: Caught exception when handling command LockControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, keys=[SessionCreationMetaDataKey(FQgZzMUpaFIWVPc7RKrrbuacvIyOjApj1EWlKKqc)], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTx:perf18:37}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:perf18:37, already completed
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:388)
[JBossINF] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:381)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:361)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:139)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.invokeAsync(LockControlCommand.java:122)
[JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF]
[JBossINF] [0m[33m11:42:35,287 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf20:336 timed out. Rolling back after 74863 ms
[JBossINF] [0m[33m11:42:35,287 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf20:334 timed out. Rolling back after 76777 ms
[JBossINF] [0m[33m11:42:35,287 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf21:331 timed out. Rolling back after 74864 ms
[JBossINF] [0m[33m11:42:35,287 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf18:31 timed out. Rolling back after 74787 ms
2018/08/22 11:43:17:160 EDT [DEBUG][RMI TCP Connection(44)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
meanwhile on perf18 we observe:
{noformat}
[JBossINF] [0m[31m11:42:26,135 ERROR [io.undertow.request] (default task-2) UT005071: Undertow request failed HttpServerExchange{ GET /clusterbench/session request {Cookie=[$Version=0; JSESSIONID=FQgZzMUpaFIWVPc7RKrrbuacvIyOjApj1EWlKKqc.perf19; $Path=/clusterbench], Content-Length=[0], User-Agent=[Jakarta Commons-HttpClient/3.1], Host=[perf17:8080]} response {Content-Type=[text/html;charset=UTF-8], Content-Length=[80], Date=[Wed, 22 Aug 2018 15:42:06 GMT]}}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 56 from perf19
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 56 from perf19
[JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] ... 1 more
[JBossINF] Suppressed: org.infinispan.util.logging.TraceException
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41)
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
[JBossINF] ... 1 more
{noformat}
h3. [perf20|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
failover:
{noformat}
[JBossINF] [0m[33m11:46:40,348 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p5-t11) ISPN000071: Caught exception when handling command LockControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, keys=[SessionCreationMetaDataKey(jKed4QLgTwwLw_q5p34roD_P-Bygv8umfynvDuHB)], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTx:perf19:53}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:perf19:53, already completed
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:388)
[JBossINF] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:381)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:361)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:139)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.invokeAsync(LockControlCommand.java:122)
[JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF]
[JBossINF] [0m[33m11:46:56,082 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf21:952 timed out. Rolling back after 83324 ms
[JBossINF] [0m[33m11:46:56,082 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf21:954 timed out. Rolling back after 81452 ms
[JBossINF] [0m[33m11:46:56,082 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf21:950 timed out. Rolling back after 83328 ms
[JBossINF] [0m[33m11:46:56,083 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf19:46 timed out. Rolling back after 81377 ms
[JBossINF] [0m[33m11:46:56,083 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf18:42 timed out. Rolling back after 83337 ms
2018/08/22 11:47:31:216 EDT [DEBUG][RMI TCP Connection(31)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
meanwhile on perf19:
{noformat}
[JBossINF] [0m[31m11:46:40,357 ERROR [io.undertow.request] (default task-2) UT005071: Undertow request failed HttpServerExchange{ GET /clusterbench/session request {Cookie=[$Version=0; JSESSIONID=jKed4QLgTwwLw_q5p34roD_P-Bygv8umfynvDuHB.perf20; $Path=/clusterbench], Content-Length=[0], User-Agent=[Jakarta Commons-HttpClient/3.1], Host=[perf17:8080]} response {Content-Type=[text/html;charset=UTF-8], Content-Length=[80], Date=[Wed, 22 Aug 2018 15:46:20 GMT]}}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 57 from perf20
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 57 from perf20
[JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] ... 1 more
[JBossINF] Suppressed: org.infinispan.util.logging.TraceException
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41)
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
[JBossINF] ... 1 more
{noformat}
h3. [perf21|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
failover:
{noformat}
[JBossINF] [0m[33m11:50:54,509 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p6-t20) ISPN000071: Caught exception when handling command LockControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, keys=[SessionCreationMetaDataKey(VKJbgfNUhPq0zSCYr-QmL_TNDcqv85k1Y20kOSqf)], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTx:perf19:139}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:perf19:139, already completed
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:388)
[JBossINF] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:381)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:361)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:139)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.invokeAsync(LockControlCommand.java:122)
[JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF]
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf20:47 timed out. Rolling back after 72674 ms
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf20:46 timed out. Rolling back after 72676 ms
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf20:49 timed out. Rolling back after 72677 ms
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf20:48 timed out. Rolling back after 72675 ms
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf19:81 timed out. Rolling back after 70864 ms
2018/08/22 11:51:45:682 EDT [DEBUG][RMI TCP Connection(24)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
meanwhile on perf19:
{noformat}
[JBossINF] [0m[31m11:50:54,512 ERROR [io.undertow.request] (default task-3) UT005071: Undertow request failed HttpServerExchange{ GET /clusterbench/session request {Cookie=[$Version=0; JSESSIONID=VKJbgfNUhPq0zSCYr-QmL_TNDcqv85k1Y20kOSqf.perf21; $Path=/clusterbench], Content-Length=[0], User-Agent=[Jakarta Commons-HttpClient/3.1], Host=[perf17:8080]} response {Content-Type=[text/html;charset=UTF-8], Content-Length=[80], Date=[Wed, 22 Aug 2018 15:50:34 GMT]}}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 135 from perf21
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 135 from perf21
[JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] ... 1 more
[JBossINF] Suppressed: org.infinispan.util.logging.TraceException
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41)
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
[JBossINF] ... 1 more
{noformat}
was (Author: tommaso-borgato):
[~pferraro] With both distributions the server often fails to shutdown within 60 seconds and the jvm is killed: "Server shutdown timed out. Terminating forcefully".
In both cases we observe the WARN log message: "ISPN000197: Error updating cluster member list".
In the test performed on wildfly:master build 22 Aug 2018, we always observe the WARN log message "ISPN000071: Caught exception when handling command LockControlCommand{...}" as the last log from the server before being forcefully terminated.
h2. eap-7.2.0.CD14-20180815
h3. [perf18|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[0m16:13:04,626 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel ejb
[JBossINF] [0m[0m16:13:04,660 INFO [org.jboss.as] (MSC service thread 1-7) WFLYSRV0050: JBoss EAP CD 7.2.0.CD14 (WildFly Core 6.0.0.Final-redhat-20180815) stopped in 10153ms
[JBossINF] [0m[0m16:28:20,518 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel ejb
2018/08/21 16:30:10:437 EDT [DEBUG][RMI TCP Connection(61)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
h3. [perf19|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[0m16:15:20,382 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000080: Disconnecting JGroups channel ejb
...
2018/08/21 16:17:20:263 EDT [DEBUG][RMI TCP Connection(44)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
[JBossINF] [0m[0m16:30:23,976 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel ejb
2018/08/21 16:32:13:927 EDT [DEBUG][RMI TCP Connection(53)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
h3. [perf20|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[0m16:19:34,766 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000080: Disconnecting JGroups channel ejb
...
2018/08/21 16:21:34:692 EDT [DEBUG][RMI TCP Connection(31)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
[JBossINF] [0m[0m16:35:37,668 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel ejb
2018/08/21 16:37:27:615 EDT [DEBUG][RMI TCP Connection(42)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
h3. [perf21|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[0m16:23:49,076 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000080: Disconnecting JGroups channel ejb
...
2018/08/21 16:25:49:018 EDT [DEBUG][RMI TCP Connection(22)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
h2. wildfly:master (22 Aug 2018)
h3. [perf18|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
OK
h3. [perf19|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[33m11:42:26,122 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p7-t4) ISPN000071: Caught exception when handling command LockControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, keys=[SessionCreationMetaDataKey(FQgZzMUpaFIWVPc7RKrrbuacvIyOjApj1EWlKKqc)], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTx:perf18:37}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:perf18:37, already completed
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:388)
[JBossINF] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:381)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:361)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:139)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.invokeAsync(LockControlCommand.java:122)
[JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF]
[JBossINF] [0m[33m11:42:35,287 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf20:336 timed out. Rolling back after 74863 ms
[JBossINF] [0m[33m11:42:35,287 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf20:334 timed out. Rolling back after 76777 ms
[JBossINF] [0m[33m11:42:35,287 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf21:331 timed out. Rolling back after 74864 ms
[JBossINF] [0m[33m11:42:35,287 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf18:31 timed out. Rolling back after 74787 ms
2018/08/22 11:43:17:160 EDT [DEBUG][RMI TCP Connection(44)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
meanwhile on perf18 we observe:
{noformat}
[JBossINF] [0m[31m11:42:26,135 ERROR [io.undertow.request] (default task-2) UT005071: Undertow request failed HttpServerExchange{ GET /clusterbench/session request {Cookie=[$Version=0; JSESSIONID=FQgZzMUpaFIWVPc7RKrrbuacvIyOjApj1EWlKKqc.perf19; $Path=/clusterbench], Content-Length=[0], User-Agent=[Jakarta Commons-HttpClient/3.1], Host=[perf17:8080]} response {Content-Type=[text/html;charset=UTF-8], Content-Length=[80], Date=[Wed, 22 Aug 2018 15:42:06 GMT]}}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 56 from perf19
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 56 from perf19
[JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] ... 1 more
[JBossINF] Suppressed: org.infinispan.util.logging.TraceException
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41)
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
[JBossINF] ... 1 more
{noformat}
h3. [perf20|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[33m11:46:40,348 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p5-t11) ISPN000071: Caught exception when handling command LockControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, keys=[SessionCreationMetaDataKey(jKed4QLgTwwLw_q5p34roD_P-Bygv8umfynvDuHB)], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTx:perf19:53}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:perf19:53, already completed
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:388)
[JBossINF] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:381)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:361)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:139)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.invokeAsync(LockControlCommand.java:122)
[JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF]
[JBossINF] [0m[33m11:46:56,082 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf21:952 timed out. Rolling back after 83324 ms
[JBossINF] [0m[33m11:46:56,082 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf21:954 timed out. Rolling back after 81452 ms
[JBossINF] [0m[33m11:46:56,082 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf21:950 timed out. Rolling back after 83328 ms
[JBossINF] [0m[33m11:46:56,083 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf19:46 timed out. Rolling back after 81377 ms
[JBossINF] [0m[33m11:46:56,083 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf18:42 timed out. Rolling back after 83337 ms
2018/08/22 11:47:31:216 EDT [DEBUG][RMI TCP Connection(31)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
meanwhile on perf19:
{noformat}
[JBossINF] [0m[31m11:46:40,357 ERROR [io.undertow.request] (default task-2) UT005071: Undertow request failed HttpServerExchange{ GET /clusterbench/session request {Cookie=[$Version=0; JSESSIONID=jKed4QLgTwwLw_q5p34roD_P-Bygv8umfynvDuHB.perf20; $Path=/clusterbench], Content-Length=[0], User-Agent=[Jakarta Commons-HttpClient/3.1], Host=[perf17:8080]} response {Content-Type=[text/html;charset=UTF-8], Content-Length=[80], Date=[Wed, 22 Aug 2018 15:46:20 GMT]}}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 57 from perf20
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 57 from perf20
[JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] ... 1 more
[JBossINF] Suppressed: org.infinispan.util.logging.TraceException
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41)
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
[JBossINF] ... 1 more
{noformat}
h3. [perf21|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[33m11:50:54,509 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p6-t20) ISPN000071: Caught exception when handling command LockControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, keys=[SessionCreationMetaDataKey(VKJbgfNUhPq0zSCYr-QmL_TNDcqv85k1Y20kOSqf)], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTx:perf19:139}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:perf19:139, already completed
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:388)
[JBossINF] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:381)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:361)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:139)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.invokeAsync(LockControlCommand.java:122)
[JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF]
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf20:47 timed out. Rolling back after 72674 ms
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf20:46 timed out. Rolling back after 72676 ms
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf20:49 timed out. Rolling back after 72677 ms
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf20:48 timed out. Rolling back after 72675 ms
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf19:81 timed out. Rolling back after 70864 ms
2018/08/22 11:51:45:682 EDT [DEBUG][RMI TCP Connection(24)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
meanwhile on perf19:
{noformat}
[JBossINF] [0m[31m11:50:54,512 ERROR [io.undertow.request] (default task-3) UT005071: Undertow request failed HttpServerExchange{ GET /clusterbench/session request {Cookie=[$Version=0; JSESSIONID=VKJbgfNUhPq0zSCYr-QmL_TNDcqv85k1Y20kOSqf.perf21; $Path=/clusterbench], Content-Length=[0], User-Agent=[Jakarta Commons-HttpClient/3.1], Host=[perf17:8080]} response {Content-Type=[text/html;charset=UTF-8], Content-Length=[80], Date=[Wed, 22 Aug 2018 15:50:34 GMT]}}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 135 from perf21
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 135 from perf21
[JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] ... 1 more
[JBossINF] Suppressed: org.infinispan.util.logging.TraceException
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41)
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
[JBossINF] ... 1 more
{noformat}
> Server in cluster hangs during start after previous kill
> --------------------------------------------------------
>
> Key: WFLY-10736
> URL: https://issues.jboss.org/browse/WFLY-10736
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Reporter: Miroslav Novak
> Assignee: Paul Ferraro
> Priority: Blocker
> Labels: blocker-WF14
> Attachments: Lodh2TestCase.testRemoteJcaInboundOnly-traces.zip, Lodh2TestCase.testRemoteJcaInboundOnly.zip, clusterKilTest.zip, logs-with-workaround.zip, node-1-thread-dump-before-kill-shutdown-sequence.txt, server-with-mdb.log, standalone-full-ha-1.xml, standalone-full-ha-2.xml
>
>
> There is regression in JGroups or Infinispan in one of our tests for fault tolerance of JMS bridges. However work on JMS bridge appears to be unrelated. Issue was hit in WF weekly run.
> Test Scenario:
> * There are two servers. InQueue is deployed on Node 1,
> * OutQueue is deployed on Node 2. Both servers are started.
> * Large byte messages are sent to InQueue deployed on Node 1. Bridge between servers/queues transfers messages from node 1 to node 2.
> * Node 1 is killed and started again.
> * All messages are received from OutQueue deployed on Node 2.
> Result:
> Node 1 does not start after kill and hangs. There is following exception logged in node 2:
> {code}
> :26:17,894 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100000: Node node-1 joined the cluster
> 09:26:18,520 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,523 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,868 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=17, phase=READ_OLD_WRITE_ALL, rebalanceId=6, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[node-2: 122, node-1: 134]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[node-2: 84, node-1: 90, node-1: 82]}, unionCH=null, actualMembers=[node-2, node-1, node-1], persistentUUIDs=[12443bfb-e88a-46f3-919e-9213bf38ce19, 2873237f-d881-463f-8a5a-940bf1d764e5, a05ea8af-a83b-42a9-b937-dc2da1cae6d1]}
> 09:26:18,869 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-2]ISPN100002: Started rebalance with topology id 17
> 09:26:18,870 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t5) [Context=default][Scope=node-2]ISPN100003: Node node-2 finished rebalance phase with topology id 17
> 09:26:18,981 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-1]ISPN100003: Node node-1 finished rebalance phase with topology id 17
> 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p15-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> ... 1 more
> [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1]
> 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p16-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> ... 1 more
> [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1]
> {code}
> There is default JGroups udp stack configured which is used by Infinispan. Both of the servers (jgroups udp) are bound to 127.0.0.1. Node 2 has port offset 1000.
> Attaching thread dump from node 1 when it hangs during start.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0