[JBoss JIRA] (WFLY-10906) Tests regressions after WFLY-10480
by Jim Ma (JIRA)
[ https://issues.jboss.org/browse/WFLY-10906?page=com.atlassian.jira.plugin... ]
Jim Ma commented on WFLY-10906:
-------------------------------
There is small issue in elytron team provided code which converts subject to securityIdentity : identity.withRoleMapper(roleCategory, (rolesToMap) -> Roles.fromSet(roles)) doesn't add roleMapper to this "identity". To fix this this roleMapper loss , simply returns the identity.withRoleMapper()'s result when roles is not empty: https://github.com/jimma/wildfly/commit/6014fff1d28284d56d768514737885de4.... PR is sent : https://github.com/wildfly/wildfly/pull/11565
> Tests regressions after WFLY-10480
> ----------------------------------
>
> Key: WFLY-10906
> URL: https://issues.jboss.org/browse/WFLY-10906
> Project: WildFly
> Issue Type: Bug
> Components: Security, Web Services
> Affects Versions: 14.0.0.Beta2
> Reporter: Jan Blizňák
> Assignee: Jim Ma
> Priority: Blocker
>
> There are tests regressions after WFLY-10480 was merged which modified org.jboss.as.webservices.security.ElytronSecurityDomainContextImpl.
> Namely these tests in JBWS testsuite fail when running with elytron profile:
> UsernameAuthorizationDigestEjbTestCase.test
> UsernameAuthorizationDigestEjbTestCase.testConcurrent
> UsernameAuthorizationEJBTestCase.test
> All fail with the same message and stacktrace:
> {code:java}
> WFLYEJB0364: Invocation on method: public java.lang.String org.jboss.test.ws.jaxws.samples.wsse.policy.jaas.EJBDigestServiceImpl.sayHello() of bean: EjbEndpoint is not allowed
> {code}
> {code:java}
> 13:13:09,199 ERROR [org.jboss.as.ejb3.invocation] (default task-3) WFLYEJB0034: EJB Invocation failed on component EjbEndpoint for method public java.lang.String org.jboss.test.ws.jaxws.samples.wsse.policy.jaas.EJBDigestServiceImpl.sayHello(): javax.ejb.EJBAccessException: WFLYEJB0364: Invocation on method: public java.lang.String org.jboss.test.ws.jaxws.samples.wsse.policy.jaas.EJBDigestServiceImpl.sayHello() of bean: EjbEndpoint is not allowed
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.security.RolesAllowedInterceptor.processInvocation(RolesAllowedInterceptor.java:67)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.security.SecurityDomainInterceptor.processInvocation(SecurityDomainInterceptor.java:44)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee@14.0.0.CR1-SNAPSHOT//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
> at org.jboss.as.ee@14.0.0.CR1-SNAPSHOT//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.invokeInternal(AbstractInvocationHandler.java:161)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.lambda$invoke$0(AbstractInvocationHandler.java:116)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:382)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.Scoped.runAsFunctionEx(Scoped.java:161)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.Scoped.runAs(Scoped.java:73)
> at org.jboss.as.webservices//org.jboss.as.webservices.security.ElytronSecurityDomainContextImpl.runAs(ElytronSecurityDomainContextImpl.java:88)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:115)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.JBossWSInvoker.performInvocation(JBossWSInvoker.java:170)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
> at org.apache.cxf.impl//org.apache.cxf.jaxws.AbstractJAXWSMethodInvoker.invoke(AbstractJAXWSMethodInvoker.java:232)
> at org.apache.cxf.impl//org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:85)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.JBossWSInvoker.invoke(JBossWSInvoker.java:146)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor$2.run(ServiceInvokerInterceptor.java:126)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:131)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
> at org.apache.cxf.impl//org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.RequestHandlerImpl.handleHttpRequest(RequestHandlerImpl.java:110)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.transport.ServletHelper.callRequestHandler(ServletHelper.java:134)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.CXFServletExt.invoke(CXFServletExt.java:88)
> at org.apache.cxf.impl//org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:301)
> at org.apache.cxf.impl//org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:220)
> at javax.servlet.api@1.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:706)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.CXFServletExt.service(CXFServletExt.java:136)
> at org.jboss.ws.spi@3.2.2.Final//org.jboss.wsf.spi.deployment.WSFServlet.service(WSFServlet.java:140)
> at javax.servlet.api@1.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> at io.undertow.core@2.0.13.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> at io.undertow.core@2.0.13.Final//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> at java.base/java.lang.Thread.run(Thread.java:844)
> [0m[33m13:13:09,199 WARNING [org.apache.cxf.phase.PhaseInterceptorChain] (default task-3) Application {http://www.jboss.org/jbossws/ws-extensions/wssecuritypolicy}SecurityService#{http://www.jboss.org/jbossws/ws-extensions/wssecuritypolicy}sayHello has thrown exception, unwinding now: org.apache.cxf.interceptor.Fault: WFLYEJB0364: Invocation on method: public java.lang.String org.jboss.test.ws.jaxws.samples.wsse.policy.jaas.EJBDigestServiceImpl.sayHello() of bean: EjbEndpoint is not allowed
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162)
> at org.apache.cxf.impl//org.apache.cxf.jaxws.AbstractJAXWSMethodInvoker.createFault(AbstractJAXWSMethodInvoker.java:267)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:128)
> at org.apache.cxf.impl//org.apache.cxf.jaxws.AbstractJAXWSMethodInvoker.invoke(AbstractJAXWSMethodInvoker.java:232)
> at org.apache.cxf.impl//org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:85)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.JBossWSInvoker.invoke(JBossWSInvoker.java:146)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor$2.run(ServiceInvokerInterceptor.java:126)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:131)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
> at org.apache.cxf.impl//org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.RequestHandlerImpl.handleHttpRequest(RequestHandlerImpl.java:110)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.transport.ServletHelper.callRequestHandler(ServletHelper.java:134)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.CXFServletExt.invoke(CXFServletExt.java:88)
> at org.apache.cxf.impl//org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:301)
> at org.apache.cxf.impl//org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:220)
> at javax.servlet.api@1.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:706)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.CXFServletExt.service(CXFServletExt.java:136)
> at org.jboss.ws.spi@3.2.2.Final//org.jboss.wsf.spi.deployment.WSFServlet.service(WSFServlet.java:140)
> at javax.servlet.api@1.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> at io.undertow.core@2.0.13.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> at io.undertow.core@2.0.13.Final//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> at java.base/java.lang.Thread.run(Thread.java:844)
> Caused by: javax.ejb.EJBAccessException: WFLYEJB0364: Invocation on method: public java.lang.String org.jboss.test.ws.jaxws.samples.wsse.policy.jaas.EJBDigestServiceImpl.sayHello() of bean: EjbEndpoint is not allowed
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.security.RolesAllowedInterceptor.processInvocation(RolesAllowedInterceptor.java:67)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.security.SecurityDomainInterceptor.processInvocation(SecurityDomainInterceptor.java:44)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee@14.0.0.CR1-SNAPSHOT//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
> at org.jboss.as.ee@14.0.0.CR1-SNAPSHOT//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.invokeInternal(AbstractInvocationHandler.java:161)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.lambda$invoke$0(AbstractInvocationHandler.java:116)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:382)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.Scoped.runAsFunctionEx(Scoped.java:161)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.Scoped.runAs(Scoped.java:73)
> at org.jboss.as.webservices//org.jboss.as.webservices.security.ElytronSecurityDomainContextImpl.runAs(ElytronSecurityDomainContextImpl.java:88)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:115)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.JBossWSInvoker.performInvocation(JBossWSInvoker.java:170)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
> ... 62 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (DROOLS-2917) DMN resolveTypeRef returning null in BKM causes NPE during KieContainer compilation
by Matteo Mortari (JIRA)
[ https://issues.jboss.org/browse/DROOLS-2917?page=com.atlassian.jira.plugi... ]
Matteo Mortari updated DROOLS-2917:
-----------------------------------
Description:
The following stacktrace is reported
{code:java}
10:52:34.793 [main] ERROR org.kie.dmn.core.util.MsgUtil.logMessage:65 - Error compiling FEEL expression '"John"' for name 'First Name' on node 'an hardcoded Person': Exception raised: NullPointerException
10:52:34.796 [main] DEBUG org.kie.dmn.core.util.MsgUtil.logMessage:70 - Error compiling FEEL expression '"John"' for name 'First Name' on node 'an hardcoded Person': Exception raised: NullPointerException
java.lang.NullPointerException: null
at org.kie.dmn.core.compiler.DMNFEELHelper.compileFeelExpression(DMNFEELHelper.java:120) ~[classes/:na]
at org.kie.dmn.core.compiler.DMNEvaluatorCompiler.compileLiteralExpression(DMNEvaluatorCompiler.java:738) [classes/:na]
at org.kie.dmn.core.compiler.DMNEvaluatorCompiler.compileExpression(DMNEvaluatorCompiler.java:109) [classes/:na]
at org.kie.dmn.core.compiler.DMNEvaluatorCompiler.compileContext(DMNEvaluatorCompiler.java:243) [classes/:na]
at org.kie.dmn.core.compiler.DMNEvaluatorCompiler.compileExpression(DMNEvaluatorCompiler.java:115) [classes/:na]
at org.kie.dmn.core.compiler.DMNEvaluatorCompiler.compileFunctionDefinition(DMNEvaluatorCompiler.java:305) [classes/:na]
at org.kie.dmn.core.compiler.DMNEvaluatorCompiler.compileExpression(DMNEvaluatorCompiler.java:113) [classes/:na]
at org.kie.dmn.core.compiler.BusinessKnowledgeModelCompiler.compileEvaluator(BusinessKnowledgeModelCompiler.java:77) [classes/:na]
at org.kie.dmn.core.compiler.DMNCompilerImpl.processDrgElements(DMNCompilerImpl.java:308) [classes/:na]
at org.kie.dmn.core.compiler.DMNCompilerImpl.compile(DMNCompilerImpl.java:220) [classes/:na]
at org.kie.dmn.core.compiler.DMNCompilerImpl.compile(DMNCompilerImpl.java:152) [classes/:na]
at org.kie.dmn.core.compiler.DMNCompilerImpl.compile(DMNCompilerImpl.java:135) [classes/:na]
at org.kie.dmn.core.assembler.DMNAssemblerService.compileResourceToModel(DMNAssemblerService.java:140) [classes/:na]
at org.kie.dmn.core.assembler.DMNAssemblerService.internalAddResource(DMNAssemblerService.java:116) [classes/:na]
at org.kie.dmn.core.assembler.DMNAssemblerService.addResources(DMNAssemblerService.java:81) [classes/:na]
at org.drools.compiler.builder.impl.KnowledgeBuilderImpl.addPackageForExternalType(KnowledgeBuilderImpl.java:826) [drools-compiler-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
at org.drools.compiler.builder.impl.CompositeKnowledgeBuilderImpl.buildOthers(CompositeKnowledgeBuilderImpl.java:157) [drools-compiler-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
at org.drools.compiler.builder.impl.CompositeKnowledgeBuilderImpl.build(CompositeKnowledgeBuilderImpl.java:105) [drools-compiler-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
at org.drools.compiler.kie.builder.impl.AbstractKieProject.buildKnowledgePackages(AbstractKieProject.java:251) [drools-compiler-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
at org.drools.compiler.kie.builder.impl.AbstractKieProject.verify(AbstractKieProject.java:74) [drools-compiler-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
at org.drools.compiler.kie.builder.impl.KieBuilderImpl.buildKieProject(KieBuilderImpl.java:267) [drools-compiler-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
at org.drools.compiler.kie.builder.impl.KieBuilderImpl.buildAll(KieBuilderImpl.java:235) [drools-compiler-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
at org.drools.compiler.kie.builder.impl.KieBuilderImpl.buildAll(KieBuilderImpl.java:200) [drools-compiler-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
at org.kie.dmn.core.util.KieHelper.createJar(KieHelper.java:57) [classes/:na]
at org.kie.dmn.core.util.KieHelper.createAndDeployJar(KieHelper.java:43) [classes/:na]
at org.kie.dmn.core.util.KieHelper.getKieContainer(KieHelper.java:36) [classes/:na]
at org.kie.dmn.core.util.DMNRuntimeUtil.createRuntime(DMNRuntimeUtil.java:61) [test-classes/:na]
{code}
> DMN resolveTypeRef returning null in BKM causes NPE during KieContainer compilation
> -----------------------------------------------------------------------------------
>
> Key: DROOLS-2917
> URL: https://issues.jboss.org/browse/DROOLS-2917
> Project: Drools
> Issue Type: Bug
> Components: dmn engine
> Reporter: Matteo Mortari
> Assignee: Matteo Mortari
>
> The following stacktrace is reported
> {code:java}
> 10:52:34.793 [main] ERROR org.kie.dmn.core.util.MsgUtil.logMessage:65 - Error compiling FEEL expression '"John"' for name 'First Name' on node 'an hardcoded Person': Exception raised: NullPointerException
> 10:52:34.796 [main] DEBUG org.kie.dmn.core.util.MsgUtil.logMessage:70 - Error compiling FEEL expression '"John"' for name 'First Name' on node 'an hardcoded Person': Exception raised: NullPointerException
> java.lang.NullPointerException: null
> at org.kie.dmn.core.compiler.DMNFEELHelper.compileFeelExpression(DMNFEELHelper.java:120) ~[classes/:na]
> at org.kie.dmn.core.compiler.DMNEvaluatorCompiler.compileLiteralExpression(DMNEvaluatorCompiler.java:738) [classes/:na]
> at org.kie.dmn.core.compiler.DMNEvaluatorCompiler.compileExpression(DMNEvaluatorCompiler.java:109) [classes/:na]
> at org.kie.dmn.core.compiler.DMNEvaluatorCompiler.compileContext(DMNEvaluatorCompiler.java:243) [classes/:na]
> at org.kie.dmn.core.compiler.DMNEvaluatorCompiler.compileExpression(DMNEvaluatorCompiler.java:115) [classes/:na]
> at org.kie.dmn.core.compiler.DMNEvaluatorCompiler.compileFunctionDefinition(DMNEvaluatorCompiler.java:305) [classes/:na]
> at org.kie.dmn.core.compiler.DMNEvaluatorCompiler.compileExpression(DMNEvaluatorCompiler.java:113) [classes/:na]
> at org.kie.dmn.core.compiler.BusinessKnowledgeModelCompiler.compileEvaluator(BusinessKnowledgeModelCompiler.java:77) [classes/:na]
> at org.kie.dmn.core.compiler.DMNCompilerImpl.processDrgElements(DMNCompilerImpl.java:308) [classes/:na]
> at org.kie.dmn.core.compiler.DMNCompilerImpl.compile(DMNCompilerImpl.java:220) [classes/:na]
> at org.kie.dmn.core.compiler.DMNCompilerImpl.compile(DMNCompilerImpl.java:152) [classes/:na]
> at org.kie.dmn.core.compiler.DMNCompilerImpl.compile(DMNCompilerImpl.java:135) [classes/:na]
> at org.kie.dmn.core.assembler.DMNAssemblerService.compileResourceToModel(DMNAssemblerService.java:140) [classes/:na]
> at org.kie.dmn.core.assembler.DMNAssemblerService.internalAddResource(DMNAssemblerService.java:116) [classes/:na]
> at org.kie.dmn.core.assembler.DMNAssemblerService.addResources(DMNAssemblerService.java:81) [classes/:na]
> at org.drools.compiler.builder.impl.KnowledgeBuilderImpl.addPackageForExternalType(KnowledgeBuilderImpl.java:826) [drools-compiler-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
> at org.drools.compiler.builder.impl.CompositeKnowledgeBuilderImpl.buildOthers(CompositeKnowledgeBuilderImpl.java:157) [drools-compiler-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
> at org.drools.compiler.builder.impl.CompositeKnowledgeBuilderImpl.build(CompositeKnowledgeBuilderImpl.java:105) [drools-compiler-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
> at org.drools.compiler.kie.builder.impl.AbstractKieProject.buildKnowledgePackages(AbstractKieProject.java:251) [drools-compiler-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
> at org.drools.compiler.kie.builder.impl.AbstractKieProject.verify(AbstractKieProject.java:74) [drools-compiler-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
> at org.drools.compiler.kie.builder.impl.KieBuilderImpl.buildKieProject(KieBuilderImpl.java:267) [drools-compiler-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
> at org.drools.compiler.kie.builder.impl.KieBuilderImpl.buildAll(KieBuilderImpl.java:235) [drools-compiler-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
> at org.drools.compiler.kie.builder.impl.KieBuilderImpl.buildAll(KieBuilderImpl.java:200) [drools-compiler-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
> at org.kie.dmn.core.util.KieHelper.createJar(KieHelper.java:57) [classes/:na]
> at org.kie.dmn.core.util.KieHelper.createAndDeployJar(KieHelper.java:43) [classes/:na]
> at org.kie.dmn.core.util.KieHelper.getKieContainer(KieHelper.java:36) [classes/:na]
> at org.kie.dmn.core.util.DMNRuntimeUtil.createRuntime(DMNRuntimeUtil.java:61) [test-classes/:na]
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (DROOLS-2917) DMN resolveTypeRef returning null in BKM causes NPE during KieContainer compilation
by Matteo Mortari (JIRA)
Matteo Mortari created DROOLS-2917:
--------------------------------------
Summary: DMN resolveTypeRef returning null in BKM causes NPE during KieContainer compilation
Key: DROOLS-2917
URL: https://issues.jboss.org/browse/DROOLS-2917
Project: Drools
Issue Type: Bug
Components: dmn engine
Reporter: Matteo Mortari
Assignee: Matteo Mortari
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (WFLY-10736) Server in cluster hangs during start after previous kill
by tommaso borgato (JIRA)
[ https://issues.jboss.org/browse/WFLY-10736?page=com.atlassian.jira.plugin... ]
tommaso borgato edited comment on WFLY-10736 at 8/23/18 4:50 AM:
-----------------------------------------------------------------
[~pferraro] With both distributions the server often fails to shutdown within 60 seconds and the jvm is killed: "Server shutdown timed out. Terminating forcefully".
In both cases we observe the WARN log message: "ISPN000197: Error updating cluster member list".
In the test performed on wildfly:master build 22 Aug 2018, we always observe the WARN log message "ISPN000071: Caught exception when handling command LockControlCommand{...}" as the last log from the server before being forcefully terminated.
h2. eap-7.2.0.CD14-20180815
h3. [perf18|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[0m16:13:04,626 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel ejb
[JBossINF] [0m[0m16:13:04,660 INFO [org.jboss.as] (MSC service thread 1-7) WFLYSRV0050: JBoss EAP CD 7.2.0.CD14 (WildFly Core 6.0.0.Final-redhat-20180815) stopped in 10153ms
[JBossINF] [0m[0m16:28:20,518 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel ejb
2018/08/21 16:30:10:437 EDT [DEBUG][RMI TCP Connection(61)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
h3. [perf19|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[0m16:15:20,382 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000080: Disconnecting JGroups channel ejb
...
2018/08/21 16:17:20:263 EDT [DEBUG][RMI TCP Connection(44)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
[JBossINF] [0m[0m16:30:23,976 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel ejb
2018/08/21 16:32:13:927 EDT [DEBUG][RMI TCP Connection(53)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
h3. [perf20|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[0m16:19:34,766 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000080: Disconnecting JGroups channel ejb
...
2018/08/21 16:21:34:692 EDT [DEBUG][RMI TCP Connection(31)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
[JBossINF] [0m[0m16:35:37,668 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel ejb
2018/08/21 16:37:27:615 EDT [DEBUG][RMI TCP Connection(42)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
h3. [perf21|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[0m16:23:49,076 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000080: Disconnecting JGroups channel ejb
...
2018/08/21 16:25:49:018 EDT [DEBUG][RMI TCP Connection(22)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
h2. wildfly:master (22 Aug 2018)
h3. [perf18|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
OK
h3. [perf19|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[33m11:42:26,122 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p7-t4) ISPN000071: Caught exception when handling command LockControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, keys=[SessionCreationMetaDataKey(FQgZzMUpaFIWVPc7RKrrbuacvIyOjApj1EWlKKqc)], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTx:perf18:37}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:perf18:37, already completed
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:388)
[JBossINF] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:381)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:361)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:139)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.invokeAsync(LockControlCommand.java:122)
[JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF]
[JBossINF] [0m[33m11:42:35,287 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf20:336 timed out. Rolling back after 74863 ms
[JBossINF] [0m[33m11:42:35,287 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf20:334 timed out. Rolling back after 76777 ms
[JBossINF] [0m[33m11:42:35,287 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf21:331 timed out. Rolling back after 74864 ms
[JBossINF] [0m[33m11:42:35,287 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf18:31 timed out. Rolling back after 74787 ms
2018/08/22 11:43:17:160 EDT [DEBUG][RMI TCP Connection(44)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
meanwhile on perf18 we observe:
{noformat}
[JBossINF] [0m[31m11:42:26,135 ERROR [io.undertow.request] (default task-2) UT005071: Undertow request failed HttpServerExchange{ GET /clusterbench/session request {Cookie=[$Version=0; JSESSIONID=FQgZzMUpaFIWVPc7RKrrbuacvIyOjApj1EWlKKqc.perf19; $Path=/clusterbench], Content-Length=[0], User-Agent=[Jakarta Commons-HttpClient/3.1], Host=[perf17:8080]} response {Content-Type=[text/html;charset=UTF-8], Content-Length=[80], Date=[Wed, 22 Aug 2018 15:42:06 GMT]}}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 56 from perf19
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 56 from perf19
[JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] ... 1 more
[JBossINF] Suppressed: org.infinispan.util.logging.TraceException
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41)
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
[JBossINF] ... 1 more
{noformat}
h3. [perf20|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[33m11:46:40,348 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p5-t11) ISPN000071: Caught exception when handling command LockControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, keys=[SessionCreationMetaDataKey(jKed4QLgTwwLw_q5p34roD_P-Bygv8umfynvDuHB)], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTx:perf19:53}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:perf19:53, already completed
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:388)
[JBossINF] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:381)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:361)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:139)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.invokeAsync(LockControlCommand.java:122)
[JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF]
[JBossINF] [0m[33m11:46:56,082 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf21:952 timed out. Rolling back after 83324 ms
[JBossINF] [0m[33m11:46:56,082 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf21:954 timed out. Rolling back after 81452 ms
[JBossINF] [0m[33m11:46:56,082 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf21:950 timed out. Rolling back after 83328 ms
[JBossINF] [0m[33m11:46:56,083 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf19:46 timed out. Rolling back after 81377 ms
[JBossINF] [0m[33m11:46:56,083 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf18:42 timed out. Rolling back after 83337 ms
2018/08/22 11:47:31:216 EDT [DEBUG][RMI TCP Connection(31)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
meanwhile on perf19:
{noformat}
[JBossINF] [0m[31m11:46:40,357 ERROR [io.undertow.request] (default task-2) UT005071: Undertow request failed HttpServerExchange{ GET /clusterbench/session request {Cookie=[$Version=0; JSESSIONID=jKed4QLgTwwLw_q5p34roD_P-Bygv8umfynvDuHB.perf20; $Path=/clusterbench], Content-Length=[0], User-Agent=[Jakarta Commons-HttpClient/3.1], Host=[perf17:8080]} response {Content-Type=[text/html;charset=UTF-8], Content-Length=[80], Date=[Wed, 22 Aug 2018 15:46:20 GMT]}}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 57 from perf20
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 57 from perf20
[JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] ... 1 more
[JBossINF] Suppressed: org.infinispan.util.logging.TraceException
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41)
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
[JBossINF] ... 1 more
{noformat}
h3. [perf21|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[33m11:50:54,509 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p6-t20) ISPN000071: Caught exception when handling command LockControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, keys=[SessionCreationMetaDataKey(VKJbgfNUhPq0zSCYr-QmL_TNDcqv85k1Y20kOSqf)], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTx:perf19:139}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:perf19:139, already completed
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:388)
[JBossINF] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:381)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:361)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:139)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.invokeAsync(LockControlCommand.java:122)
[JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF]
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf20:47 timed out. Rolling back after 72674 ms
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf20:46 timed out. Rolling back after 72676 ms
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf20:49 timed out. Rolling back after 72677 ms
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf20:48 timed out. Rolling back after 72675 ms
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf19:81 timed out. Rolling back after 70864 ms
2018/08/22 11:51:45:682 EDT [DEBUG][RMI TCP Connection(24)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
meanwhile on perf19:
{noformat}
[JBossINF] [0m[31m11:50:54,512 ERROR [io.undertow.request] (default task-3) UT005071: Undertow request failed HttpServerExchange{ GET /clusterbench/session request {Cookie=[$Version=0; JSESSIONID=VKJbgfNUhPq0zSCYr-QmL_TNDcqv85k1Y20kOSqf.perf21; $Path=/clusterbench], Content-Length=[0], User-Agent=[Jakarta Commons-HttpClient/3.1], Host=[perf17:8080]} response {Content-Type=[text/html;charset=UTF-8], Content-Length=[80], Date=[Wed, 22 Aug 2018 15:50:34 GMT]}}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 135 from perf21
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 135 from perf21
[JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] ... 1 more
[JBossINF] Suppressed: org.infinispan.util.logging.TraceException
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41)
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
[JBossINF] ... 1 more
{noformat}
was (Author: tommaso-borgato):
With both distributions the server often fails to shutdown within 60 seconds and the jvm is killed: "Server shutdown timed out. Terminating forcefully".
In both cases we observe the WARN log message: "ISPN000197: Error updating cluster member list".
In the test performed on wildfly:master build 22 Aug 2018, we always observe the WARN log message "ISPN000071: Caught exception when handling command LockControlCommand{...}" as the last log from the server before being forcefully terminated.
h2. eap-7.2.0.CD14-20180815
h3. [perf18|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[0m16:13:04,626 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel ejb
[JBossINF] [0m[0m16:13:04,660 INFO [org.jboss.as] (MSC service thread 1-7) WFLYSRV0050: JBoss EAP CD 7.2.0.CD14 (WildFly Core 6.0.0.Final-redhat-20180815) stopped in 10153ms
[JBossINF] [0m[0m16:28:20,518 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel ejb
2018/08/21 16:30:10:437 EDT [DEBUG][RMI TCP Connection(61)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
h3. [perf19|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[0m16:15:20,382 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000080: Disconnecting JGroups channel ejb
...
2018/08/21 16:17:20:263 EDT [DEBUG][RMI TCP Connection(44)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
[JBossINF] [0m[0m16:30:23,976 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel ejb
2018/08/21 16:32:13:927 EDT [DEBUG][RMI TCP Connection(53)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
h3. [perf20|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[0m16:19:34,766 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000080: Disconnecting JGroups channel ejb
...
2018/08/21 16:21:34:692 EDT [DEBUG][RMI TCP Connection(31)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
[JBossINF] [0m[0m16:35:37,668 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel ejb
2018/08/21 16:37:27:615 EDT [DEBUG][RMI TCP Connection(42)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
h3. [perf21|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[0m16:23:49,076 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000080: Disconnecting JGroups channel ejb
...
2018/08/21 16:25:49:018 EDT [DEBUG][RMI TCP Connection(22)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
h2. wildfly:master (22 Aug 2018)
h3. [perf18|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
OK
h3. [perf19|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[33m11:42:26,122 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p7-t4) ISPN000071: Caught exception when handling command LockControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, keys=[SessionCreationMetaDataKey(FQgZzMUpaFIWVPc7RKrrbuacvIyOjApj1EWlKKqc)], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTx:perf18:37}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:perf18:37, already completed
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:388)
[JBossINF] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:381)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:361)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:139)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.invokeAsync(LockControlCommand.java:122)
[JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF]
[JBossINF] [0m[33m11:42:35,287 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf20:336 timed out. Rolling back after 74863 ms
[JBossINF] [0m[33m11:42:35,287 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf20:334 timed out. Rolling back after 76777 ms
[JBossINF] [0m[33m11:42:35,287 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf21:331 timed out. Rolling back after 74864 ms
[JBossINF] [0m[33m11:42:35,287 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf18:31 timed out. Rolling back after 74787 ms
2018/08/22 11:43:17:160 EDT [DEBUG][RMI TCP Connection(44)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
meanwhile on perf18 we observe:
{noformat}
[JBossINF] [0m[31m11:42:26,135 ERROR [io.undertow.request] (default task-2) UT005071: Undertow request failed HttpServerExchange{ GET /clusterbench/session request {Cookie=[$Version=0; JSESSIONID=FQgZzMUpaFIWVPc7RKrrbuacvIyOjApj1EWlKKqc.perf19; $Path=/clusterbench], Content-Length=[0], User-Agent=[Jakarta Commons-HttpClient/3.1], Host=[perf17:8080]} response {Content-Type=[text/html;charset=UTF-8], Content-Length=[80], Date=[Wed, 22 Aug 2018 15:42:06 GMT]}}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 56 from perf19
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 56 from perf19
[JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] ... 1 more
[JBossINF] Suppressed: org.infinispan.util.logging.TraceException
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41)
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
[JBossINF] ... 1 more
{noformat}
h3. [perf20|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[33m11:46:40,348 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p5-t11) ISPN000071: Caught exception when handling command LockControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, keys=[SessionCreationMetaDataKey(jKed4QLgTwwLw_q5p34roD_P-Bygv8umfynvDuHB)], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTx:perf19:53}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:perf19:53, already completed
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:388)
[JBossINF] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:381)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:361)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:139)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.invokeAsync(LockControlCommand.java:122)
[JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF]
[JBossINF] [0m[33m11:46:56,082 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf21:952 timed out. Rolling back after 83324 ms
[JBossINF] [0m[33m11:46:56,082 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf21:954 timed out. Rolling back after 81452 ms
[JBossINF] [0m[33m11:46:56,082 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf21:950 timed out. Rolling back after 83328 ms
[JBossINF] [0m[33m11:46:56,083 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf19:46 timed out. Rolling back after 81377 ms
[JBossINF] [0m[33m11:46:56,083 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf18:42 timed out. Rolling back after 83337 ms
2018/08/22 11:47:31:216 EDT [DEBUG][RMI TCP Connection(31)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
meanwhile on perf19:
{noformat}
[JBossINF] [0m[31m11:46:40,357 ERROR [io.undertow.request] (default task-2) UT005071: Undertow request failed HttpServerExchange{ GET /clusterbench/session request {Cookie=[$Version=0; JSESSIONID=jKed4QLgTwwLw_q5p34roD_P-Bygv8umfynvDuHB.perf20; $Path=/clusterbench], Content-Length=[0], User-Agent=[Jakarta Commons-HttpClient/3.1], Host=[perf17:8080]} response {Content-Type=[text/html;charset=UTF-8], Content-Length=[80], Date=[Wed, 22 Aug 2018 15:46:20 GMT]}}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 57 from perf20
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 57 from perf20
[JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] ... 1 more
[JBossINF] Suppressed: org.infinispan.util.logging.TraceException
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41)
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
[JBossINF] ... 1 more
{noformat}
h3. [perf21|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[33m11:50:54,509 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p6-t20) ISPN000071: Caught exception when handling command LockControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, keys=[SessionCreationMetaDataKey(VKJbgfNUhPq0zSCYr-QmL_TNDcqv85k1Y20kOSqf)], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTx:perf19:139}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:perf19:139, already completed
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:388)
[JBossINF] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:381)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:361)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:139)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.invokeAsync(LockControlCommand.java:122)
[JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF]
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf20:47 timed out. Rolling back after 72674 ms
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf20:46 timed out. Rolling back after 72676 ms
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf20:49 timed out. Rolling back after 72677 ms
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf20:48 timed out. Rolling back after 72675 ms
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf19:81 timed out. Rolling back after 70864 ms
2018/08/22 11:51:45:682 EDT [DEBUG][RMI TCP Connection(24)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
meanwhile on perf19:
{noformat}
[JBossINF] [0m[31m11:50:54,512 ERROR [io.undertow.request] (default task-3) UT005071: Undertow request failed HttpServerExchange{ GET /clusterbench/session request {Cookie=[$Version=0; JSESSIONID=VKJbgfNUhPq0zSCYr-QmL_TNDcqv85k1Y20kOSqf.perf21; $Path=/clusterbench], Content-Length=[0], User-Agent=[Jakarta Commons-HttpClient/3.1], Host=[perf17:8080]} response {Content-Type=[text/html;charset=UTF-8], Content-Length=[80], Date=[Wed, 22 Aug 2018 15:50:34 GMT]}}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 135 from perf21
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 135 from perf21
[JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] ... 1 more
[JBossINF] Suppressed: org.infinispan.util.logging.TraceException
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41)
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
[JBossINF] ... 1 more
{noformat}
> Server in cluster hangs during start after previous kill
> --------------------------------------------------------
>
> Key: WFLY-10736
> URL: https://issues.jboss.org/browse/WFLY-10736
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Reporter: Miroslav Novak
> Assignee: Paul Ferraro
> Priority: Blocker
> Labels: blocker-WF14
> Attachments: Lodh2TestCase.testRemoteJcaInboundOnly-traces.zip, Lodh2TestCase.testRemoteJcaInboundOnly.zip, clusterKilTest.zip, logs-with-workaround.zip, node-1-thread-dump-before-kill-shutdown-sequence.txt, server-with-mdb.log, standalone-full-ha-1.xml, standalone-full-ha-2.xml
>
>
> There is regression in JGroups or Infinispan in one of our tests for fault tolerance of JMS bridges. However work on JMS bridge appears to be unrelated. Issue was hit in WF weekly run.
> Test Scenario:
> * There are two servers. InQueue is deployed on Node 1,
> * OutQueue is deployed on Node 2. Both servers are started.
> * Large byte messages are sent to InQueue deployed on Node 1. Bridge between servers/queues transfers messages from node 1 to node 2.
> * Node 1 is killed and started again.
> * All messages are received from OutQueue deployed on Node 2.
> Result:
> Node 1 does not start after kill and hangs. There is following exception logged in node 2:
> {code}
> :26:17,894 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100000: Node node-1 joined the cluster
> 09:26:18,520 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,523 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,868 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=17, phase=READ_OLD_WRITE_ALL, rebalanceId=6, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[node-2: 122, node-1: 134]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[node-2: 84, node-1: 90, node-1: 82]}, unionCH=null, actualMembers=[node-2, node-1, node-1], persistentUUIDs=[12443bfb-e88a-46f3-919e-9213bf38ce19, 2873237f-d881-463f-8a5a-940bf1d764e5, a05ea8af-a83b-42a9-b937-dc2da1cae6d1]}
> 09:26:18,869 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-2]ISPN100002: Started rebalance with topology id 17
> 09:26:18,870 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t5) [Context=default][Scope=node-2]ISPN100003: Node node-2 finished rebalance phase with topology id 17
> 09:26:18,981 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-1]ISPN100003: Node node-1 finished rebalance phase with topology id 17
> 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p15-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> ... 1 more
> [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1]
> 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p16-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> ... 1 more
> [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1]
> {code}
> There is default JGroups udp stack configured which is used by Infinispan. Both of the servers (jgroups udp) are bound to 127.0.0.1. Node 2 has port offset 1000.
> Attaching thread dump from node 1 when it hangs during start.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (WFLY-10736) Server in cluster hangs during start after previous kill
by tommaso borgato (JIRA)
[ https://issues.jboss.org/browse/WFLY-10736?page=com.atlassian.jira.plugin... ]
tommaso borgato commented on WFLY-10736:
----------------------------------------
With both distributions the server often fails to shutdown within 60 seconds and the jvm is killed: "Server shutdown timed out. Terminating forcefully".
In both cases we observe the WARN log message: "ISPN000197: Error updating cluster member list".
In the test performed on wildfly:master build 22 Aug 2018, we always observe the WARN log message "ISPN000071: Caught exception when handling command LockControlCommand{...}" as the last log from the server before being forcefully terminated.
h2. eap-7.2.0.CD14-20180815
h3. [perf18|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[0m16:13:04,626 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel ejb
[JBossINF] [0m[0m16:13:04,660 INFO [org.jboss.as] (MSC service thread 1-7) WFLYSRV0050: JBoss EAP CD 7.2.0.CD14 (WildFly Core 6.0.0.Final-redhat-20180815) stopped in 10153ms
[JBossINF] [0m[0m16:28:20,518 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel ejb
2018/08/21 16:30:10:437 EDT [DEBUG][RMI TCP Connection(61)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
h3. [perf19|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[0m16:15:20,382 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000080: Disconnecting JGroups channel ejb
...
2018/08/21 16:17:20:263 EDT [DEBUG][RMI TCP Connection(44)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
[JBossINF] [0m[0m16:30:23,976 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel ejb
2018/08/21 16:32:13:927 EDT [DEBUG][RMI TCP Connection(53)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
h3. [perf20|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[0m16:19:34,766 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000080: Disconnecting JGroups channel ejb
...
2018/08/21 16:21:34:692 EDT [DEBUG][RMI TCP Connection(31)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
[JBossINF] [0m[0m16:35:37,668 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel ejb
2018/08/21 16:37:27:615 EDT [DEBUG][RMI TCP Connection(42)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
h3. [perf21|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[0m16:23:49,076 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000080: Disconnecting JGroups channel ejb
...
2018/08/21 16:25:49:018 EDT [DEBUG][RMI TCP Connection(22)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
h2. wildfly:master (22 Aug 2018)
h3. [perf18|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
OK
h3. [perf19|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[33m11:42:26,122 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p7-t4) ISPN000071: Caught exception when handling command LockControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, keys=[SessionCreationMetaDataKey(FQgZzMUpaFIWVPc7RKrrbuacvIyOjApj1EWlKKqc)], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTx:perf18:37}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:perf18:37, already completed
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:388)
[JBossINF] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:381)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:361)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:139)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.invokeAsync(LockControlCommand.java:122)
[JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF]
[JBossINF] [0m[33m11:42:35,287 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf20:336 timed out. Rolling back after 74863 ms
[JBossINF] [0m[33m11:42:35,287 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf20:334 timed out. Rolling back after 76777 ms
[JBossINF] [0m[33m11:42:35,287 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf21:331 timed out. Rolling back after 74864 ms
[JBossINF] [0m[33m11:42:35,287 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf18:31 timed out. Rolling back after 74787 ms
2018/08/22 11:43:17:160 EDT [DEBUG][RMI TCP Connection(44)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
meanwhile on perf18 we observe:
{noformat}
[JBossINF] [0m[31m11:42:26,135 ERROR [io.undertow.request] (default task-2) UT005071: Undertow request failed HttpServerExchange{ GET /clusterbench/session request {Cookie=[$Version=0; JSESSIONID=FQgZzMUpaFIWVPc7RKrrbuacvIyOjApj1EWlKKqc.perf19; $Path=/clusterbench], Content-Length=[0], User-Agent=[Jakarta Commons-HttpClient/3.1], Host=[perf17:8080]} response {Content-Type=[text/html;charset=UTF-8], Content-Length=[80], Date=[Wed, 22 Aug 2018 15:42:06 GMT]}}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 56 from perf19
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 56 from perf19
[JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] ... 1 more
[JBossINF] Suppressed: org.infinispan.util.logging.TraceException
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41)
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
[JBossINF] ... 1 more
{noformat}
h3. [perf20|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[33m11:46:40,348 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p5-t11) ISPN000071: Caught exception when handling command LockControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, keys=[SessionCreationMetaDataKey(jKed4QLgTwwLw_q5p34roD_P-Bygv8umfynvDuHB)], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTx:perf19:53}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:perf19:53, already completed
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:388)
[JBossINF] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:381)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:361)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:139)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.invokeAsync(LockControlCommand.java:122)
[JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF]
[JBossINF] [0m[33m11:46:56,082 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf21:952 timed out. Rolling back after 83324 ms
[JBossINF] [0m[33m11:46:56,082 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf21:954 timed out. Rolling back after 81452 ms
[JBossINF] [0m[33m11:46:56,082 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf21:950 timed out. Rolling back after 83328 ms
[JBossINF] [0m[33m11:46:56,083 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf19:46 timed out. Rolling back after 81377 ms
[JBossINF] [0m[33m11:46:56,083 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p9-t1) ISPN000326: Remote transaction GlobalTx:perf18:42 timed out. Rolling back after 83337 ms
2018/08/22 11:47:31:216 EDT [DEBUG][RMI TCP Connection(31)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
meanwhile on perf19:
{noformat}
[JBossINF] [0m[31m11:46:40,357 ERROR [io.undertow.request] (default task-2) UT005071: Undertow request failed HttpServerExchange{ GET /clusterbench/session request {Cookie=[$Version=0; JSESSIONID=jKed4QLgTwwLw_q5p34roD_P-Bygv8umfynvDuHB.perf20; $Path=/clusterbench], Content-Length=[0], User-Agent=[Jakarta Commons-HttpClient/3.1], Host=[perf17:8080]} response {Content-Type=[text/html;charset=UTF-8], Content-Length=[80], Date=[Wed, 22 Aug 2018 15:46:20 GMT]}}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 57 from perf20
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 57 from perf20
[JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] ... 1 more
[JBossINF] Suppressed: org.infinispan.util.logging.TraceException
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41)
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
[JBossINF] ... 1 more
{noformat}
h3. [perf21|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_e...]
{noformat}
[JBossINF] [0m[33m11:50:54,509 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p6-t20) ISPN000071: Caught exception when handling command LockControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, keys=[SessionCreationMetaDataKey(VKJbgfNUhPq0zSCYr-QmL_TNDcqv85k1Y20kOSqf)], flags=[FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTx:perf19:139}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:perf19:139, already completed
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:388)
[JBossINF] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:381)
[JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:361)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.createContext(LockControlCommand.java:139)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.invokeAsync(LockControlCommand.java:122)
[JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF]
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf20:47 timed out. Rolling back after 72674 ms
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf20:46 timed out. Rolling back after 72676 ms
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf20:49 timed out. Rolling back after 72677 ms
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf20:48 timed out. Rolling back after 72675 ms
[JBossINF] [0m[33m11:50:59,714 WARN [org.infinispan.transaction.impl.TransactionTable] (timeout-thread--p10-t1) ISPN000326: Remote transaction GlobalTx:perf19:81 timed out. Rolling back after 70864 ms
2018/08/22 11:51:45:682 EDT [DEBUG][RMI TCP Connection(24)-10.16.90.52] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Server shutdown timed out. Terminating forcefully.
{noformat}
meanwhile on perf19:
{noformat}
[JBossINF] [0m[31m11:50:54,512 ERROR [io.undertow.request] (default task-3) UT005071: Undertow request failed HttpServerExchange{ GET /clusterbench/session request {Cookie=[$Version=0; JSESSIONID=VKJbgfNUhPq0zSCYr-QmL_TNDcqv85k1Y20kOSqf.perf21; $Path=/clusterbench], Content-Length=[0], User-Agent=[Jakarta Commons-HttpClient/3.1], Host=[perf17:8080]} response {Content-Type=[text/html;charset=UTF-8], Content-Length=[80], Date=[Wed, 22 Aug 2018 15:50:34 GMT]}}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 135 from perf21
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 135 from perf21
[JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
[JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] ... 1 more
[JBossINF] Suppressed: org.infinispan.util.logging.TraceException
[JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41)
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:479)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:529)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:659)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:78)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:68)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:59)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:37)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:208)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:211)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:937)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
[JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.doErrorDispatch(HttpServletResponseImpl.java:178)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:334)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
[JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
[JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
[JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
[JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
[JBossINF] ... 1 more
{noformat}
> Server in cluster hangs during start after previous kill
> --------------------------------------------------------
>
> Key: WFLY-10736
> URL: https://issues.jboss.org/browse/WFLY-10736
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Reporter: Miroslav Novak
> Assignee: Paul Ferraro
> Priority: Blocker
> Labels: blocker-WF14
> Attachments: Lodh2TestCase.testRemoteJcaInboundOnly-traces.zip, Lodh2TestCase.testRemoteJcaInboundOnly.zip, clusterKilTest.zip, logs-with-workaround.zip, node-1-thread-dump-before-kill-shutdown-sequence.txt, server-with-mdb.log, standalone-full-ha-1.xml, standalone-full-ha-2.xml
>
>
> There is regression in JGroups or Infinispan in one of our tests for fault tolerance of JMS bridges. However work on JMS bridge appears to be unrelated. Issue was hit in WF weekly run.
> Test Scenario:
> * There are two servers. InQueue is deployed on Node 1,
> * OutQueue is deployed on Node 2. Both servers are started.
> * Large byte messages are sent to InQueue deployed on Node 1. Bridge between servers/queues transfers messages from node 1 to node 2.
> * Node 1 is killed and started again.
> * All messages are received from OutQueue deployed on Node 2.
> Result:
> Node 1 does not start after kill and hangs. There is following exception logged in node 2:
> {code}
> :26:17,894 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100000: Node node-1 joined the cluster
> 09:26:18,520 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,523 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,868 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=17, phase=READ_OLD_WRITE_ALL, rebalanceId=6, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[node-2: 122, node-1: 134]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[node-2: 84, node-1: 90, node-1: 82]}, unionCH=null, actualMembers=[node-2, node-1, node-1], persistentUUIDs=[12443bfb-e88a-46f3-919e-9213bf38ce19, 2873237f-d881-463f-8a5a-940bf1d764e5, a05ea8af-a83b-42a9-b937-dc2da1cae6d1]}
> 09:26:18,869 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-2]ISPN100002: Started rebalance with topology id 17
> 09:26:18,870 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t5) [Context=default][Scope=node-2]ISPN100003: Node node-2 finished rebalance phase with topology id 17
> 09:26:18,981 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-1]ISPN100003: Node node-1 finished rebalance phase with topology id 17
> 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p15-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> ... 1 more
> [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1]
> 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p16-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> ... 1 more
> [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1]
> {code}
> There is default JGroups udp stack configured which is used by Infinispan. Both of the servers (jgroups udp) are bound to 127.0.0.1. Node 2 has port offset 1000.
> Attaching thread dump from node 1 when it hangs during start.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (WFCORE-4055) Activate jdk.unsupported module on modular JDK by default
by Richard Opalka (JIRA)
[ https://issues.jboss.org/browse/WFCORE-4055?page=com.atlassian.jira.plugi... ]
Richard Opalka resolved WFCORE-4055.
------------------------------------
Release Notes Text: jdk.unsupported module is active by default. There's no need to explicitly activate it.
Fix Version/s: (was: 6.0.0.CR4)
Resolution: Rejected
> Activate jdk.unsupported module on modular JDK by default
> ---------------------------------------------------------
>
> Key: WFCORE-4055
> URL: https://issues.jboss.org/browse/WFCORE-4055
> Project: WildFly Core
> Issue Type: Bug
> Components: Management, Security, Server, Test Suite
> Reporter: Richard Opalka
> Assignee: Richard Opalka
>
> WildFlySecurityManager uses sun.misc.Unsafe.
> Unsafe works only with jdk.unsupported module enabled.
> jdk.unsupported is not resolved by default so we need to activate it.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (WFWIP-119) [Artemis 2.x upgrade] Lost message in replicated HA scenario with MDB
by Erich Duda (JIRA)
[ https://issues.jboss.org/browse/WFWIP-119?page=com.atlassian.jira.plugin.... ]
Erich Duda commented on WFWIP-119:
----------------------------------
bq. Also: the test stopped the replica before live when I ran manually. Do you know in what circunstance that happens? I am playing with some possible options on clearing up tokens, but I need to understand this first.
Yes the test stops at first backup, then it stops live and then it stops server 4. At this point, servers 1, 2 and 4 are stopped. Then the servers are started in order 1, 4, 2.
> [Artemis 2.x upgrade] Lost message in replicated HA scenario with MDB
> ---------------------------------------------------------------------
>
> Key: WFWIP-119
> URL: https://issues.jboss.org/browse/WFWIP-119
> Project: WildFly WIP
> Issue Type: Bug
> Components: Artemis
> Reporter: Erich Duda
> Assignee: Martyn Taylor
> Priority: Blocker
>
> *Scenario*
> * Start live/backup pair in replicated dedicated topology and start one more server which is in Artemis cluster with live.
> * Send messages to queue InQueue.
> * Start another server with MDB connected to cluster with live server which resend messages from InQueue to OutQueue.
> * Stop backup server and then live server.
> * Stop server which was in cluster with live server.
> * Start live server and server in cluster.
> * Start backup server.
> The test fails because one message get lost. It wasn't sent to OutQueue.
> Logs from failed test including traces you can find in \[1\].
> \[1\] https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messag...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (WFLY-10906) Tests regressions after WFLY-10480
by Jim Ma (JIRA)
[ https://issues.jboss.org/browse/WFLY-10906?page=com.atlassian.jira.plugin... ]
Jim Ma reassigned WFLY-10906:
-----------------------------
Assignee: Jim Ma (was: Alessio Soldano)
> Tests regressions after WFLY-10480
> ----------------------------------
>
> Key: WFLY-10906
> URL: https://issues.jboss.org/browse/WFLY-10906
> Project: WildFly
> Issue Type: Bug
> Components: Security, Web Services
> Affects Versions: 14.0.0.Beta2
> Reporter: Jan Blizňák
> Assignee: Jim Ma
> Priority: Blocker
>
> There are tests regressions after WFLY-10480 was merged which modified org.jboss.as.webservices.security.ElytronSecurityDomainContextImpl.
> Namely these tests in JBWS testsuite fail when running with elytron profile:
> UsernameAuthorizationDigestEjbTestCase.test
> UsernameAuthorizationDigestEjbTestCase.testConcurrent
> UsernameAuthorizationEJBTestCase.test
> All fail with the same message and stacktrace:
> {code:java}
> WFLYEJB0364: Invocation on method: public java.lang.String org.jboss.test.ws.jaxws.samples.wsse.policy.jaas.EJBDigestServiceImpl.sayHello() of bean: EjbEndpoint is not allowed
> {code}
> {code:java}
> 13:13:09,199 ERROR [org.jboss.as.ejb3.invocation] (default task-3) WFLYEJB0034: EJB Invocation failed on component EjbEndpoint for method public java.lang.String org.jboss.test.ws.jaxws.samples.wsse.policy.jaas.EJBDigestServiceImpl.sayHello(): javax.ejb.EJBAccessException: WFLYEJB0364: Invocation on method: public java.lang.String org.jboss.test.ws.jaxws.samples.wsse.policy.jaas.EJBDigestServiceImpl.sayHello() of bean: EjbEndpoint is not allowed
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.security.RolesAllowedInterceptor.processInvocation(RolesAllowedInterceptor.java:67)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.security.SecurityDomainInterceptor.processInvocation(SecurityDomainInterceptor.java:44)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee@14.0.0.CR1-SNAPSHOT//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
> at org.jboss.as.ee@14.0.0.CR1-SNAPSHOT//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.invokeInternal(AbstractInvocationHandler.java:161)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.lambda$invoke$0(AbstractInvocationHandler.java:116)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:382)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.Scoped.runAsFunctionEx(Scoped.java:161)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.Scoped.runAs(Scoped.java:73)
> at org.jboss.as.webservices//org.jboss.as.webservices.security.ElytronSecurityDomainContextImpl.runAs(ElytronSecurityDomainContextImpl.java:88)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:115)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.JBossWSInvoker.performInvocation(JBossWSInvoker.java:170)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
> at org.apache.cxf.impl//org.apache.cxf.jaxws.AbstractJAXWSMethodInvoker.invoke(AbstractJAXWSMethodInvoker.java:232)
> at org.apache.cxf.impl//org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:85)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.JBossWSInvoker.invoke(JBossWSInvoker.java:146)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor$2.run(ServiceInvokerInterceptor.java:126)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:131)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
> at org.apache.cxf.impl//org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.RequestHandlerImpl.handleHttpRequest(RequestHandlerImpl.java:110)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.transport.ServletHelper.callRequestHandler(ServletHelper.java:134)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.CXFServletExt.invoke(CXFServletExt.java:88)
> at org.apache.cxf.impl//org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:301)
> at org.apache.cxf.impl//org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:220)
> at javax.servlet.api@1.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:706)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.CXFServletExt.service(CXFServletExt.java:136)
> at org.jboss.ws.spi@3.2.2.Final//org.jboss.wsf.spi.deployment.WSFServlet.service(WSFServlet.java:140)
> at javax.servlet.api@1.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> at io.undertow.core@2.0.13.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> at io.undertow.core@2.0.13.Final//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> at java.base/java.lang.Thread.run(Thread.java:844)
> [0m[33m13:13:09,199 WARNING [org.apache.cxf.phase.PhaseInterceptorChain] (default task-3) Application {http://www.jboss.org/jbossws/ws-extensions/wssecuritypolicy}SecurityService#{http://www.jboss.org/jbossws/ws-extensions/wssecuritypolicy}sayHello has thrown exception, unwinding now: org.apache.cxf.interceptor.Fault: WFLYEJB0364: Invocation on method: public java.lang.String org.jboss.test.ws.jaxws.samples.wsse.policy.jaas.EJBDigestServiceImpl.sayHello() of bean: EjbEndpoint is not allowed
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162)
> at org.apache.cxf.impl//org.apache.cxf.jaxws.AbstractJAXWSMethodInvoker.createFault(AbstractJAXWSMethodInvoker.java:267)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:128)
> at org.apache.cxf.impl//org.apache.cxf.jaxws.AbstractJAXWSMethodInvoker.invoke(AbstractJAXWSMethodInvoker.java:232)
> at org.apache.cxf.impl//org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:85)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.JBossWSInvoker.invoke(JBossWSInvoker.java:146)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor$2.run(ServiceInvokerInterceptor.java:126)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:131)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
> at org.apache.cxf.impl//org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.RequestHandlerImpl.handleHttpRequest(RequestHandlerImpl.java:110)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.transport.ServletHelper.callRequestHandler(ServletHelper.java:134)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.CXFServletExt.invoke(CXFServletExt.java:88)
> at org.apache.cxf.impl//org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:301)
> at org.apache.cxf.impl//org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:220)
> at javax.servlet.api@1.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:706)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.CXFServletExt.service(CXFServletExt.java:136)
> at org.jboss.ws.spi@3.2.2.Final//org.jboss.wsf.spi.deployment.WSFServlet.service(WSFServlet.java:140)
> at javax.servlet.api@1.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
> at io.undertow.core@2.0.13.Final//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
> at io.undertow.core@2.0.13.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow@14.0.0.CR1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> at io.undertow.servlet@2.0.13.Final//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> at io.undertow.core@2.0.13.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> at io.undertow.core@2.0.13.Final//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> at org.jboss.threads@2.3.2.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> at java.base/java.lang.Thread.run(Thread.java:844)
> Caused by: javax.ejb.EJBAccessException: WFLYEJB0364: Invocation on method: public java.lang.String org.jboss.test.ws.jaxws.samples.wsse.policy.jaas.EJBDigestServiceImpl.sayHello() of bean: EjbEndpoint is not allowed
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.security.RolesAllowedInterceptor.processInvocation(RolesAllowedInterceptor.java:67)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.security.SecurityDomainInterceptor.processInvocation(SecurityDomainInterceptor.java:44)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3@14.0.0.CR1-SNAPSHOT//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee@14.0.0.CR1-SNAPSHOT//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation@1.5.1.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
> at org.jboss.as.ee@14.0.0.CR1-SNAPSHOT//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.invokeInternal(AbstractInvocationHandler.java:161)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.lambda$invoke$0(AbstractInvocationHandler.java:116)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:382)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.Scoped.runAsFunctionEx(Scoped.java:161)
> at org.wildfly.security.elytron-private@1.5.4.Final//org.wildfly.security.auth.server.Scoped.runAs(Scoped.java:73)
> at org.jboss.as.webservices//org.jboss.as.webservices.security.ElytronSecurityDomainContextImpl.runAs(ElytronSecurityDomainContextImpl.java:88)
> at org.jboss.as.webservices//org.jboss.as.webservices.invocation.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:115)
> at org.jboss.ws.cxf.jbossws-cxf-server@5.2.4-SNAPSHOT//org.jboss.wsf.stack.cxf.JBossWSInvoker.performInvocation(JBossWSInvoker.java:170)
> at org.apache.cxf@3.2.5.jbossorg-1//org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
> ... 62 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (WFWIP-21) [Artemis 2.x upgrade] Lost message when MDB is resending messages under high load
by Erich Duda (JIRA)
[ https://issues.jboss.org/browse/WFWIP-21?page=com.atlassian.jira.plugin.s... ]
Erich Duda commented on WFWIP-21:
---------------------------------
The run \[1\] finished. The reproducer passed 50 times with Artemis 1.5.x. So it is confirmed that this is a regression.
\[1\] https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/mnovak-verifier-...
> [Artemis 2.x upgrade] Lost message when MDB is resending messages under high load
> ---------------------------------------------------------------------------------
>
> Key: WFWIP-21
> URL: https://issues.jboss.org/browse/WFWIP-21
> Project: WildFly WIP
> Issue Type: Bug
> Components: Artemis
> Reporter: Miroslav Novak
> Assignee: Martyn Taylor
> Priority: Blocker
> Labels: activemq, feature-branch-blocker
>
> Test scenario:
> * Start cluster A of nodes node-1, node-3
> * Start cluster B of nodes node-2, node-4
> * Send messages to queue on node-1
> * Deploy mdbs to servers in cluster A. This mdb reads messages from local queue, sends them to remote queue on cluster B and inserts them into database
> * Deploy mdbs to servers in cluster B. This mdb reads messages from local queue and inserts them into database
> * Cause CPU overload (for 5 min) on server node-2 when mdbs on cluster1 and 2 are processing mesages
> * Restart failed server
> * Let MDBs to process remaining messages
> Pass Criteria: Number of sent message is equal number of records(lines) in database and messages in
> Actual Result:
> Sometimes happens that one record is missing in database which means that one message was not processed be MDB in cluster 2.
> This looks like broker related regression against Artemis 1.5.
> Wildfly: https://github.com/jmesnil/wildfly/tree/WFLY-9407_upgrade_artemis_2.4.0_w... (06c878a313d3cad323889d017e60fd5533204d1a)
> Artemis tag 2.5.0.Final
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0
[JBoss JIRA] (WFLY-10894) [WFLY] Tests for enabling constraint drive authentication mode for elytron
by Bartosz Spyrko-Śmietanko (JIRA)
[ https://issues.jboss.org/browse/WFLY-10894?page=com.atlassian.jira.plugin... ]
Bartosz Spyrko-Śmietanko updated WFLY-10894:
--------------------------------------------
Summary: [WFLY] Tests for enabling constraint drive authentication mode for elytron (was: [WFLY] enable constraint drive authentication mode for elytron)
> [WFLY] Tests for enabling constraint drive authentication mode for elytron
> --------------------------------------------------------------------------
>
> Key: WFLY-10894
> URL: https://issues.jboss.org/browse/WFLY-10894
> Project: WildFly
> Issue Type: Bug
> Components: Security, Web (Undertow)
> Reporter: Bartosz Spyrko-Śmietanko
> Assignee: Bartosz Spyrko-Śmietanko
> Labels: test_included
>
> Integration code for JBEAP-15054
> ----
> When elytron is enabled constraint driven authentication method (i.e. proactive-authentication=false) has no effect.
> If you try to request an unsecured page sending in an invalid user with basic authentication, you should get the page returned with constraint drive authentication and a 401 with proactive authentication. This is what happens without Elytron enabled. But if you enable Elytron it gives a 401 in both cases.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
- 1
- 0