[JBoss JIRA] (WFLY-9920) XAException/HeuristicRollbackException and others due to TimeoutException: Timed out waiting for topology
by Radoslav Husar (Jira)
[ https://issues.jboss.org/browse/WFLY-9920?page=com.atlassian.jira.plugin.... ]
Radoslav Husar commented on WFLY-9920:
--------------------------------------
[~tommaso-borgato] Lets resolve this then? Or is there anything on here pending?
> XAException/HeuristicRollbackException and others due to TimeoutException: Timed out waiting for topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: WFLY-9920
> URL: https://issues.jboss.org/browse/WFLY-9920
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 12.0.0.Final, 13.0.0.Beta1
> Reporter: Michal Vinkler
> Assignee: Paul Ferraro
> Priority: Critical
>
> Seen in many failover tests:
> failover-http-session-jvmkill-repl-sync
> failover-http-session-shutdown-dist-sync
> failover-http-session-shutdown-repl-sync
> failover-http-granular-jvmkill-dist-sync
> failover-http-granular-shutdown-repl-sync
> failover-ejb-ejbservlet-jvmkill-dist-sync
> failover-ejb-ejbservlet-undeploy-dist-sync
> During our failover testing, we have seen many occurrences of "TimeoutException: Timed out waiting for topology X".
> Here are the variants we have identified:
> {code:title=ISPN000136: Error executing command PrepareCommand}
> [JBossINF] [0m[31m06:14:11,769 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p9-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionAttributesKey(u8mzm-09YKM6zLEb7aaTZapiXz3-y4i8OOMegJRL), SessionCreationMetaDataKey(u8mzm-09YKM6zLEb7aaTZapiXz3-y4i8OOMegJRL), SessionAccessMetaDataKey(u8mzm-09YKM6zLEb7aaTZapiXz3-y4i8OOMegJRL)]: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 29
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor$CancellableRetry.run(BaseStateTransferInterceptor.java:337)
> [JBossINF] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> {code}
> link: http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-fa...
> {code:title=ISPN000097: Error while processing a prepare in a single-phase transaction}
> [JBossINF] [0m[31m06:14:11,773 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-65) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 29
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
> [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:166)
> [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:126)
> [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:446)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:493)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110)
> [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945)
> [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:577)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1979)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1481)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1360)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 29
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor$CancellableRetry.run(BaseStateTransferInterceptor.java:337)
> [JBossINF] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] ... 1 more
> {code}
> link: http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-fa...
> {code:title=ISPN000927: exception while committing: javax.transaction.xa.XAException}
> [JBossINF] [0m[33m06:14:11,780 WARN [org.infinispan.commons.tx.TransactionImpl] (default task-65) ISPN000927: exception while committing: javax.transaction.xa.XAException
> [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:222)
> [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:168)
> [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:126)
> [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:446)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:493)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110)
> [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945)
> [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:577)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1979)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1481)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1360)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 29
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
> [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:166)
> [JBossINF] ... 30 more
> [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 29
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor$CancellableRetry.run(BaseStateTransferInterceptor.java:337)
> [JBossINF] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] ... 1 more
> {code}
> link: http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-fa...
> {code:title=UT005023: ... HeuristicRollbackException}
> [JBossINF] [0m[31m08:57:51,979 ERROR [io.undertow.request] (default task-56) UT005023: Exception handling request to /clusterbench-granular/granular: org.infinispan.commons.CacheException: javax.transaction.HeuristicRollbackException
> [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:102)
> [JBossINF] at org.wildfly.clustering.server.registry.CacheRegistry.getEntry(CacheRegistry.java:174)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanRouteLocator.locate(InfinispanRouteLocator.java:57)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionIdentifierCodec.encode(DistributableSessionIdentifierCodec.java:48)
> [JBossINF] at org.wildfly.extension.undertow.session.CodecSessionConfig.findSessionId(CodecSessionConfig.java:60)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl$ServletContextSessionConfig.findSessionId(ServletContextImpl.java:1215)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:158)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:933)
> [JBossINF] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:69)
> [JBossINF] at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
> [JBossINF] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1979)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1481)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1360)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] Caused by: javax.transaction.HeuristicRollbackException
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:478)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:493)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110)
> [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
> [JBossINF] ... 36 more
> [JBossINF] Caused by: javax.transaction.xa.XAException
> [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:222)
> [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:168)
> [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:126)
> [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:446)
> [JBossINF] ... 40 more
> [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 21
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
> [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:166)
> [JBossINF] ... 43 more
> [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 21
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor$CancellableRetry.run(BaseStateTransferInterceptor.java:337)
> [JBossINF] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] ... 1 more
> {code}
> link: http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-fa...
> The client is affected, getting 500 as a response:
> {code}
> 2018/02/16 08:57:51:748 EST [WARN ][Runner - 1950] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>>
> org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>
> at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:164)
> at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
> at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> link: http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-fa...
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 8 months
[JBoss JIRA] (WFWIP-142) Vendor metrics - loaded_modules reports incorrect value
by Jeff Mesnil (Jira)
[ https://issues.jboss.org/browse/WFWIP-142?page=com.atlassian.jira.plugin.... ]
Jeff Mesnil resolved WFWIP-142.
-------------------------------
Resolution: Done
loaded_modules metric are no longer exposed
> Vendor metrics - loaded_modules reports incorrect value
> -------------------------------------------------------
>
> Key: WFWIP-142
> URL: https://issues.jboss.org/browse/WFWIP-142
> Project: WildFly WIP
> Issue Type: Bug
> Components: MP Metrics
> Reporter: Rostislav Svoboda
> Assignee: Jeff Mesnil
> Priority: Blocker
>
> Vendor metrics - loaded_modules reports incorrect value
> {code}
> curl http://localhost:10090/metrics/vendor 2>/dev/null | grep loaded_modules
> # HELP vendor:loaded_modules Number of loaded modules
> # TYPE vendor:loaded_modules gauge
> vendor:loaded_modules 23.0
> {code}
> When I tried Thorntail 2 based deployment I got {{vendor:loaded_modules 298.0}}
> WF probably reports incorrect value for loaded modules.
> The same value is reported when I deploy {{quickstart/helloworld-rs/target/helloworld-rs.war}} application. For sure new modules are triggered by deplyment.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 8 months
[JBoss JIRA] (WFWIP-143) Deployment of application (quickstart/helloworld-rs) triggers CDI warnings for io.smallrye.metrics stuff
by Jeff Mesnil (Jira)
[ https://issues.jboss.org/browse/WFWIP-143?page=com.atlassian.jira.plugin.... ]
Jeff Mesnil resolved WFWIP-143.
-------------------------------
Resolution: Done
issue resolved in smallrye-metrics 1.1.2
> Deployment of application (quickstart/helloworld-rs) triggers CDI warnings for io.smallrye.metrics stuff
> --------------------------------------------------------------------------------------------------------
>
> Key: WFWIP-143
> URL: https://issues.jboss.org/browse/WFWIP-143
> Project: WildFly WIP
> Issue Type: Bug
> Components: MP Metrics
> Reporter: Rostislav Svoboda
> Assignee: Jeff Mesnil
> Priority: Blocker
>
> Deployment of application (quickstart/helloworld-rs) triggers CDI warning for io.smallrye.metrics stuff
> {code}
> 13:42:42,723 WARN [org.jboss.weld.Bootstrap] (MSC service thread 1-2) WELD-000146: BeforeBeanDiscovery.addAnnotatedType(AnnotatedType<?>) used for class io.smallrye.metrics.MetricProducer is deprecated from CDI 1.1!
> 13:42:42,734 WARN [org.jboss.weld.Bootstrap] (MSC service thread 1-2) WELD-000146: BeforeBeanDiscovery.addAnnotatedType(AnnotatedType<?>) used for class io.smallrye.metrics.interceptors.MetricNameFactory is deprecated from CDI 1.1!
> 13:42:42,736 WARN [org.jboss.weld.Bootstrap] (MSC service thread 1-2) WELD-000146: BeforeBeanDiscovery.addAnnotatedType(AnnotatedType<?>) used for class io.smallrye.metrics.interceptors.MetricsInterceptor is deprecated from CDI 1.1!
> 13:42:42,741 WARN [org.jboss.weld.Bootstrap] (MSC service thread 1-2) WELD-000146: BeforeBeanDiscovery.addAnnotatedType(AnnotatedType<?>) used for class io.smallrye.metrics.MetricRegistries is deprecated from CDI 1.1!
> 13:42:42,744 WARN [org.jboss.weld.Bootstrap] (MSC service thread 1-2) WELD-000146: BeforeBeanDiscovery.addAnnotatedType(AnnotatedType<?>) used for class io.smallrye.metrics.interceptors.MeteredInterceptor is deprecated from CDI 1.1!
> 13:42:42,750 WARN [org.jboss.weld.Bootstrap] (MSC service thread 1-2) WELD-000146: BeforeBeanDiscovery.addAnnotatedType(AnnotatedType<?>) used for class io.smallrye.metrics.interceptors.CountedInterceptor is deprecated from CDI 1.1!
> 13:42:42,753 WARN [org.jboss.weld.Bootstrap] (MSC service thread 1-2) WELD-000146: BeforeBeanDiscovery.addAnnotatedType(AnnotatedType<?>) used for class io.smallrye.metrics.interceptors.TimedInterceptor is deprecated from CDI 1.1!
> 13:42:42,756 WARN [org.jboss.weld.Bootstrap] (MSC service thread 1-2) WELD-000146: BeforeBeanDiscovery.addAnnotatedType(AnnotatedType<?>) used for class io.smallrye.metrics.MetricsRequestHandler is deprecated from CDI 1.1!
> {code}
> These WARN messages must go away by using proper api. Some EAP customers are doing log auditing and analysis for each entry which is INFO+.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 8 months
[JBoss JIRA] (WFWIP-140) Server shutdown or microprofile-metrics-smallrye subsystem removal causes INFO log entry for each metric
by Jeff Mesnil (Jira)
[ https://issues.jboss.org/browse/WFWIP-140?page=com.atlassian.jira.plugin.... ]
Jeff Mesnil resolved WFWIP-140.
-------------------------------
Resolution: Done
issue resolved in smallrye-metrics 1.1.2
> Server shutdown or microprofile-metrics-smallrye subsystem removal causes INFO log entry for each metric
> --------------------------------------------------------------------------------------------------------
>
> Key: WFWIP-140
> URL: https://issues.jboss.org/browse/WFWIP-140
> Project: WildFly WIP
> Issue Type: Bug
> Components: MP Metrics
> Reporter: Rostislav Svoboda
> Assignee: Jeff Mesnil
> Priority: Blocker
>
> Server shutdown causes INFO log entry for each metric.
> microprofile-metrics-smallrye subsystem removal causes INFO log entry for each metric.
> I think this should be decreased to DEBUG level.
> {code}
> 12:09:38,411 INFO [io.smallrye.metrics.MetricsRegistryImpl] (MSC service thread 1-8) Remove metric [name: classloader.currentLoadedClass.count]
> 12:09:38,412 INFO [io.smallrye.metrics.MetricsRegistryImpl] (MSC service thread 1-8) Remove metric [name: classloader.totalLoadedClass.count]
> 12:09:38,412 INFO [io.smallrye.metrics.MetricsRegistryImpl] (MSC service thread 1-8) Remove metric [name: classloader.totalUnloadedClass.count]
> 12:09:38,412 INFO [io.smallrye.metrics.MetricsRegistryImpl] (MSC service thread 1-8) Remove metric [name: cpu.availableProcessors]
> 12:09:38,412 INFO [io.smallrye.metrics.MetricsRegistryImpl] (MSC service thread 1-8) Remove metric [name: cpu.systemLoadAverage]
> 12:09:38,412 INFO [io.smallrye.metrics.MetricsRegistryImpl] (MSC service thread 1-8) Remove metric [name: gc.G1 Old Generation.count]
> ...
> {code}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 8 months
[JBoss JIRA] (WFWIP-128) Subsystem metrics names split in incorrect places
by Jeff Mesnil (Jira)
[ https://issues.jboss.org/browse/WFWIP-128?page=com.atlassian.jira.plugin.... ]
Jeff Mesnil resolved WFWIP-128.
-------------------------------
Resolution: Done
fixed in smallrye-metrics 1.1.2
> Subsystem metrics names split in incorrect places
> -------------------------------------------------
>
> Key: WFWIP-128
> URL: https://issues.jboss.org/browse/WFWIP-128
> Project: WildFly WIP
> Issue Type: Bug
> Components: MP Metrics
> Reporter: Rostislav Svoboda
> Assignee: Jeff Mesnil
> Priority: Blocker
>
> Subsystem metrics names split (underscore) in incorrect places. So far I noticed these things in jgroups and messaging.
> Underscore is used in some attributes quite often, conversion of {{/}} to {{_}} adds some confusion.
> JGroups
> * jgroups_channel_ee_protocol_un_ic_as_t3 ...
> * jgroups_channel_ee_protocol_pbcast_gm_s_view ...
> * jgroups_channel_ee_protocol_fr_ag3_level ...
> * jgroups_channel_ee_protocol_fd_so_ck_id ...
> Messaging
> * jms_queue_dl_q ...
> {code}
> [standalone@localhost:10090 channel=ee] ls /subsystem=jgroups/channel=ee/protocol
> FD_ALL FRAG3 MFC UDP UNICAST3 pbcast.GMS pbcast.STABLE
> FD_SOCK MERGE3 PING UFC VERIFY_SUSPECT pbcast.NAKACK2
> [standalone@localhost:10090 channel=ee] ls /subsystem=jgroups/channel=ee/protocol=FD_SOCK
> after_creation_hook=undefined external_addr=undefined log_suspected_messages=true pingable_members=[rs]
> bind_addr=/127.0.0.1 external_port=0 members=(1) rs port_range=50
> cache_max_age=10000 get_cache_timeout=1000 node_crash_monitor_running=false sock_conn_timeout=1000
> cache_max_elements=200 id=2 num_suspect_events_generated=0 start_port=0
> client_bind_port=0 keep_alive=true num_suspected_members=0 stats=false
> client_bind_port_actual=0 level=undefined num_tries=3 suspect_msg_interval=5000
> ergonomics=true local_address=rs ping_dest=null suspected_members=[]
> {code}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 8 months
[JBoss JIRA] (WFLY-11338) Upgrade Artemis to 2.6.3.jbossorg-0012
by ehsavoie Hugonnet (Jira)
ehsavoie Hugonnet created WFLY-11338:
----------------------------------------
Summary: Upgrade Artemis to 2.6.3.jbossorg-0012
Key: WFLY-11338
URL: https://issues.jboss.org/browse/WFLY-11338
Project: WildFly
Issue Type: Component Upgrade
Components: JMS
Affects Versions: 14.0.0.Final
Reporter: ehsavoie Hugonnet
Assignee: ehsavoie Hugonnet
Upgrade Artemis to 2.6.3.jbossorg-0012
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 8 months
[JBoss JIRA] (JGRP-2298) InputStream is processed after the different version error
by Bela Ban (Jira)
[ https://issues.jboss.org/browse/JGRP-2298?page=com.atlassian.jira.plugin.... ]
Bela Ban edited comment on JGRP-2298 at 11/13/18 9:49 AM:
----------------------------------------------------------
Since even {{length}} might be incorrect, we don't know how many bytes to discard. IMO, the safest solution is to throw an exception, catch it in TCP (TcpConnection) and simply close the connection.
was (Author: belaban):
Since even {{Length}} might be incorrect, we don;'t know how many bytes to discard. IMP, the safest solution would be to throw an exception, catch it in TCP (TcpConnection) and simply close the connection.
> InputStream is processed after the different version error
> ----------------------------------------------------------
>
> Key: JGRP-2298
> URL: https://issues.jboss.org/browse/JGRP-2298
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 4.0.11, 4.0.13
> Environment: OS: Debian 8
> Reporter: Boris Kantor
> Assignee: Bela Ban
> Priority: Major
> Fix For: 4.0.16
>
>
> Occasionally JGroups shows up a warning when receiving a message with a different JGroups version:
> {{JGRP000010: packet from 10.10.10.130:7800 has different version (0.0.0) than ours (4.0.11); packet is discarded | (Log4J2LogImpl.java:91)}}
> Although the message is then discarded, the socket connection will be kept open. The input stream still contains data that will be processed and falsly interpreted as the next message.
> If you have bad luck, the next bytes will match the correct version number. This leads then to following errors:
> {code:java}
> 2018-07-17 10:15:34.052 [Connect | ] WARN TCP - JGRP000010: packet from 10.10.10.130:7800 has different version (0.0.0) than ours (4.0.11); packet is discarded | (Log4J2LogImpl.java:91)
> 2018-07-17 10:15:34.059 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 8237 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.065 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 8237 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.072 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.io.IOException: length has to be 4 or 16 bytes (was 47 bytes)
> at org.jgroups.stack.IpAddress.readFrom(IpAddress.java:171)
> at org.jgroups.util.Util.readAddress(Util.java:1379)
> at org.jgroups.Message.readFrom(Message.java:731)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.076 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 28789 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.077 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 8237 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.078 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.io.IOException: length has to be 4 or 16 bytes (was 47 bytes)
> at org.jgroups.stack.IpAddress.readFrom(IpAddress.java:171)
> at org.jgroups.util.Util.readAddress(Util.java:1379)
> at org.jgroups.Message.readFrom(Message.java:731)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.078 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 8237 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.079 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.io.IOException: length has to be 4 or 16 bytes (was 47 bytes)
> at org.jgroups.stack.IpAddress.readFrom(IpAddress.java:171)
> at org.jgroups.util.Util.readAddress(Util.java:1379)
> at org.jgroups.Message.readFrom(Message.java:731)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.080 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 12320 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.089 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 25193 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.107 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 25970 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.util.Util.readMessageBatch(Util.java:1193)
> at org.jgroups.protocols.TP.handleMessageBatch(TP.java:1329)
> at org.jgroups.protocols.TP.receive(TP.java:1321)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> When it comes to the payload of the message the size of the payload can be as big as 2GB ({{len=in.readInt();}}). On small machines this will lead to an OutOfMemoryError.
> There is a need for some handling of such misinterpreted messages. Like closing and reopening a socket connection or completely ignoring the following bytes until a new message begins or an other algorithm that would prevent the processing of the remaining bytes.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 8 months
[JBoss JIRA] (JGRP-2298) InputStream is processed after the different version error
by Bela Ban (Jira)
[ https://issues.jboss.org/browse/JGRP-2298?page=com.atlassian.jira.plugin.... ]
Bela Ban edited comment on JGRP-2298 at 11/13/18 9:49 AM:
----------------------------------------------------------
Hmm... at connection setup, the peer sends both a cookie _and_ the version number, and if either of them doesn't match, the connection is closed.
This means, you actually got *past* this point and a peer connection turned sour during normal communication? Is this a real scenario? The only 2 things I could think of that could cause this are (1) peer crash *after* sending half of a message and (2) malicious code.
(1) would be handled as the connection would be terminated. Do you suspect (2)?
was (Author: belaban):
Hmm... at connection setup, the peer sends both a cookie _and_ the version number, and if either of them doesn't match, the connection is closed.
This means, you actually got *past* this point and a peer connection turned sour during normal communication? Is this a real scenario? The only 2 things I could think of that could cause this are (1) peer crash after sending half of a message and (2) malicious code.
(1) would be handled as the connection would be terminated. Do you suspect (2)?
> InputStream is processed after the different version error
> ----------------------------------------------------------
>
> Key: JGRP-2298
> URL: https://issues.jboss.org/browse/JGRP-2298
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 4.0.11, 4.0.13
> Environment: OS: Debian 8
> Reporter: Boris Kantor
> Assignee: Bela Ban
> Priority: Major
> Fix For: 4.0.16
>
>
> Occasionally JGroups shows up a warning when receiving a message with a different JGroups version:
> {{JGRP000010: packet from 10.10.10.130:7800 has different version (0.0.0) than ours (4.0.11); packet is discarded | (Log4J2LogImpl.java:91)}}
> Although the message is then discarded, the socket connection will be kept open. The input stream still contains data that will be processed and falsly interpreted as the next message.
> If you have bad luck, the next bytes will match the correct version number. This leads then to following errors:
> {code:java}
> 2018-07-17 10:15:34.052 [Connect | ] WARN TCP - JGRP000010: packet from 10.10.10.130:7800 has different version (0.0.0) than ours (4.0.11); packet is discarded | (Log4J2LogImpl.java:91)
> 2018-07-17 10:15:34.059 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 8237 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.065 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 8237 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.072 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.io.IOException: length has to be 4 or 16 bytes (was 47 bytes)
> at org.jgroups.stack.IpAddress.readFrom(IpAddress.java:171)
> at org.jgroups.util.Util.readAddress(Util.java:1379)
> at org.jgroups.Message.readFrom(Message.java:731)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.076 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 28789 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.077 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 8237 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.078 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.io.IOException: length has to be 4 or 16 bytes (was 47 bytes)
> at org.jgroups.stack.IpAddress.readFrom(IpAddress.java:171)
> at org.jgroups.util.Util.readAddress(Util.java:1379)
> at org.jgroups.Message.readFrom(Message.java:731)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.078 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 8237 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.079 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.io.IOException: length has to be 4 or 16 bytes (was 47 bytes)
> at org.jgroups.stack.IpAddress.readFrom(IpAddress.java:171)
> at org.jgroups.util.Util.readAddress(Util.java:1379)
> at org.jgroups.Message.readFrom(Message.java:731)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.080 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 12320 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.089 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 25193 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.107 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 25970 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.util.Util.readMessageBatch(Util.java:1193)
> at org.jgroups.protocols.TP.handleMessageBatch(TP.java:1329)
> at org.jgroups.protocols.TP.receive(TP.java:1321)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> When it comes to the payload of the message the size of the payload can be as big as 2GB ({{len=in.readInt();}}). On small machines this will lead to an OutOfMemoryError.
> There is a need for some handling of such misinterpreted messages. Like closing and reopening a socket connection or completely ignoring the following bytes until a new message begins or an other algorithm that would prevent the processing of the remaining bytes.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 8 months
[JBoss JIRA] (WFLY-11337) Hibernate tests failures with security manager
by Martin Choma (Jira)
[ https://issues.jboss.org/browse/WFLY-11337?page=com.atlassian.jira.plugin... ]
Martin Choma reassigned WFLY-11337:
-----------------------------------
Assignee: (was: Stefan Guilhen)
> Hibernate tests failures with security manager
> ----------------------------------------------
>
> Key: WFLY-11337
> URL: https://issues.jboss.org/browse/WFLY-11337
> Project: WildFly
> Issue Type: Bug
> Components: Security Manager, Test Suite
> Affects Versions: 14.0.0.Final
> Reporter: Martin Choma
> Priority: Major
>
> As a limitation documented by JBEAP-15368 there is lot of tests (approximately 75) using hibernate that are failing with security manager. E.g in package
> * org.jboss.as.test.compat.jpa.hibernate.transformer
> * org.jboss.as.test.integration.hibernate
> * org.jboss.as.test.integration.hibernate.envers
> This failure contains exception like this
> {noformat}
> Caused by: java.security.AccessControlException: WFSM000001: Permission check failed (permission "("java.io.FilePermission" "/store/repository/org/hibernate/hibernate-envers/5.3.7.Final/hibernate-envers-5.3.7.Final.jar" "read")" in code source "(vfs:/content/hibernate4naturalid_test.ear/beans.jar <no signer certificates>)" of "ModuleClassLoader for Module "deployment.hibernate4naturalid_test.ear.beans.jar" from Service Module Loader")
> {noformat}
> Is there way to workaround this limitation in testsuite?
> [1] https://ci.wildfly.org/viewLog.html?buildId=128138&buildTypeId=WF_MasterS...
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 8 months