[JBoss JIRA] (WFLY-11292) Legacy EJB Client: High fail rate
by Flavia Rainone (Jira)
[ https://issues.jboss.org/browse/WFLY-11292?page=com.atlassian.jira.plugin... ]
Flavia Rainone resolved WFLY-11292.
-----------------------------------
Resolution: Done
> Legacy EJB Client: High fail rate
> ---------------------------------
>
> Key: WFLY-11292
> URL: https://issues.jboss.org/browse/WFLY-11292
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, EJB
> Affects Versions: 15.0.0.Beta1
> Reporter: tommaso borgato
> Assignee: Flavia Rainone
> Priority: Blocker
> Attachments: jboss-ejb-client.properties, logs-10clients.zip, logs.zip
>
>
> This bug is being filed as Blocker because we are observing and elevated fail rate: roughly a thousand each run for (about 0.3%).
> h2. WildFly Built from master branch on 6 Nov 2018
> With this WildFly version (client org.jboss:jboss-ejb-client-legacy:3.0.2.Final-redhat-1) in a scenario with 4 clustered nodes where nodes are failed via jboss shut-down / restart: after node 1 of 4 is shut-down, a a series of errors start on the client side the yield to 1097 errors on a total of 340218 samples;
> find [here|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/...] the complete logs;
> the start of client errors is here:
> {noformat}
> 2018/11/07 05:28:51:255 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failing node 0 (perf18)
> 2018/11/07 05:28:51:270 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Total: Sessions: 2000, active: 2000, samples: 5118, throughput: 511.7 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 1 ms, max: 54 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5118 (100%)
> 2018/11/07 05:28:51:270 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - unknown-node: Sessions: 2000, active: 2000, samples: 5118, throughput: 511.7 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 1 ms, max: 54 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5118 (100%)
> 05:28:53,257 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-2) EJBCLIENT000016: Channel Channel ID c540daf7 (outbound) of Remoting connection 4ed5523c to perf18/10.16.90.54:8080 of endpoint "config-based-ejb-client-endpoint" <664ac6d5> can no longer process messages
> 05:28:53,277 ERROR [org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver] (Remoting "config-based-ejb-client-endpoint" task-2) Failed to open channel for context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@3cd51445, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)302066a5,channel=jboss.ejb,nodename=perf18]}
> org.jboss.remoting3.NotOpenException: Cannot open new channel because close was initiated
> at org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen(RemoteConnectionHandler.java:198)
> at org.jboss.remoting3.remote.RemoteConnectionHandler.open(RemoteConnectionHandler.java:335)
> at org.jboss.remoting3.ConnectionImpl.openChannel(ConnectionImpl.java:109)
> at org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection.openChannel(ConnectionPool.java:292)
> at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.associate(RemotingConnectionEJBReceiver.java:180)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:399)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:349)
> at org.jboss.ejb.client.remoting.EJBClientContextConnectionReconnectHandler.reconnect(EJBClientContextConnectionReconnectHandler.java:67)
> at org.jboss.ejb.client.EJBClientContext$ReconnectAttempt.run(EJBClientContext.java:1474)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> h2. WildFly Built from master branch on 7 Nov 2018
> Same situation.
> find [here|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/...] the complete logs;
> the start of client errors is here:
> {noformat}
> 2018/11/07 09:23:22:916 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failing node 0 (perf18)
> 2018/11/07 09:23:22:930 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Total: Sessions: 2000, active: 2000, samples: 5125, throughput: 512.4 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 2 ms, max: 32 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5125 (100%)
> 2018/11/07 09:23:22:930 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - unknown-node: Sessions: 2000, active: 2000, samples: 5125, throughput: 512.4 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 2 ms, max: 32 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5125 (100%)
> 09:23:24,924 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-4) EJBCLIENT000016: Channel Channel ID d940c707 (outbound) of Remoting connection 73cfdf01 to perf18/10.16.90.54:8080 of endpoint "config-based-ejb-client-endpoint" <7501616b> can no longer process messages
> 09:23:24,949 ERROR [org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver] (Remoting "config-based-ejb-client-endpoint" task-6) Failed to open channel for context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@56248025, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)5269f48d,channel=jboss.ejb,nodename=perf18]}
> org.jboss.remoting3.NotOpenException: Cannot open new channel because close was initiated
> at org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen(RemoteConnectionHandler.java:198)
> at org.jboss.remoting3.remote.RemoteConnectionHandler.open(RemoteConnectionHandler.java:335)
> at org.jboss.remoting3.ConnectionImpl.openChannel(ConnectionImpl.java:109)
> at org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection.openChannel(ConnectionPool.java:292)
> at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.associate(RemotingConnectionEJBReceiver.java:180)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:399)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:349)
> at org.jboss.ejb.client.remoting.EJBClientContextConnectionReconnectHandler.reconnect(EJBClientContextConnectionReconnectHandler.java:67)
> at org.jboss.ejb.client.EJBClientContext$ReconnectAttempt.run(EJBClientContext.java:1474)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 8 months
[JBoss JIRA] (WFLY-11489) SFSB not sticky on a single cluster node when clustering of the bean is disabled
by Richard Achmatowicz (Jira)
[ https://issues.jboss.org/browse/WFLY-11489?page=com.atlassian.jira.plugin... ]
Richard Achmatowicz edited comment on WFLY-11489 at 2/6/19 11:40 AM:
---------------------------------------------------------------------
[~brian.stansberry] I have added the test case that Paul created a while ago. I have also changed the ejb-client version from 4.0.15.Final to 4.1.0.Final as that will be the next release and that is the release which will contain EJBCLIENT-319. Using 4.0.15.Final was an error on my part.
Also, I ran the WFLY testsuite with the modified 4.1.0.Final-SNAPSHOT and all tests, including the new one, passed.
was (Author: rachmato):
[~brian.stansberry] I have added the test case that Paul created a while ago. I have also changed the ejb-client version from 4.0.15.Final to 4.1.0.Final as that will be the next release and that is the release which will contain EJBCLIENT-319. Using 4.0.15.Final was an error on my part.
> SFSB not sticky on a single cluster node when clustering of the bean is disabled
> --------------------------------------------------------------------------------
>
> Key: WFLY-11489
> URL: https://issues.jboss.org/browse/WFLY-11489
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, Remoting
> Affects Versions: 15.0.0.Final
> Environment: A clustered environment with 2 nodes. Both nodes started with an unmodified {{standalone-ha.xml}} configuration (see _Steps to Reproduce_)
> This issue happens on 15.0.0.Final and was tested as well as on current head (sha: 372697282dccefd0b9df48e6aa4dcb69e1c4b40f).
> Reporter: Jörg Bäsner
> Assignee: Richard Achmatowicz
> Priority: Major
> Attachments: reproducer.zip
>
>
> In case a stateful session bean is annotated with:
> {{@Stateful(passivationCapable=false)}}
> then the serialization is *disabled* and thus the bean is only available on a single cluster node.
> When a client now calls two different methods on this stateful bean it intermittently ends up on different cluster nodes, resulting in the following Exception:
> {code}
> ERROR [org.jboss.as.ejb3.invocation] (default task-2) WFLYEJB0034: EJB Invocation failed on component TestSessionEJB for method public abstract void test.ITestSession.method2(java.lang.String,java.lang.String) throws javax.ejb.EJBException,java.rmi.RemoteException: javax.ejb.NoSuchEJBException: WFLYEJB0168: Could not find EJB with id UUIDSessionID [4b0f4a27-40ba-411b-8852-0108a5ec64f4]
> at org.jboss.as.ejb3.component.stateful.StatefulComponentInstanceInterceptor.processInvocation(StatefulComponentInstanceInterceptor.java:55)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
> at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
> at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:618)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:406)
> at org.jboss.as.ejb3.remote.AssociationImpl.invokeWithIdentity(AssociationImpl.java:565)
> at org.jboss.as.ejb3.remote.AssociationImpl.invokeMethod(AssociationImpl.java:546)
> at org.jboss.as.ejb3.remote.AssociationImpl.lambda$receiveInvocationRequest$0(AssociationImpl.java:197)
> at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 8 months
[JBoss JIRA] (WFLY-11489) SFSB not sticky on a single cluster node when clustering of the bean is disabled
by Richard Achmatowicz (Jira)
[ https://issues.jboss.org/browse/WFLY-11489?page=com.atlassian.jira.plugin... ]
Richard Achmatowicz commented on WFLY-11489:
--------------------------------------------
[~brian.stansberry] I have added the test case that Paul created a while ago. I have also changed the ejb-client version from 4.0.15.Final to 4.1.0.Final as that will be the next release and that is the release which will contain EJBCLIENT-319. Using 4.0.15.Final was an error on my part.
> SFSB not sticky on a single cluster node when clustering of the bean is disabled
> --------------------------------------------------------------------------------
>
> Key: WFLY-11489
> URL: https://issues.jboss.org/browse/WFLY-11489
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, Remoting
> Affects Versions: 15.0.0.Final
> Environment: A clustered environment with 2 nodes. Both nodes started with an unmodified {{standalone-ha.xml}} configuration (see _Steps to Reproduce_)
> This issue happens on 15.0.0.Final and was tested as well as on current head (sha: 372697282dccefd0b9df48e6aa4dcb69e1c4b40f).
> Reporter: Jörg Bäsner
> Assignee: Richard Achmatowicz
> Priority: Major
> Attachments: reproducer.zip
>
>
> In case a stateful session bean is annotated with:
> {{@Stateful(passivationCapable=false)}}
> then the serialization is *disabled* and thus the bean is only available on a single cluster node.
> When a client now calls two different methods on this stateful bean it intermittently ends up on different cluster nodes, resulting in the following Exception:
> {code}
> ERROR [org.jboss.as.ejb3.invocation] (default task-2) WFLYEJB0034: EJB Invocation failed on component TestSessionEJB for method public abstract void test.ITestSession.method2(java.lang.String,java.lang.String) throws javax.ejb.EJBException,java.rmi.RemoteException: javax.ejb.NoSuchEJBException: WFLYEJB0168: Could not find EJB with id UUIDSessionID [4b0f4a27-40ba-411b-8852-0108a5ec64f4]
> at org.jboss.as.ejb3.component.stateful.StatefulComponentInstanceInterceptor.processInvocation(StatefulComponentInstanceInterceptor.java:55)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
> at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
> at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:618)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:406)
> at org.jboss.as.ejb3.remote.AssociationImpl.invokeWithIdentity(AssociationImpl.java:565)
> at org.jboss.as.ejb3.remote.AssociationImpl.invokeMethod(AssociationImpl.java:546)
> at org.jboss.as.ejb3.remote.AssociationImpl.lambda$receiveInvocationRequest$0(AssociationImpl.java:197)
> at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 8 months
[JBoss JIRA] (LOGMGR-236) java.lang.ArrayIndexOutOfBoundsException: 76 at org.jboss.logmanager.JDKSpecific.calculateCaller(JDKSpecific.java:112)
by James Perkins (Jira)
[ https://issues.jboss.org/browse/LOGMGR-236?page=com.atlassian.jira.plugin... ]
James Perkins moved WFLY-11671 to LOGMGR-236:
---------------------------------------------
Project: JBoss Log Manager (was: WildFly)
Key: LOGMGR-236 (was: WFLY-11671)
Component/s: core
(was: Logging)
Affects Version/s: (was: 14.0.1.Final)
(was: 15.0.1.Final)
> java.lang.ArrayIndexOutOfBoundsException: 76 at org.jboss.logmanager.JDKSpecific.calculateCaller(JDKSpecific.java:112)
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: LOGMGR-236
> URL: https://issues.jboss.org/browse/LOGMGR-236
> Project: JBoss Log Manager
> Issue Type: Bug
> Components: core
> Environment: $ uname -ar
> Linux manuel-XPS-13-9360 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
> $ java -version
> openjdk version "1.8.0_191"
> OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-0ubuntu0.18.04.1-b12)
> OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
> $ ./jboss-cli.sh --version
> JBoss Admin Command-line Interface
> JBOSS_HOME: /home/manuel/wildfly-15.0.1.Final
> Release: <connect to the controller and re-run the version command to see the release info>
> JAVA_HOME: null
> java.version: 1.8.0_191
> java.vm.vendor: Oracle Corporation
> java.vm.version: 25.191-b12
> os.name: Linux
> os.version: 4.15.0-43-generic
> Maybe another bug, the server was running in JBoss Developer Studio when I executed jboss-cli
> Reporter: Manuel Blechschmidt
> Assignee: James Perkins
> Priority: Major
>
> We are collecting Logs in a CDI RequestScoped bean. Afterwards we have a JAX RS filter that attaches these logs in the HTTP Warnings Header. In wildfly 12 is works great in wildfly 14 & 15 the following error happens:
> {code}
> 08:17:16,856 WARNING [de.example.logging.cdi.HttpWarningsHeaderFilter] (default task-1) Problem during formatting warning: java.lang.ArrayIndexOutOfBoundsException: 76
> at org.jboss.logmanager.JDKSpecific.calculateCaller(JDKSpecific.java:112)
> at org.jboss.logmanager.ExtLogRecord.calculateCaller(ExtLogRecord.java:335)
> at org.jboss.logmanager.ExtLogRecord.getSourceClassName(ExtLogRecord.java:399)
> at java.util.logging.SimpleFormatter.format(SimpleFormatter.java:143)
> at de.example.logging.cdi.HttpWarningsHeaderFilter.lambda$1(HttpWarningsHeaderFilter.java:34)
> at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
> at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at de.example.logging.cdi.HttpWarningsHeaderFilter.filter(HttpWarningsHeaderFilter.java:39)
> at de.example.logging.cdi.HttpWarningsHeaderFilter$Proxy$_$$_WeldClientProxy.filter(Unknown Source)
> at org.jboss.resteasy.core.interception.ContainerResponseContextImpl.filter(ContainerResponseContextImpl.java:356)
> at org.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:205)
> at org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:82)
> at org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:56)
> at org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:528)
> at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:459)
> at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:229)
> at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:135)
> at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:355)
> {code}
> {code}
> package de.example.logging.cdi;
> import java.io.IOException;
> import java.util.List;
> import java.util.logging.Formatter;
> import java.util.logging.Level;
> import java.util.logging.Logger;
> import java.util.logging.SimpleFormatter;
> import java.util.stream.Collectors;
> import javax.inject.Inject;
> import javax.ws.rs.container.ContainerRequestContext;
> import javax.ws.rs.container.ContainerResponseContext;
> import javax.ws.rs.container.ContainerResponseFilter;
> import javax.ws.rs.ext.Provider;
> @Provider
> public class HttpWarningsHeaderFilter implements ContainerResponseFilter {
> private static Logger log = Logger.getLogger(HttpWarningsHeaderFilter.class.getName());
> private static final Formatter simpleFormatter = new SimpleFormatter();
> @Inject
> Logs logs;
> @Override
> public void filter(ContainerRequestContext requestContext, ContainerResponseContext responseContext)
> throws IOException {
> List<String> logStrings = logs.getLogRecords().stream()
> // only output INFO or more important warnings (SERVERE, WARNING)
> .filter(lr -> lr.getLevel().intValue() >= Level.INFO.intValue()).map(lr -> {
> try {
> return simpleFormatter.format(lr);
> } catch (Exception e) {
> log.log(Level.WARNING, "Problem during formatting warning", e);
> return lr.getMessage();
> }
> }).collect(Collectors.toList());
> responseContext.getHeaders().add("Warnings", logStrings);
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 8 months
[JBoss JIRA] (WFLY-11678) ManagedExecutorService persists contextClassLoader reference to cause app classloader leaks
by James Perkins (Jira)
[ https://issues.jboss.org/browse/WFLY-11678?page=com.atlassian.jira.plugin... ]
James Perkins updated WFLY-11678:
---------------------------------
Priority: Critical (was: Major)
> ManagedExecutorService persists contextClassLoader reference to cause app classloader leaks
> -------------------------------------------------------------------------------------------
>
> Key: WFLY-11678
> URL: https://issues.jboss.org/browse/WFLY-11678
> Project: WildFly
> Issue Type: Bug
> Components: EE
> Environment: -- EAP 7.1.4
> Reporter: Lei Yu
> Assignee: James Perkins
> Priority: Critical
>
> managed executor:
> {code}
> <managed-executor-services>
> <managed-executor-service name="default" jndi-name="java:jboss/ee/concurrency/executor/default" context-service="default" hung-task-threshold="60000" keepalive-time="5000"/>
> </managed-executor-services>
> {code}
> With each undeploy, the application classloader is leaked and not released because contextClassLoader references persisted on the executor threads:
> {code}
> Class Name | Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread @ 0xc3677570 EE-ManagedExecutorService-default-Thread-13 Thread| 1 | 160 | 88 | 1,344
> '- contextClassLoader org.jboss.modules.ModuleClassLoader @ 0xc32b9d98 | 1 | 88 | 88 | 366,800
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> {code}
> Tracing with byteman, what's strange is that it sets the app classloader on setup:
> {code}
> Thread.setContextClassLoader: Thread[EE-ManagedExecutorService-default-Thread-16,5,ServerService ThreadGroup] ModuleClassLoader for Module "deployment.services-1.0.4-rc1.war" from Service Module Loader
> java.lang.Thread.setContextClassLoader(Thread.java:1477)
> org.wildfly.security.manager.WildFlySecurityManager.setCurrentContextClassLoaderPrivileged(WildFlySecurityManager.java:1272)
> org.jboss.as.ee.concurrent.handle.ClassLoaderContextHandleFactory$ClassLoaderSetupContextHandle.setup(ClassLoaderContextHandleFactory.java:83)
> org.jboss.as.ee.concurrent.ConcurrentContext$ChainedSetupContextHandle.setup(ConcurrentContext.java:166)
> org.jboss.as.ee.concurrent.DefaultContextSetupProviderImpl.setup(DefaultContextSetupProviderImpl.java:58)
> org.glassfish.enterprise.concurrent.internal.ManagedFutureTask.setupContext(ManagedFutureTask.java:121)
> org.glassfish.enterprise.concurrent.internal.ManagedThreadPoolExecutor.beforeExecute(ManagedThreadPoolExecutor.java:109)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> {code}
> And still sets that app classloader after task execution in the reset:
> {code}
> Thread.setContextClassLoader: Thread[EE-ManagedExecutorService-default-Thread-16,5,ServerService ThreadGroup] ModuleClassLoader for Module "deployment.services-1.0.4-rc1.war" from Service Module Loader
> java.lang.Thread.setContextClassLoader(Thread.java:1477)
> org.wildfly.security.manager.WildFlySecurityManager.setCurrentContextClassLoaderPrivileged(WildFlySecurityManager.java:1272)
> org.jboss.as.ee.concurrent.handle.ClassLoaderContextHandleFactory$ClassLoaderResetContextHandle.reset(ClassLoaderContextHandleFactory.java:113)
> org.jboss.as.ee.concurrent.ConcurrentContext$ChainedResetContextHandle.reset(ConcurrentContext.java:284)
> org.jboss.as.ee.concurrent.DefaultContextSetupProviderImpl.reset(DefaultContextSetupProviderImpl.java:63)
> org.glassfish.enterprise.concurrent.internal.ManagedFutureTask.resetContext(ManagedFutureTask.java:134)
> org.glassfish.enterprise.concurrent.internal.ManagedThreadPoolExecutor.afterExecute(ManagedThreadPoolExecutor.java:90)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> {code}
> So the thread is sitting in the pool with the app classloader. Is there some way we should be able to get the context loader cleared or unset from the app loader in the reset?
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 8 months