[JBoss JIRA] (WFLY-11292) Legacy EJB Client: High fail rate
by David Lloyd (Jira)
[ https://issues.jboss.org/browse/WFLY-11292?page=com.atlassian.jira.plugin... ]
David Lloyd commented on WFLY-11292:
------------------------------------
How exactly is this a blocker?
> Legacy EJB Client: High fail rate
> ---------------------------------
>
> Key: WFLY-11292
> URL: https://issues.jboss.org/browse/WFLY-11292
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, EJB
> Affects Versions: 15.0.0.Alpha1
> Reporter: tommaso borgato
> Assignee: Flavia Rainone
> Priority: Blocker
> Attachments: jboss-ejb-client.properties, logs-10clients.zip, logs.zip
>
>
> This bug is being filed as Blocker because we are observing and elevated fail rate: roughly a thousand each run for (about 0.3%).
> h2. WildFly Built from master branch on 6 Nov 2018
> With this WildFly version (client org.jboss:jboss-ejb-client-legacy:3.0.2.Final-redhat-1) in a scenario with 4 clustered nodes where nodes are failed via jboss shut-down / restart: after node 1 of 4 is shut-down, a a series of errors start on the client side the yield to 1097 errors on a total of 340218 samples;
> find [here|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/...] the complete logs;
> the start of client errors is here:
> {noformat}
> 2018/11/07 05:28:51:255 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failing node 0 (perf18)
> 2018/11/07 05:28:51:270 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Total: Sessions: 2000, active: 2000, samples: 5118, throughput: 511.7 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 1 ms, max: 54 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5118 (100%)
> 2018/11/07 05:28:51:270 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - unknown-node: Sessions: 2000, active: 2000, samples: 5118, throughput: 511.7 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 1 ms, max: 54 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5118 (100%)
> 05:28:53,257 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-2) EJBCLIENT000016: Channel Channel ID c540daf7 (outbound) of Remoting connection 4ed5523c to perf18/10.16.90.54:8080 of endpoint "config-based-ejb-client-endpoint" <664ac6d5> can no longer process messages
> 05:28:53,277 ERROR [org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver] (Remoting "config-based-ejb-client-endpoint" task-2) Failed to open channel for context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@3cd51445, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)302066a5,channel=jboss.ejb,nodename=perf18]}
> org.jboss.remoting3.NotOpenException: Cannot open new channel because close was initiated
> at org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen(RemoteConnectionHandler.java:198)
> at org.jboss.remoting3.remote.RemoteConnectionHandler.open(RemoteConnectionHandler.java:335)
> at org.jboss.remoting3.ConnectionImpl.openChannel(ConnectionImpl.java:109)
> at org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection.openChannel(ConnectionPool.java:292)
> at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.associate(RemotingConnectionEJBReceiver.java:180)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:399)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:349)
> at org.jboss.ejb.client.remoting.EJBClientContextConnectionReconnectHandler.reconnect(EJBClientContextConnectionReconnectHandler.java:67)
> at org.jboss.ejb.client.EJBClientContext$ReconnectAttempt.run(EJBClientContext.java:1474)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> h2. WildFly Built from master branch on 7 Nov 2018
> Same situation.
> find [here|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/...] the complete logs;
> the start of client errors is here:
> {noformat}
> 2018/11/07 09:23:22:916 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failing node 0 (perf18)
> 2018/11/07 09:23:22:930 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Total: Sessions: 2000, active: 2000, samples: 5125, throughput: 512.4 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 2 ms, max: 32 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5125 (100%)
> 2018/11/07 09:23:22:930 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - unknown-node: Sessions: 2000, active: 2000, samples: 5125, throughput: 512.4 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 2 ms, max: 32 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5125 (100%)
> 09:23:24,924 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-4) EJBCLIENT000016: Channel Channel ID d940c707 (outbound) of Remoting connection 73cfdf01 to perf18/10.16.90.54:8080 of endpoint "config-based-ejb-client-endpoint" <7501616b> can no longer process messages
> 09:23:24,949 ERROR [org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver] (Remoting "config-based-ejb-client-endpoint" task-6) Failed to open channel for context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@56248025, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)5269f48d,channel=jboss.ejb,nodename=perf18]}
> org.jboss.remoting3.NotOpenException: Cannot open new channel because close was initiated
> at org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen(RemoteConnectionHandler.java:198)
> at org.jboss.remoting3.remote.RemoteConnectionHandler.open(RemoteConnectionHandler.java:335)
> at org.jboss.remoting3.ConnectionImpl.openChannel(ConnectionImpl.java:109)
> at org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection.openChannel(ConnectionPool.java:292)
> at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.associate(RemotingConnectionEJBReceiver.java:180)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:399)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:349)
> at org.jboss.ejb.client.remoting.EJBClientContextConnectionReconnectHandler.reconnect(EJBClientContextConnectionReconnectHandler.java:67)
> at org.jboss.ejb.client.EJBClientContext$ReconnectAttempt.run(EJBClientContext.java:1474)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 6 months
[JBoss JIRA] (WFCORE-4207) Add capabilities in the Thread subsystem resource definitions
by Brian Stansberry (Jira)
[ https://issues.jboss.org/browse/WFCORE-4207?page=com.atlassian.jira.plugi... ]
Brian Stansberry commented on WFCORE-4207:
------------------------------------------
We need to be careful that this is the right thing to do.
The problem is these are not used as single resource definitions, as they originally were in the threads subsystem. They are just reusable code.
So, if subsystem X uses this code and creates an executor of some type named "default" it's not wrong for subsystem Y to also create the same kind of pool with the same name. But if they end up with the same capability name, that will no longer work.
Now, it's possible that this was already a problem, as this kind of thing would have led to a duplicate ServiceName problem. In which case we don't need to worry about it so much.
> Add capabilities in the Thread subsystem resource definitions
> -------------------------------------------------------------
>
> Key: WFCORE-4207
> URL: https://issues.jboss.org/browse/WFCORE-4207
> Project: WildFly Core
> Issue Type: Enhancement
> Components: Management
> Affects Versions: 7.0.0.Alpha5
> Reporter: ehsavoie Hugonnet
> Assignee: ehsavoie Hugonnet
> Priority: Major
>
> The deprecated thread subsystem resource definitions are used in several subsystems. Adding support for capabilities should help them get the model validation easier.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 6 months
[JBoss JIRA] (WFCORE-4211) Missing docs: Embedded container does not work without the JVM being launched with java.se module on Java 11
by Brian Stansberry (Jira)
[ https://issues.jboss.org/browse/WFCORE-4211?page=com.atlassian.jira.plugi... ]
Brian Stansberry updated WFCORE-4211:
-------------------------------------
Summary: Missing docs: Embedded container does not work without the JVM being launched with java.se module on Java 11 (was: Embedded container does not work without the JVM being launched with java.se module on Java 11)
> Missing docs: Embedded container does not work without the JVM being launched with java.se module on Java 11
> ------------------------------------------------------------------------------------------------------------
>
> Key: WFCORE-4211
> URL: https://issues.jboss.org/browse/WFCORE-4211
> Project: WildFly Core
> Issue Type: Task
> Components: Embedded
> Reporter: James Perkins
> Assignee: James Perkins
> Priority: Major
> Labels: Documentation, Java11
> Fix For: 7.0.0.Beta1
>
>
> With the embedded API the container can't be started on Java 11 without the {{--add-module=java.se}} argument being added to the JVM. There may not be a way to solve this, but it should likely at lest be documented. If this becomes just a documentation issue then the type should likely be changed to "Task".
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 6 months
[JBoss JIRA] (WFLY-11324) BATCH mode should not use Infinispan's XA transaction table
by Paul Ferraro (Jira)
Paul Ferraro created WFLY-11324:
-----------------------------------
Summary: BATCH mode should not use Infinispan's XA transaction table
Key: WFLY-11324
URL: https://issues.jboss.org/browse/WFLY-11324
Project: WildFly
Issue Type: Bug
Components: Clustering
Affects Versions: 14.0.0.Final
Reporter: Paul Ferraro
Assignee: Paul Ferraro
Currently, our logic for configuring BATCH mode is causing Infinispan to use the XA transaction table, instead of the the local transaction table. This not only makes commits more costly, but should avoid some graceful shutdown bugs with the XA transaction table.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 6 months
[JBoss JIRA] (WFLY-11323) Fixing flaky test WritableServiceBasedNamingStoreTestCase.testPermissions
by Brian Stansberry (Jira)
[ https://issues.jboss.org/browse/WFLY-11323?page=com.atlassian.jira.plugin... ]
Brian Stansberry commented on WFLY-11323:
-----------------------------------------
Here's the failure:
{code}
java.lang.ExceptionInInitializerError
at org.jboss.as.naming.NamingContext.check(NamingContext.java:585)
at org.jboss.as.naming.NamingContext.lookup(NamingContext.java:197)
at org.jboss.as.naming.NamingContext.lookup(NamingContext.java:184)
at org.jboss.as.naming.SecurityHelper.performAction(SecurityHelper.java:168)
at org.jboss.as.naming.SecurityHelper.access$000(SecurityHelper.java:51)
at org.jboss.as.naming.SecurityHelper$1.call(SecurityHelper.java:117)
at org.jboss.as.naming.SecurityHelper$4.run(SecurityHelper.java:213)
at java.security.AccessController.doPrivileged(Native Method)
at org.jboss.as.naming.SecurityHelper.runWithSecurityManager(SecurityHelper.java:210)
at org.jboss.as.naming.SecurityHelper.testActionWithPermission(SecurityHelper.java:114)
at org.jboss.as.naming.WritableServiceBasedNamingStoreTestCase.testPermissions(WritableServiceBasedNamingStoreTestCase.java:263)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessDeclaredMembers")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
at java.security.AccessController.checkPermission(AccessController.java:884)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at java.lang.Class.checkMemberAccess(Class.java:2348)
at java.lang.Class.getDeclaredField(Class.java:2067)
at org.wildfly.security.manager.WildFlySecurityManager.<clinit>(WildFlySecurityManager.java:113)
... 35 more
{code}
I believe the problem is WildFlySecurityManager.<clinit> is doing stuff that fails if it runs when the restricted permission scheme the test sets up is in effect. The proposed fix works around this by doing a lookup call before that restricted scheme is put in place, which has the effect of causing the WildFlySecurityManager.<clinit> to run.
> Fixing flaky test WritableServiceBasedNamingStoreTestCase.testPermissions
> -------------------------------------------------------------------------
>
> Key: WFLY-11323
> URL: https://issues.jboss.org/browse/WFLY-11323
> Project: WildFly
> Issue Type: Bug
> Components: Naming
> Environment: Any
> Reporter: ORD Testers
> Priority: Minor
>
> testPermissions fails when run by itself even when there’s no bug in the code.
> This test can be fixed by looking up the name before testActionWithPermission is called, as shown in the following pull request.
> https://github.com/wildfly/wildfly/pull/11833
> Let me know if you want to discuss more.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 6 months
[JBoss JIRA] (WFLY-11165) Eliminate unessential module dependencies on the org.jboss.as.transactions module
by Brian Stansberry (Jira)
[ https://issues.jboss.org/browse/WFLY-11165?page=com.atlassian.jira.plugin... ]
Brian Stansberry commented on WFLY-11165:
-----------------------------------------
Hi [~ochaloup] -- with the work on WFLY-11166 and WFLY-11167, plus just cleaning out some unneeded dependencies on org.jboss.as.transactions, the "simple" part of this is done. If you'd like I can resolve this and further work can be in other tasks or in WFLY-9588, or if you'd like to take this over I can assign to you.
Here's are the remaining cases where modules are depending on org.jboss.as.transactions:
{code}
$ git grep org.jboss.as.transactions | grep module.xml
feature-pack/src/main/resources/modules/system/layers/base/javax/orb/api/main/module.xml: <module name="org.jboss.as.transactions" optional="true"/>
feature-pack/src/main/resources/modules/system/layers/base/org/jboss/as/connector/main/module.xml: <module name="org.jboss.as.transactions"/>
feature-pack/src/main/resources/modules/system/layers/base/org/jboss/as/ejb3/main/module.xml: <module name="org.jboss.as.transactions"/>
feature-pack/src/main/resources/modules/system/layers/base/org/jboss/as/jpa/main/module.xml: <module name="org.jboss.as.transactions"/>
feature-pack/src/main/resources/modules/system/layers/base/org/jboss/as/transactions/main/module.xml:<module xmlns="urn:jboss:module:1.5" name="org.jboss.as.transactions">
feature-pack/src/main/resources/modules/system/layers/base/org/jboss/as/xts/main/module.xml: <module name="org.jboss.as.transactions"/>
feature-pack/src/main/resources/modules/system/layers/base/org/jboss/jts/integration/main/module.xml: <module name="org.jboss.as.transactions">
feature-pack/src/main/resources/modules/system/layers/base/org/jboss/jts/main/module.xml: <module name="org.jboss.as.transactions">
{code}
1) javax.orb.api is configured by the iiop subsystem to load classes from org.jboss.as.txn. Changing that would require refactoring the 'transactions' maven module to split those classes from the subsystem. Possible, if you think it's worth it; sounds like the kind of thing you are thinking about in WFLY-9588.
2) org.jboss.as.connector is importing a couple ServiceName constants; those should be capabilities a la WFLY-11167. There's also use of JBossContextXATerminator, but I suspect that could be converted to just JBossXATerminator.
3) org.jboss.as.ejb3 is using some service name constants that should be capabilities a la WFLY-11167, but it's also using UserTransactionAccessControlService and UserTransactionAccessControl.
4) org.jboss.as.jpa is using UserTransactionRegistryService.SERVICE_NAME. This should be a capability a la WFLY-11167.
5) org.jboss.as.xts uses org.jboss.as.txn.deployment.TransactionRollbackSetupAction
Of these, 2) and 4) sound like the kinds of things I talked about in the issue description. The others are cases where classes from the module really do need to be used, unless the module is refactored.
> Eliminate unessential module dependencies on the org.jboss.as.transactions module
> ---------------------------------------------------------------------------------
>
> Key: WFLY-11165
> URL: https://issues.jboss.org/browse/WFLY-11165
> Project: WildFly
> Issue Type: Enhancement
> Components: Class Loading, Management, Transactions
> Reporter: Brian Stansberry
> Assignee: Brian Stansberry
> Priority: Major
>
> As part of our work on getting maximum benefit from Galleon, we want to eliminate unnecessary hard dependencies between Galleon packages, most of which are just provisioning wrappers around the FS content for our JBoss Modules modules. So that means cutting dependency links between modules.
> The org.jboss.as.transactions module necessarily has a large dependency tree, so if other modules depend on it, they then also have a large dependency tree, perhaps much larger than whatever they really need. So I want to eliminate such dependencies whereever possible.
> There are quite a few modules that depend on org.jboss.as.transactions solely for ServiceName constants for services that provide value types not from org.jboss.as.transactions, e.g. TransactionManager which is from the javax.transaction.api module. So the main (perhaps only) task here will be to eliminate those links.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 6 months
[JBoss JIRA] (WFLY-11323) Fixing flaky test WritableServiceBasedNamingStoreTestCase.testPermissions
by ORD Testers (Jira)
[ https://issues.jboss.org/browse/WFLY-11323?page=com.atlassian.jira.plugin... ]
ORD Testers updated WFLY-11323:
-------------------------------
Description:
testPermissions fails when run by itself even when there’s no bug in the code.
This test can be fixed by looking up the name before testActionWithPermission is called, as shown in the following pull request.
https://github.com/wildfly/wildfly/pull/11833
Let me know if you want to discuss more.
was:
testPermissions fails when run by itself even when there’s no bug in the code.
This test can be fixed by looking up the name before testActionWithPermission is called, as shown in the following pull request.
(*to be replaced with a link to pull request when a pull request is made*)
Let me know if you want to discuss more.
> Fixing flaky test WritableServiceBasedNamingStoreTestCase.testPermissions
> -------------------------------------------------------------------------
>
> Key: WFLY-11323
> URL: https://issues.jboss.org/browse/WFLY-11323
> Project: WildFly
> Issue Type: Bug
> Components: Naming
> Environment: Any
> Reporter: ORD Testers
> Priority: Minor
>
> testPermissions fails when run by itself even when there’s no bug in the code.
> This test can be fixed by looking up the name before testActionWithPermission is called, as shown in the following pull request.
> https://github.com/wildfly/wildfly/pull/11833
> Let me know if you want to discuss more.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 6 months
[JBoss JIRA] (WFLY-11323) Fixing flaky test WritableServiceBasedNamingStoreTestCase.testPermissions
by ORD Testers (Jira)
ORD Testers created WFLY-11323:
----------------------------------
Summary: Fixing flaky test WritableServiceBasedNamingStoreTestCase.testPermissions
Key: WFLY-11323
URL: https://issues.jboss.org/browse/WFLY-11323
Project: WildFly
Issue Type: Bug
Components: Naming
Environment: Any
Reporter: ORD Testers
testPermissions fails when run by itself even when there’s no bug in the code.
This test can be fixed by looking up the name before testActionWithPermission is called, as shown in the following pull request.
(*to be replaced with a link to pull request when a pull request is made*)
Let me know if you want to discuss more.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 6 months