[JBoss JIRA] (DROOLS-3255) Contextual menu updates
by Gabriele Cardosi (Jira)
[ https://issues.jboss.org/browse/DROOLS-3255?page=com.atlassian.jira.plugi... ]
Gabriele Cardosi reassigned DROOLS-3255:
----------------------------------------
Assignee: Gabriele Cardosi (was: Daniele Zonca)
> Contextual menu updates
> -----------------------
>
> Key: DROOLS-3255
> URL: https://issues.jboss.org/browse/DROOLS-3255
> Project: Drools
> Issue Type: Sub-task
> Components: Scenario Simulation and Testing
> Reporter: Daniele Zonca
> Assignee: Gabriele Cardosi
> Priority: Major
> Labels: ScenarioSimulation
>
> - Contextual menus: Remove “Scenario: Insert top/bottom row.”
> - Contextual menus in headers: Remove "Scenario: Insert top/bottom row” and replace those options with only Scenario: “Insert row below”
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 6 months
[JBoss JIRA] (WFLY-11292) Legacy EJB Client: High fail rate
by Richard Achmatowicz (Jira)
[ https://issues.jboss.org/browse/WFLY-11292?page=com.atlassian.jira.plugin... ]
Richard Achmatowicz edited comment on WFLY-11292 at 11/8/18 11:24 AM:
----------------------------------------------------------------------
I've had a quick look at the logs. What I suspected above is not happening, so we can forget about that. What I did see is a case of a channel being opened, a topology update arriving, the channel being closed, and then an invocation attempt on that channel (or at least that's what it looks like to me):
{code}
----------------------
2018-11-08 15:51:55,213 INFO o.j.e.c.remoting: EJBCLIENT000017: Received server version 3 and marshalling strategies [river]
2018-11-08 15:51:55,213 INFO o.j.e.c.remoting: EJBCLIENT000013: Successful version handshake completed for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@201671ea, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)1d986363,channel=jboss.ejb,nodename=wildfly1]} on channel Channel ID f91c460d (outbound) of Remoting connection 7e6a4bd2 to /172.16.140.12:8080 of endpoint "config-based-ejb-client-endpoint" <36a26cd3>
---------------------
=== 1,3,4
2018-11-08 15:51:55,214 DEBUG o.j.e.c.r.ClusterTopologyMessageHandler: Received a cluster node(s) addition message, for cluster named ejb with 3 nodes [ClusterNode{clusterName='ejb', nodeName='wildfly3', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='172.16.140.5', destinationPort=8080}], resolvedDestination=[Destination address=172.16.140.5, destination port=8080]}, ClusterNode{clusterName='ejb', nodeName='wildfly4', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='172.16.140.10', destinationPort=8080}], resolvedDestination=[Destination address=172.16.140.10, destination port=8080]}, ClusterNode{clusterName='ejb', nodeName='wildfly1', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='172.16.140.12', destinationPort=8080}], resolvedDestination=[Destination address=172.16.140.12, destination port=8080]}]
--------------------
2018-11-08 15:51:55,220 INFO o.j.e.c.remoting: EJBCLIENT000016: Channel Channel ID f91c460d (outbound) of Remoting connection 7e6a4bd2 to /172.16.140.12:8080 of endpoint "config-based-ejb-client-endpoint" <36a26cd3> can no longer process messages
--------------------
2018-11-08 15:51:55,221 INFO o.j.e.c.remoting: EJBCLIENT000017: Received server version 3 and marshalling strategies [river]
2018-11-08 15:51:55,221 INFO o.j.e.c.remoting: EJBCLIENT000013: Successful version handshake completed for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@201671ea, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)1d986363,channel=jboss.ejb,nodename=wildfly1]} on channel Channel ID 819e63e9 (outbound) of Remoting connection 7e6a4bd2 to /172.16.140.12:8080 of endpoint "config-based-ejb-client-endpoint" <36a26cd3>
=== 1,3,4
2018-11-08 15:51:55,222 DEBUG o.j.e.c.r.ClusterTopologyMessageHandler: Received a cluster node(s) addition message, for cluster named ejb with 3 nodes [ClusterNode{clusterName='ejb', nodeName='wildfly3', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='172.16.140.5', destinationPort=8080}], resolvedDestination=[Destination address=172.16.140.5, destination port=8080]}, ClusterNode{clusterName='ejb', nodeName='wildfly4', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='172.16.140.10', destinationPort=8080}], resolvedDestination=[Destination address=172.16.140.10, destination port=8080]}, ClusterNode{clusterName='ejb', nodeName='wildfly1', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='172.16.140.12', destinationPort=8080}], resolvedDestination=[Destination address=172.16.140.12, destination port=8080]}]
=========
2018-11-08 15:51:55,221 ERROR o.j.e.c.j.ClusteringEJBLegacyRequestSampler: Thread Group 1-671: Serial 50: Error getting response.
javax.ejb.EJBException: java.io.IOException: Channel Channel ID f91c460d (outbound) of Remoting connection 7e6a4bd2 to /172.16.140.12:8080 of endpoint "config-based-ejb-client-endpoint" <36a26cd3> has been closed
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:240) ~[jboss-ejb-client-legacy-3.0.2.Final-redhat-1.jar:3.0.2.Final-redhat-1]
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:185) ~[jboss-ejb-client-legacy-3.0.2.Final-redhat-1.jar:3.0.2.Final-redhat-1]
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:148) ~[jboss-ejb-client-legacy-3.0.2.Final-redhat-1.jar:3.0.2.Final-redhat-1]
at com.sun.proxy.$Proxy21.getSerialAndIncrement(Unknown Source) ~[?:?]
at org.jboss.eapqe.clustering.jmeter.ClusteringEJBLegacyRequestSampler.runTest(ClusteringEJBLegacyRequestSampler.java:126) [clustering-jmeter-samplers.jar:?]
at org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSampler.java:197) [ApacheJMeter_java.jar:5.0 r1840935]
{code}
Need to find out why these channels are being closed.
was (Author: rachmato):
I've had a quick look at the logs. What I suspected above is not happening, so we can forget about that. What I did see is a case of a channel being opened, a topology update arriving, the channel being closed, and then an invocation attempt on that channel (or at least that's what it looks like to me):
{code}
----------------------
2018-11-08 15:51:55,213 INFO o.j.e.c.remoting: EJBCLIENT000017: Received server version 3 and marshalling strategies [river]
2018-11-08 15:51:55,213 INFO o.j.e.c.remoting: EJBCLIENT000013: Successful version handshake completed for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@201671ea, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)1d986363,channel=jboss.ejb,nodename=wildfly1]} on channel Channel ID f91c460d (outbound) of Remoting connection 7e6a4bd2 to /172.16.140.12:8080 of endpoint "config-based-ejb-client-endpoint" <36a26cd3>
---------------------
=== 1,3,4
2018-11-08 15:51:55,214 DEBUG o.j.e.c.r.ClusterTopologyMessageHandler: Received a cluster node(s) addition message, for cluster named ejb with 3 nodes [ClusterNode{clusterName='ejb', nodeName='wildfly3', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='172.16.140.5', destinationPort=8080}], resolvedDestination=[Destination address=172.16.140.5, destination port=8080]}, ClusterNode{clusterName='ejb', nodeName='wildfly4', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='172.16.140.10', destinationPort=8080}], resolvedDestination=[Destination address=172.16.140.10, destination port=8080]}, ClusterNode{clusterName='ejb', nodeName='wildfly1', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='172.16.140.12', destinationPort=8080}], resolvedDestination=[Destination address=172.16.140.12, destination port=8080]}]
--------------------
2018-11-08 15:51:55,220 INFO o.j.e.c.remoting: EJBCLIENT000016: Channel Channel ID f91c460d (outbound) of Remoting connection 7e6a4bd2 to /172.16.140.12:8080 of endpoint "config-based-ejb-client-endpoint" <36a26cd3> can no longer process messages
--------------------
2018-11-08 15:51:55,221 INFO o.j.e.c.remoting: EJBCLIENT000017: Received server version 3 and marshalling strategies [river]
2018-11-08 15:51:55,221 INFO o.j.e.c.remoting: EJBCLIENT000013: Successful version handshake completed for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@201671ea, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)1d986363,channel=jboss.ejb,nodename=wildfly1]} on channel Channel ID 819e63e9 (outbound) of Remoting connection 7e6a4bd2 to /172.16.140.12:8080 of endpoint "config-based-ejb-client-endpoint" <36a26cd3>
=== 1,3,4
2018-11-08 15:51:55,222 DEBUG o.j.e.c.r.ClusterTopologyMessageHandler: Received a cluster node(s) addition message, for cluster named ejb with 3 nodes [ClusterNode{clusterName='ejb', nodeName='wildfly3', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='172.16.140.5', destinationPort=8080}], resolvedDestination=[Destination address=172.16.140.5, destination port=8080]}, ClusterNode{clusterName='ejb', nodeName='wildfly4', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='172.16.140.10', destinationPort=8080}], resolvedDestination=[Destination address=172.16.140.10, destination port=8080]}, ClusterNode{clusterName='ejb', nodeName='wildfly1', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='172.16.140.12', destinationPort=8080}], resolvedDestination=[Destination address=172.16.140.12, destination port=8080]}]
=========
2018-11-08 15:51:55,221 ERROR o.j.e.c.j.ClusteringEJBLegacyRequestSampler: Thread Group 1-671: Serial 50: Error getting response.
javax.ejb.EJBException: java.io.IOException: Channel Channel ID f91c460d (outbound) of Remoting connection 7e6a4bd2 to /172.16.140.12:8080 of endpoint "config-based-ejb-client-endpoint" <36a26cd3> has been closed
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:240) ~[jboss-ejb-client-legacy-3.0.2.Final-redhat-1.jar:3.0.2.Final-redhat-1]
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:185) ~[jboss-ejb-client-legacy-3.0.2.Final-redhat-1.jar:3.0.2.Final-redhat-1]
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:148) ~[jboss-ejb-client-legacy-3.0.2.Final-redhat-1.jar:3.0.2.Final-redhat-1]
at com.sun.proxy.$Proxy21.getSerialAndIncrement(Unknown Source) ~[?:?]
at org.jboss.eapqe.clustering.jmeter.ClusteringEJBLegacyRequestSampler.runTest(ClusteringEJBLegacyRequestSampler.java:126) [clustering-jmeter-samplers.jar:?]
at org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSampler.java:197) [ApacheJMeter_java.jar:5.0 r1840935]
{code}
> Legacy EJB Client: High fail rate
> ---------------------------------
>
> Key: WFLY-11292
> URL: https://issues.jboss.org/browse/WFLY-11292
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, EJB
> Affects Versions: 15.0.0.Alpha1
> Reporter: tommaso borgato
> Assignee: Flavia Rainone
> Priority: Blocker
> Attachments: jboss-ejb-client.properties, logs.zip
>
>
> This bug is being filed as Blocker because we are observing and elevated fail rate: roughly a thousand each run for (about 0.3%).
> h2. WildFly Built from master branch on 6 Nov 2018
> With this WildFly version (client org.jboss:jboss-ejb-client-legacy:3.0.2.Final-redhat-1) in a scenario with 4 clustered nodes where nodes are failed via jboss shut-down / restart: after node 1 of 4 is shut-down, a a series of errors start on the client side the yield to 1097 errors on a total of 340218 samples;
> find [here|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/...] the complete logs;
> the start of client errors is here:
> {noformat}
> 2018/11/07 05:28:51:255 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failing node 0 (perf18)
> 2018/11/07 05:28:51:270 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Total: Sessions: 2000, active: 2000, samples: 5118, throughput: 511.7 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 1 ms, max: 54 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5118 (100%)
> 2018/11/07 05:28:51:270 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - unknown-node: Sessions: 2000, active: 2000, samples: 5118, throughput: 511.7 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 1 ms, max: 54 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5118 (100%)
> 05:28:53,257 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-2) EJBCLIENT000016: Channel Channel ID c540daf7 (outbound) of Remoting connection 4ed5523c to perf18/10.16.90.54:8080 of endpoint "config-based-ejb-client-endpoint" <664ac6d5> can no longer process messages
> 05:28:53,277 ERROR [org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver] (Remoting "config-based-ejb-client-endpoint" task-2) Failed to open channel for context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@3cd51445, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)302066a5,channel=jboss.ejb,nodename=perf18]}
> org.jboss.remoting3.NotOpenException: Cannot open new channel because close was initiated
> at org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen(RemoteConnectionHandler.java:198)
> at org.jboss.remoting3.remote.RemoteConnectionHandler.open(RemoteConnectionHandler.java:335)
> at org.jboss.remoting3.ConnectionImpl.openChannel(ConnectionImpl.java:109)
> at org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection.openChannel(ConnectionPool.java:292)
> at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.associate(RemotingConnectionEJBReceiver.java:180)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:399)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:349)
> at org.jboss.ejb.client.remoting.EJBClientContextConnectionReconnectHandler.reconnect(EJBClientContextConnectionReconnectHandler.java:67)
> at org.jboss.ejb.client.EJBClientContext$ReconnectAttempt.run(EJBClientContext.java:1474)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> h2. WildFly Built from master branch on 7 Nov 2018
> Same situation.
> find [here|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/...] the complete logs;
> the start of client errors is here:
> {noformat}
> 2018/11/07 09:23:22:916 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failing node 0 (perf18)
> 2018/11/07 09:23:22:930 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Total: Sessions: 2000, active: 2000, samples: 5125, throughput: 512.4 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 2 ms, max: 32 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5125 (100%)
> 2018/11/07 09:23:22:930 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - unknown-node: Sessions: 2000, active: 2000, samples: 5125, throughput: 512.4 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 2 ms, max: 32 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5125 (100%)
> 09:23:24,924 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-4) EJBCLIENT000016: Channel Channel ID d940c707 (outbound) of Remoting connection 73cfdf01 to perf18/10.16.90.54:8080 of endpoint "config-based-ejb-client-endpoint" <7501616b> can no longer process messages
> 09:23:24,949 ERROR [org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver] (Remoting "config-based-ejb-client-endpoint" task-6) Failed to open channel for context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@56248025, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)5269f48d,channel=jboss.ejb,nodename=perf18]}
> org.jboss.remoting3.NotOpenException: Cannot open new channel because close was initiated
> at org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen(RemoteConnectionHandler.java:198)
> at org.jboss.remoting3.remote.RemoteConnectionHandler.open(RemoteConnectionHandler.java:335)
> at org.jboss.remoting3.ConnectionImpl.openChannel(ConnectionImpl.java:109)
> at org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection.openChannel(ConnectionPool.java:292)
> at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.associate(RemotingConnectionEJBReceiver.java:180)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:399)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:349)
> at org.jboss.ejb.client.remoting.EJBClientContextConnectionReconnectHandler.reconnect(EJBClientContextConnectionReconnectHandler.java:67)
> at org.jboss.ejb.client.EJBClientContext$ReconnectAttempt.run(EJBClientContext.java:1474)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 6 months
[JBoss JIRA] (WFLY-11292) Legacy EJB Client: High fail rate
by Richard Achmatowicz (Jira)
[ https://issues.jboss.org/browse/WFLY-11292?page=com.atlassian.jira.plugin... ]
Richard Achmatowicz commented on WFLY-11292:
--------------------------------------------
I've had a quick look at the logs. What I suspected above is not happening, so we can forget about that. What I did see is a case of a channel being opened, a topology update arriving, the channel being closed, and then an invocation attempt on that channel (or at least that's what it looks like to me):
{code}
----------------------
2018-11-08 15:51:55,213 INFO o.j.e.c.remoting: EJBCLIENT000017: Received server version 3 and marshalling strategies [river]
2018-11-08 15:51:55,213 INFO o.j.e.c.remoting: EJBCLIENT000013: Successful version handshake completed for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@201671ea, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)1d986363,channel=jboss.ejb,nodename=wildfly1]} on channel Channel ID f91c460d (outbound) of Remoting connection 7e6a4bd2 to /172.16.140.12:8080 of endpoint "config-based-ejb-client-endpoint" <36a26cd3>
---------------------
=== 1,3,4
2018-11-08 15:51:55,214 DEBUG o.j.e.c.r.ClusterTopologyMessageHandler: Received a cluster node(s) addition message, for cluster named ejb with 3 nodes [ClusterNode{clusterName='ejb', nodeName='wildfly3', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='172.16.140.5', destinationPort=8080}], resolvedDestination=[Destination address=172.16.140.5, destination port=8080]}, ClusterNode{clusterName='ejb', nodeName='wildfly4', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='172.16.140.10', destinationPort=8080}], resolvedDestination=[Destination address=172.16.140.10, destination port=8080]}, ClusterNode{clusterName='ejb', nodeName='wildfly1', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='172.16.140.12', destinationPort=8080}], resolvedDestination=[Destination address=172.16.140.12, destination port=8080]}]
--------------------
2018-11-08 15:51:55,220 INFO o.j.e.c.remoting: EJBCLIENT000016: Channel Channel ID f91c460d (outbound) of Remoting connection 7e6a4bd2 to /172.16.140.12:8080 of endpoint "config-based-ejb-client-endpoint" <36a26cd3> can no longer process messages
--------------------
2018-11-08 15:51:55,221 INFO o.j.e.c.remoting: EJBCLIENT000017: Received server version 3 and marshalling strategies [river]
2018-11-08 15:51:55,221 INFO o.j.e.c.remoting: EJBCLIENT000013: Successful version handshake completed for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@201671ea, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)1d986363,channel=jboss.ejb,nodename=wildfly1]} on channel Channel ID 819e63e9 (outbound) of Remoting connection 7e6a4bd2 to /172.16.140.12:8080 of endpoint "config-based-ejb-client-endpoint" <36a26cd3>
=== 1,3,4
2018-11-08 15:51:55,222 DEBUG o.j.e.c.r.ClusterTopologyMessageHandler: Received a cluster node(s) addition message, for cluster named ejb with 3 nodes [ClusterNode{clusterName='ejb', nodeName='wildfly3', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='172.16.140.5', destinationPort=8080}], resolvedDestination=[Destination address=172.16.140.5, destination port=8080]}, ClusterNode{clusterName='ejb', nodeName='wildfly4', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='172.16.140.10', destinationPort=8080}], resolvedDestination=[Destination address=172.16.140.10, destination port=8080]}, ClusterNode{clusterName='ejb', nodeName='wildfly1', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='172.16.140.12', destinationPort=8080}], resolvedDestination=[Destination address=172.16.140.12, destination port=8080]}]
=========
2018-11-08 15:51:55,221 ERROR o.j.e.c.j.ClusteringEJBLegacyRequestSampler: Thread Group 1-671: Serial 50: Error getting response.
javax.ejb.EJBException: java.io.IOException: Channel Channel ID f91c460d (outbound) of Remoting connection 7e6a4bd2 to /172.16.140.12:8080 of endpoint "config-based-ejb-client-endpoint" <36a26cd3> has been closed
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:240) ~[jboss-ejb-client-legacy-3.0.2.Final-redhat-1.jar:3.0.2.Final-redhat-1]
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:185) ~[jboss-ejb-client-legacy-3.0.2.Final-redhat-1.jar:3.0.2.Final-redhat-1]
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:148) ~[jboss-ejb-client-legacy-3.0.2.Final-redhat-1.jar:3.0.2.Final-redhat-1]
at com.sun.proxy.$Proxy21.getSerialAndIncrement(Unknown Source) ~[?:?]
at org.jboss.eapqe.clustering.jmeter.ClusteringEJBLegacyRequestSampler.runTest(ClusteringEJBLegacyRequestSampler.java:126) [clustering-jmeter-samplers.jar:?]
at org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSampler.java:197) [ApacheJMeter_java.jar:5.0 r1840935]
{code}
> Legacy EJB Client: High fail rate
> ---------------------------------
>
> Key: WFLY-11292
> URL: https://issues.jboss.org/browse/WFLY-11292
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, EJB
> Affects Versions: 15.0.0.Alpha1
> Reporter: tommaso borgato
> Assignee: Flavia Rainone
> Priority: Blocker
> Attachments: jboss-ejb-client.properties, logs.zip
>
>
> This bug is being filed as Blocker because we are observing and elevated fail rate: roughly a thousand each run for (about 0.3%).
> h2. WildFly Built from master branch on 6 Nov 2018
> With this WildFly version (client org.jboss:jboss-ejb-client-legacy:3.0.2.Final-redhat-1) in a scenario with 4 clustered nodes where nodes are failed via jboss shut-down / restart: after node 1 of 4 is shut-down, a a series of errors start on the client side the yield to 1097 errors on a total of 340218 samples;
> find [here|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/...] the complete logs;
> the start of client errors is here:
> {noformat}
> 2018/11/07 05:28:51:255 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failing node 0 (perf18)
> 2018/11/07 05:28:51:270 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Total: Sessions: 2000, active: 2000, samples: 5118, throughput: 511.7 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 1 ms, max: 54 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5118 (100%)
> 2018/11/07 05:28:51:270 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - unknown-node: Sessions: 2000, active: 2000, samples: 5118, throughput: 511.7 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 1 ms, max: 54 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5118 (100%)
> 05:28:53,257 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-2) EJBCLIENT000016: Channel Channel ID c540daf7 (outbound) of Remoting connection 4ed5523c to perf18/10.16.90.54:8080 of endpoint "config-based-ejb-client-endpoint" <664ac6d5> can no longer process messages
> 05:28:53,277 ERROR [org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver] (Remoting "config-based-ejb-client-endpoint" task-2) Failed to open channel for context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@3cd51445, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)302066a5,channel=jboss.ejb,nodename=perf18]}
> org.jboss.remoting3.NotOpenException: Cannot open new channel because close was initiated
> at org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen(RemoteConnectionHandler.java:198)
> at org.jboss.remoting3.remote.RemoteConnectionHandler.open(RemoteConnectionHandler.java:335)
> at org.jboss.remoting3.ConnectionImpl.openChannel(ConnectionImpl.java:109)
> at org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection.openChannel(ConnectionPool.java:292)
> at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.associate(RemotingConnectionEJBReceiver.java:180)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:399)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:349)
> at org.jboss.ejb.client.remoting.EJBClientContextConnectionReconnectHandler.reconnect(EJBClientContextConnectionReconnectHandler.java:67)
> at org.jboss.ejb.client.EJBClientContext$ReconnectAttempt.run(EJBClientContext.java:1474)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> h2. WildFly Built from master branch on 7 Nov 2018
> Same situation.
> find [here|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/...] the complete logs;
> the start of client errors is here:
> {noformat}
> 2018/11/07 09:23:22:916 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failing node 0 (perf18)
> 2018/11/07 09:23:22:930 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Total: Sessions: 2000, active: 2000, samples: 5125, throughput: 512.4 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 2 ms, max: 32 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5125 (100%)
> 2018/11/07 09:23:22:930 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - unknown-node: Sessions: 2000, active: 2000, samples: 5125, throughput: 512.4 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 2 ms, max: 32 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5125 (100%)
> 09:23:24,924 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-4) EJBCLIENT000016: Channel Channel ID d940c707 (outbound) of Remoting connection 73cfdf01 to perf18/10.16.90.54:8080 of endpoint "config-based-ejb-client-endpoint" <7501616b> can no longer process messages
> 09:23:24,949 ERROR [org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver] (Remoting "config-based-ejb-client-endpoint" task-6) Failed to open channel for context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@56248025, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)5269f48d,channel=jboss.ejb,nodename=perf18]}
> org.jboss.remoting3.NotOpenException: Cannot open new channel because close was initiated
> at org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen(RemoteConnectionHandler.java:198)
> at org.jboss.remoting3.remote.RemoteConnectionHandler.open(RemoteConnectionHandler.java:335)
> at org.jboss.remoting3.ConnectionImpl.openChannel(ConnectionImpl.java:109)
> at org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection.openChannel(ConnectionPool.java:292)
> at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.associate(RemotingConnectionEJBReceiver.java:180)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:399)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:349)
> at org.jboss.ejb.client.remoting.EJBClientContextConnectionReconnectHandler.reconnect(EJBClientContextConnectionReconnectHandler.java:67)
> at org.jboss.ejb.client.EJBClientContext$ReconnectAttempt.run(EJBClientContext.java:1474)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 6 months
[JBoss JIRA] (WFLY-11292) Legacy EJB Client: High fail rate
by tommaso borgato (Jira)
[ https://issues.jboss.org/browse/WFLY-11292?page=com.atlassian.jira.plugin... ]
tommaso borgato edited comment on WFLY-11292 at 11/8/18 11:11 AM:
------------------------------------------------------------------
[~rachmato] I attached a file with all the logs of the execution ({{logs.zip}}) we were looking at; let me know if you need more runs / logs; I also attached the {{jboss-ejb-client.properties}} file used on the client side;
was (Author: tommaso-borgato):
[~rachmato] I attached a file with all the logs of the execution we were looking at; let me know if you need more runs / logs.
> Legacy EJB Client: High fail rate
> ---------------------------------
>
> Key: WFLY-11292
> URL: https://issues.jboss.org/browse/WFLY-11292
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, EJB
> Affects Versions: 15.0.0.Alpha1
> Reporter: tommaso borgato
> Assignee: Flavia Rainone
> Priority: Blocker
> Attachments: jboss-ejb-client.properties, logs.zip
>
>
> This bug is being filed as Blocker because we are observing and elevated fail rate: roughly a thousand each run for (about 0.3%).
> h2. WildFly Built from master branch on 6 Nov 2018
> With this WildFly version (client org.jboss:jboss-ejb-client-legacy:3.0.2.Final-redhat-1) in a scenario with 4 clustered nodes where nodes are failed via jboss shut-down / restart: after node 1 of 4 is shut-down, a a series of errors start on the client side the yield to 1097 errors on a total of 340218 samples;
> find [here|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/...] the complete logs;
> the start of client errors is here:
> {noformat}
> 2018/11/07 05:28:51:255 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failing node 0 (perf18)
> 2018/11/07 05:28:51:270 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Total: Sessions: 2000, active: 2000, samples: 5118, throughput: 511.7 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 1 ms, max: 54 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5118 (100%)
> 2018/11/07 05:28:51:270 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - unknown-node: Sessions: 2000, active: 2000, samples: 5118, throughput: 511.7 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 1 ms, max: 54 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5118 (100%)
> 05:28:53,257 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-2) EJBCLIENT000016: Channel Channel ID c540daf7 (outbound) of Remoting connection 4ed5523c to perf18/10.16.90.54:8080 of endpoint "config-based-ejb-client-endpoint" <664ac6d5> can no longer process messages
> 05:28:53,277 ERROR [org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver] (Remoting "config-based-ejb-client-endpoint" task-2) Failed to open channel for context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@3cd51445, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)302066a5,channel=jboss.ejb,nodename=perf18]}
> org.jboss.remoting3.NotOpenException: Cannot open new channel because close was initiated
> at org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen(RemoteConnectionHandler.java:198)
> at org.jboss.remoting3.remote.RemoteConnectionHandler.open(RemoteConnectionHandler.java:335)
> at org.jboss.remoting3.ConnectionImpl.openChannel(ConnectionImpl.java:109)
> at org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection.openChannel(ConnectionPool.java:292)
> at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.associate(RemotingConnectionEJBReceiver.java:180)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:399)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:349)
> at org.jboss.ejb.client.remoting.EJBClientContextConnectionReconnectHandler.reconnect(EJBClientContextConnectionReconnectHandler.java:67)
> at org.jboss.ejb.client.EJBClientContext$ReconnectAttempt.run(EJBClientContext.java:1474)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> h2. WildFly Built from master branch on 7 Nov 2018
> Same situation.
> find [here|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/...] the complete logs;
> the start of client errors is here:
> {noformat}
> 2018/11/07 09:23:22:916 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failing node 0 (perf18)
> 2018/11/07 09:23:22:930 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Total: Sessions: 2000, active: 2000, samples: 5125, throughput: 512.4 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 2 ms, max: 32 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5125 (100%)
> 2018/11/07 09:23:22:930 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - unknown-node: Sessions: 2000, active: 2000, samples: 5125, throughput: 512.4 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 2 ms, max: 32 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5125 (100%)
> 09:23:24,924 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-4) EJBCLIENT000016: Channel Channel ID d940c707 (outbound) of Remoting connection 73cfdf01 to perf18/10.16.90.54:8080 of endpoint "config-based-ejb-client-endpoint" <7501616b> can no longer process messages
> 09:23:24,949 ERROR [org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver] (Remoting "config-based-ejb-client-endpoint" task-6) Failed to open channel for context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@56248025, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)5269f48d,channel=jboss.ejb,nodename=perf18]}
> org.jboss.remoting3.NotOpenException: Cannot open new channel because close was initiated
> at org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen(RemoteConnectionHandler.java:198)
> at org.jboss.remoting3.remote.RemoteConnectionHandler.open(RemoteConnectionHandler.java:335)
> at org.jboss.remoting3.ConnectionImpl.openChannel(ConnectionImpl.java:109)
> at org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection.openChannel(ConnectionPool.java:292)
> at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.associate(RemotingConnectionEJBReceiver.java:180)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:399)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:349)
> at org.jboss.ejb.client.remoting.EJBClientContextConnectionReconnectHandler.reconnect(EJBClientContextConnectionReconnectHandler.java:67)
> at org.jboss.ejb.client.EJBClientContext$ReconnectAttempt.run(EJBClientContext.java:1474)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 6 months
[JBoss JIRA] (WFLY-11292) Legacy EJB Client: High fail rate
by tommaso borgato (Jira)
[ https://issues.jboss.org/browse/WFLY-11292?page=com.atlassian.jira.plugin... ]
tommaso borgato commented on WFLY-11292:
----------------------------------------
[~rachmato] I attached a file with all the logs of the execution we were looking at; let me know if you need more runs / logs.
> Legacy EJB Client: High fail rate
> ---------------------------------
>
> Key: WFLY-11292
> URL: https://issues.jboss.org/browse/WFLY-11292
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, EJB
> Affects Versions: 15.0.0.Alpha1
> Reporter: tommaso borgato
> Assignee: Flavia Rainone
> Priority: Blocker
> Attachments: jboss-ejb-client.properties, logs.zip
>
>
> This bug is being filed as Blocker because we are observing and elevated fail rate: roughly a thousand each run for (about 0.3%).
> h2. WildFly Built from master branch on 6 Nov 2018
> With this WildFly version (client org.jboss:jboss-ejb-client-legacy:3.0.2.Final-redhat-1) in a scenario with 4 clustered nodes where nodes are failed via jboss shut-down / restart: after node 1 of 4 is shut-down, a a series of errors start on the client side the yield to 1097 errors on a total of 340218 samples;
> find [here|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/...] the complete logs;
> the start of client errors is here:
> {noformat}
> 2018/11/07 05:28:51:255 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failing node 0 (perf18)
> 2018/11/07 05:28:51:270 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Total: Sessions: 2000, active: 2000, samples: 5118, throughput: 511.7 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 1 ms, max: 54 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5118 (100%)
> 2018/11/07 05:28:51:270 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - unknown-node: Sessions: 2000, active: 2000, samples: 5118, throughput: 511.7 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 1 ms, max: 54 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5118 (100%)
> 05:28:53,257 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-2) EJBCLIENT000016: Channel Channel ID c540daf7 (outbound) of Remoting connection 4ed5523c to perf18/10.16.90.54:8080 of endpoint "config-based-ejb-client-endpoint" <664ac6d5> can no longer process messages
> 05:28:53,277 ERROR [org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver] (Remoting "config-based-ejb-client-endpoint" task-2) Failed to open channel for context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@3cd51445, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)302066a5,channel=jboss.ejb,nodename=perf18]}
> org.jboss.remoting3.NotOpenException: Cannot open new channel because close was initiated
> at org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen(RemoteConnectionHandler.java:198)
> at org.jboss.remoting3.remote.RemoteConnectionHandler.open(RemoteConnectionHandler.java:335)
> at org.jboss.remoting3.ConnectionImpl.openChannel(ConnectionImpl.java:109)
> at org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection.openChannel(ConnectionPool.java:292)
> at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.associate(RemotingConnectionEJBReceiver.java:180)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:399)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:349)
> at org.jboss.ejb.client.remoting.EJBClientContextConnectionReconnectHandler.reconnect(EJBClientContextConnectionReconnectHandler.java:67)
> at org.jboss.ejb.client.EJBClientContext$ReconnectAttempt.run(EJBClientContext.java:1474)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> h2. WildFly Built from master branch on 7 Nov 2018
> Same situation.
> find [here|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/...] the complete logs;
> the start of client errors is here:
> {noformat}
> 2018/11/07 09:23:22:916 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failing node 0 (perf18)
> 2018/11/07 09:23:22:930 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Total: Sessions: 2000, active: 2000, samples: 5125, throughput: 512.4 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 2 ms, max: 32 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5125 (100%)
> 2018/11/07 09:23:22:930 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - unknown-node: Sessions: 2000, active: 2000, samples: 5125, throughput: 512.4 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 2 ms, max: 32 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5125 (100%)
> 09:23:24,924 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-4) EJBCLIENT000016: Channel Channel ID d940c707 (outbound) of Remoting connection 73cfdf01 to perf18/10.16.90.54:8080 of endpoint "config-based-ejb-client-endpoint" <7501616b> can no longer process messages
> 09:23:24,949 ERROR [org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver] (Remoting "config-based-ejb-client-endpoint" task-6) Failed to open channel for context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@56248025, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)5269f48d,channel=jboss.ejb,nodename=perf18]}
> org.jboss.remoting3.NotOpenException: Cannot open new channel because close was initiated
> at org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen(RemoteConnectionHandler.java:198)
> at org.jboss.remoting3.remote.RemoteConnectionHandler.open(RemoteConnectionHandler.java:335)
> at org.jboss.remoting3.ConnectionImpl.openChannel(ConnectionImpl.java:109)
> at org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection.openChannel(ConnectionPool.java:292)
> at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.associate(RemotingConnectionEJBReceiver.java:180)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:399)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:349)
> at org.jboss.ejb.client.remoting.EJBClientContextConnectionReconnectHandler.reconnect(EJBClientContextConnectionReconnectHandler.java:67)
> at org.jboss.ejb.client.EJBClientContext$ReconnectAttempt.run(EJBClientContext.java:1474)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 6 months
[JBoss JIRA] (WFLY-11292) Legacy EJB Client: High fail rate
by tommaso borgato (Jira)
[ https://issues.jboss.org/browse/WFLY-11292?page=com.atlassian.jira.plugin... ]
tommaso borgato updated WFLY-11292:
-----------------------------------
Attachment: jboss-ejb-client.properties
> Legacy EJB Client: High fail rate
> ---------------------------------
>
> Key: WFLY-11292
> URL: https://issues.jboss.org/browse/WFLY-11292
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, EJB
> Affects Versions: 15.0.0.Alpha1
> Reporter: tommaso borgato
> Assignee: Flavia Rainone
> Priority: Blocker
> Attachments: jboss-ejb-client.properties, logs.zip
>
>
> This bug is being filed as Blocker because we are observing and elevated fail rate: roughly a thousand each run for (about 0.3%).
> h2. WildFly Built from master branch on 6 Nov 2018
> With this WildFly version (client org.jboss:jboss-ejb-client-legacy:3.0.2.Final-redhat-1) in a scenario with 4 clustered nodes where nodes are failed via jboss shut-down / restart: after node 1 of 4 is shut-down, a a series of errors start on the client side the yield to 1097 errors on a total of 340218 samples;
> find [here|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/...] the complete logs;
> the start of client errors is here:
> {noformat}
> 2018/11/07 05:28:51:255 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failing node 0 (perf18)
> 2018/11/07 05:28:51:270 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Total: Sessions: 2000, active: 2000, samples: 5118, throughput: 511.7 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 1 ms, max: 54 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5118 (100%)
> 2018/11/07 05:28:51:270 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - unknown-node: Sessions: 2000, active: 2000, samples: 5118, throughput: 511.7 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 1 ms, max: 54 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5118 (100%)
> 05:28:53,257 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-2) EJBCLIENT000016: Channel Channel ID c540daf7 (outbound) of Remoting connection 4ed5523c to perf18/10.16.90.54:8080 of endpoint "config-based-ejb-client-endpoint" <664ac6d5> can no longer process messages
> 05:28:53,277 ERROR [org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver] (Remoting "config-based-ejb-client-endpoint" task-2) Failed to open channel for context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@3cd51445, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)302066a5,channel=jboss.ejb,nodename=perf18]}
> org.jboss.remoting3.NotOpenException: Cannot open new channel because close was initiated
> at org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen(RemoteConnectionHandler.java:198)
> at org.jboss.remoting3.remote.RemoteConnectionHandler.open(RemoteConnectionHandler.java:335)
> at org.jboss.remoting3.ConnectionImpl.openChannel(ConnectionImpl.java:109)
> at org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection.openChannel(ConnectionPool.java:292)
> at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.associate(RemotingConnectionEJBReceiver.java:180)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:399)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:349)
> at org.jboss.ejb.client.remoting.EJBClientContextConnectionReconnectHandler.reconnect(EJBClientContextConnectionReconnectHandler.java:67)
> at org.jboss.ejb.client.EJBClientContext$ReconnectAttempt.run(EJBClientContext.java:1474)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> h2. WildFly Built from master branch on 7 Nov 2018
> Same situation.
> find [here|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/...] the complete logs;
> the start of client errors is here:
> {noformat}
> 2018/11/07 09:23:22:916 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failing node 0 (perf18)
> 2018/11/07 09:23:22:930 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Total: Sessions: 2000, active: 2000, samples: 5125, throughput: 512.4 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 2 ms, max: 32 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5125 (100%)
> 2018/11/07 09:23:22:930 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - unknown-node: Sessions: 2000, active: 2000, samples: 5125, throughput: 512.4 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 2 ms, max: 32 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5125 (100%)
> 09:23:24,924 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-4) EJBCLIENT000016: Channel Channel ID d940c707 (outbound) of Remoting connection 73cfdf01 to perf18/10.16.90.54:8080 of endpoint "config-based-ejb-client-endpoint" <7501616b> can no longer process messages
> 09:23:24,949 ERROR [org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver] (Remoting "config-based-ejb-client-endpoint" task-6) Failed to open channel for context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@56248025, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)5269f48d,channel=jboss.ejb,nodename=perf18]}
> org.jboss.remoting3.NotOpenException: Cannot open new channel because close was initiated
> at org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen(RemoteConnectionHandler.java:198)
> at org.jboss.remoting3.remote.RemoteConnectionHandler.open(RemoteConnectionHandler.java:335)
> at org.jboss.remoting3.ConnectionImpl.openChannel(ConnectionImpl.java:109)
> at org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection.openChannel(ConnectionPool.java:292)
> at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.associate(RemotingConnectionEJBReceiver.java:180)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:399)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:349)
> at org.jboss.ejb.client.remoting.EJBClientContextConnectionReconnectHandler.reconnect(EJBClientContextConnectionReconnectHandler.java:67)
> at org.jboss.ejb.client.EJBClientContext$ReconnectAttempt.run(EJBClientContext.java:1474)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 6 months