[JBoss JIRA] (WFLY-9364) Closing an EJBClientContext sometimes hangs causing high CPU usage
by Marius Tantareanu (JIRA)
[ https://issues.jboss.org/browse/WFLY-9364?page=com.atlassian.jira.plugin.... ]
Marius Tantareanu commented on WFLY-9364:
-----------------------------------------
I manually patched jboss-client.jar by replacing classes from jboss-remoting and xnio with the classes from jboss-remoting-4.0.24.Final.jar, xnio-api-3.4.6.Final.jar and xnio-nio-3.4.6.Final.jar. Then I ran the client again with the patched jboss-client.jar. Unfortunately I was able to reproduce the issue. The only thing I noticed is that the stacktrace of the XNIO thread that causes CPU usage seems to be more dynamic now (it does not seem to be blocked in sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) most of the time). Some thread dumps shows it like below:
"Remoting "config-based-ejb-client-endpoint" I/O-1" #14144 daemon prio=5 os_prio=0 tid=0x00007f85dc750000 nid=0x1b4a runnable [0x00007f850e4c6000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.NativeThread.current(Native Method)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:326)
- locked <0x00000005cc53ca60> (a java.lang.Object)
- locked <0x00000005cc53ca40> (a java.lang.Object)
at org.xnio.nio.NioSocketConduit.read(NioSocketConduit.java:289)
at org.xnio.ssl.JsseStreamConduit.performIO(JsseStreamConduit.java:1401)
at org.xnio.ssl.JsseStreamConduit.flush(JsseStreamConduit.java:1113)
at org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162)
at org.xnio.channels.AssembledStreamChannel.flush(AssembledStreamChannel.java:201)
at org.jboss.remoting3.remote.RemotingMessageChannel.flushAction(RemotingMessageChannel.java:360)
- locked <0x00000005cc544880> (a java.lang.Object)
at org.xnio.channels.TranslatingSuspendableChannel.flush(TranslatingSuspendableChannel.java:640)
at org.jboss.remoting3.remote.RemoteConnection$RemoteWriteListener.handleEvent(RemoteConnection.java:273)
- locked <0x00000005cc5446e8> (a java.util.ArrayDeque)
at org.jboss.remoting3.remote.RemoteConnection$RemoteWriteListener.handleEvent(RemoteConnection.java:233)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.channels.TranslatingSuspendableChannel.handleWritable(TranslatingSuspendableChannel.java:252)
at org.xnio.channels.TranslatingSuspendableChannel$2.handleEvent(TranslatingSuspendableChannel.java:122)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.conduits.WriteReadyHandler$ChannelListenerHandler.writeReady(WriteReadyHandler.java:65)
at org.xnio.ssl.JsseStreamConduit.run(JsseStreamConduit.java:393)
at org.xnio.ssl.JsseStreamConduit.writeReady(JsseStreamConduit.java:527)
at org.xnio.ssl.JsseStreamConduit$1.writeReady(JsseStreamConduit.java:287)
at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:94)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:571)
> Closing an EJBClientContext sometimes hangs causing high CPU usage
> ------------------------------------------------------------------
>
> Key: WFLY-9364
> URL: https://issues.jboss.org/browse/WFLY-9364
> Project: WildFly
> Issue Type: Bug
> Affects Versions: 10.1.0.Final
> Environment: Server:
> - OS:
> Red Hat Enterprise Linux Server release 7.2 (Maipo)
> - JDK:
> java version "1.8.0_144"
> Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
> Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
> - WildFly:
> WildFly 10.1.0.Final
> Client:
> - OS:
> Red Hat Enterprise Linux Server release 7.2 (Maipo)
> - JDK:
> java version "1.8.0_144"
> Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
> Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
> Reporter: Marius Tantareanu
> Assignee: Jason Greene
> Attachments: echo-client.zip, output_rhel7.txt, output_win10.txt, simple-ear.ear, td_rhel7.txt, td_win10.txt
>
>
> EJBClientContext.close() sometimes hangs and causes high CPU usage. We have a WildFly client that uses EJB client API to invoke some EJBs remotely.
> Basically the client executes the following actions in a loop:
> - setup an EJBClientContext programatically
> - create a JNDI context
> - lookup an EJB and invoke some operations on it
> - close the JNDI context and the EJBClientContext
> After the client runs a few hundreds iterations (the actual number of iterations varies quite a lot from one run to another) it blocks while invoking EJBClientContext.close(). Also one XNIO thread in the client app is constantly consuming one CPU core (full thread dump from the client app is attached). I was only able to reproduce this when connecting over TLS (port 8443). When using the unsecure port (8080) the problem does not reproduce (or it reproduces much less frequently and I didn't run enough iterations to catch it).
> Once the client app enters this state, top shows something like (notice the CPU usage of thread 12512):
> >top -H -p 6463
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 12512 root 20 0 11.527g 222680 13788 R 99.3 0.7 141:10.99 java
> 6466 root 20 0 11.527g 222680 13788 S 0.3 0.7 0:07.05 java
> 6467 root 20 0 11.527g 222680 13788 S 0.3 0.7 0:06.97 java
> 6477 root 20 0 11.527g 222680 13788 S 0.3 0.7 0:04.24 java
> 6463 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:00.00 java
> 6464 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:03.46 java
> 6465 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:06.94 java
> 6468 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:07.02 java
> 6469 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:16.42 java
> 6470 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:00.01 java
> 6471 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:00.01 java
> 6472 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:00.00 java
> 6473 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:06.87 java
> 6474 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:06.70 java
> 6475 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:03.09 java
> 6476 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:00.00 java
> 12513 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:00.00 java
> 12514 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:00.00 java
> 12515 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:00.00 java
> 12516 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:00.00 java
> 12517 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:00.00 java
> 12518 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:00.00 java
> 12519 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:00.00 java
> 12520 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:00.00 java
> 12523 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:00.00 java
> 12524 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:00.00 java
> 12597 root 20 0 11.527g 222680 13788 S 0.0 0.7 0:00.00 java
> The thread that causes the CPU usage is the following:
> "Remoting "config-based-ejb-client-endpoint" I/O-1" #6025 daemon prio=5 os_prio=0 tid=0x00007f2f709d3000 nid=0x30e0 runnable [0x00007f2e7f9be000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005cc1844e8> (a sun.nio.ch.Util$3)
> - locked <0x00000005cc19f258> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005cc184468> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:515)
> The client app main thread is blocked as below:
> "main" #1 prio=5 os_prio=0 tid=0x00007f2f70008800 nid=0x1940 in Object.wait() [0x00007f2f76350000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.jboss.remoting3.spi.AbstractHandleableCloseable.close(AbstractHandleableCloseable.java:190)
> - locked <0x00000005cc18de48> (a java.lang.Object)
> at org.jboss.ejb.client.remoting.ConnectionPool.safeClose(ConnectionPool.java:177)
> at org.jboss.ejb.client.remoting.ConnectionPool.release(ConnectionPool.java:104)
> - locked <0x00000005cbd369e0> (a org.jboss.ejb.client.remoting.ConnectionPool)
> at org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection.close(ConnectionPool.java:198)
> at org.jboss.ejb.client.remoting.RemotingConnectionManager.safeClose(RemotingConnectionManager.java:65)
> - locked <0x00000005cc1a6840> (a java.util.Collections$SynchronizedRandomAccessList)
> at org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector$ContextCloseListener.contextClosed(ConfigBasedEJBClientContextSelector.java:220)
> at org.jboss.ejb.client.EJBClientContext.close(EJBClientContext.java:1305)
> - locked <0x00000005cc1a6888> (a org.jboss.ejb.client.EJBClientContext)
> at com.microfocus.echoclient.EchoClient.disconnect(EchoClient.java:66)
> at com.microfocus.echoclient.EchoClient.connectDisconnect(EchoClient.java:54)
> at com.microfocus.echoclient.EchoClient.main(EchoClient.java:36)
> The problem also reproduces on Windows (full thread dump of the client app is attached).
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 7 months
[JBoss JIRA] (WFCORE-3309) Bump the kernel management API version to 6.0.0
by Brian Stansberry (JIRA)
Brian Stansberry created WFCORE-3309:
----------------------------------------
Summary: Bump the kernel management API version to 6.0.0
Key: WFCORE-3309
URL: https://issues.jboss.org/browse/WFCORE-3309
Project: WildFly Core
Issue Type: Task
Components: Domain Management
Reporter: Brian Stansberry
Assignee: Brian Stansberry
We know there are going to be API changes in WF Core 4, so we need to get the API version bumped so that when those changes happen appropriate transformers can be written to transform to the previous version.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 7 months
[JBoss JIRA] (HAWKULARQE-33) [QE] - JMAN4-188 - Secure link from CloudForms to MW Provider
by Michael Foley (JIRA)
[ https://issues.jboss.org/browse/HAWKULARQE-33?page=com.atlassian.jira.plu... ]
Michael Foley commented on HAWKULARQE-33:
-----------------------------------------
[~pyadav] please retest in the upcoming test cycle. Consider using this blog post as a basis for moving forward ....
New Hawkular blog post from noreply(a)hawkular.org (Josejulio Martínez): http://ift.tt/2fiy8ZH
SSL provides identification of the communicating parties, confidentiality and integrity of the information being shared.
In production environments, network eavesdropping could be a concern. You can use SSL to provide a secure communication channel between servers.
Hawkular Services expects Hawkular Agents to connect, push information and listen for commands. These commands also include server credentials that could be vulnerable to a man-in-the-middle attack.
A previous article shows how to configure your Hawkular Services with SSL and have Hawkular Agents connect to it trough SSL. This guide will show how is done for Dockerized Hawkular Services Agents.
Preparing your certificates
Before starting we need to prepare the certificates that we are going to use. Your public and private key for Hawkular Services need to be on PEM or PKC12 format. For this guide we will use PEM.
We can create self-signed certificates (in PEM format) using:
keytool -genkey -keystore hawkular.keystore -alias hawkular -dname "CN=hawkular-services" -keyalg RSA -storepass hawkular -keypass hawkular -validity 36500 -ext san=ip:127.0.0.1,dns:hawkular-services
keytool -importkeystore -srckeystore hawkular.keystore -destkeystore hawkular.p12 -deststoretype PKCS12 -srcalias hawkular -deststorepass hawkular -destkeypass hawkular -srcstorepass hawkular
openssl pkcs12 -in hawkular.p12 -password pass:hawkular -nokeys -out hawkular-services-public.pem
openssl pkcs12 -in hawkular.p12 -password pass:hawkular -nodes -nocerts -out hawkular-services-private.key
Important
When creating the certificates, remember to include the host that Hawkular will use (as Common Name(CN) and Subject Alternative Name(SAN)), else the certificate validation will fail. Through this guide, host will be set to hawkular-services.
Warning
Always include san=ip:127.0.0.1 in your certificate, as this will be used internally by Hawkular Services.
Starting Cassandra
Hawkular Services requires a cassandra instance, you can start one by doing:
docker run --name hawkular-cassandra -e CASSANDRA_START_RPC=true -d cassandra:3.0.12
Starting Hawkular Services on SSL
Now that there is a cassandra ready, Hawkular Services can be started, it will be linked to the cassandra instance (hawkular-cassandra)
docker pull hawkular/hawkular-services
docker run --name hawkular-services --link=hawkular-cassandra -e CASSANDRA_NODES=hawkular-cassandra -e HAWKULAR_HOSTNAME=hawkular-services -e HAWKULAR_USE_SSL=true -p 8443:8443 -v `pwd`/hawkular-services-private.key:/client-secrets/hawkular-services-private.key:z -v `pwd`/hawkular-services-public.pem:/client-secrets/hawkular-services-public.pem:z hawkular/hawkular-services
Note
To avoid guessing the host, we explicitly set it to hawkular-services using HAWKULAR_HOSTNAME environmental variable.
Warning
If you don’t specify any certificate, Hawkular services will create one, but we won’t be able to connect agents unless we export the certificate.
Starting a Hawkular Agent
By now there should be a Hawkular Services listening on default secure port 8443, if any agent wants to connect, it will need to know and trust its certificate. If you are self-signing your certificate, you will need to pass Hawkular Service’s certificate when starting the agent.
docker pull hawkular/wildfly-hawkular-javaagent
docker run --name hawkular-agent-01 --link=hawkular-services -e HAWKULAR_SERVER_PROTOCOL=https -e HAWKULAR_SERVER_ADDR=hawkular-services -e HAWKULAR_SERVER_PORT=8443 -v `pwd`/hawkular-services-public.pem:/client-secrets/hawkular-services-public.pem:z hawkular/wildfly-hawkular-javaagent
Note
Host must match with the one specified in the certificate, thus setting HAWKULAR_SERVER_ADDR to hawkular-services is required. Update as needed if using other value.
Testing the setup
Any Hawkular client that supports connecting using SSL can be used. Curl, ManageIQ and HawkFX will be show below.
Note
If using self-signed certificates, the client machine needs to trust Hawkular Services certificate or the client should support to specify which certificate to trust. See here for more info.
Curl
Simplest way to test, we only need to tell curl how to resolve hawkular-services and pass the public certificate.
curl -H "Hawkular-Tenant: hawkular" --resolve "hawkular-services:8443:127.0.0.1" --cacert hawkular-services-prd -X GET https://hawkular-services:8443/hawkular/metrics/metrics
It will output metrics stored on Hawkular Services.
ManageIQ
It can be quickly tested using the dockerized ManageIQ as follow:
docker run --link=hawkular-services --privileged -d -p 8444:443 manageiq/manageiq:fine-3
Once ManageIQ has started, add a Middleware Provider, there is three secured ways to do that:
SSL — Requires that the client box trusts Hawkular Services certificate.
SSL trusting custom CA — Requires to provide the certificate to trust.
SSL without validation — No validation is performed.
We will focus on (2) and (3) as (1) will require to trust the certificates on the machine itself.
Before going onto details, navigate to https://localhost:8444/, login with default username admin and password smartvm. Click on Middleware → Configuration → Add a New Middleware Provider.
SSL trusting custom CA
We need to select SSL trusting custom CA in Security Protocol and copy the certificate from the contents of hawkular-services-public.pem. Fill the Hostname with the one used in the certificate and complete the required information.
2017 09 manageiq custom ca
Figure 1: Middleware Provider SSL trusting custom ca
SSL without validation
We need to select SSL without validation in Security Protocol. No validation regarding the certificate is made with this option, only fill the required information.
2017 09 manageiq custom ca
Figure 2: Middleware Provider SSL without validation
HawkFX
One can connect using HawkFx by either installing the certificate on the machine or disabling the verification as show in the image. You will also need to update your /etc/hosts to resolve hawkular-services.
sudo su -c "echo '127.0.0.1 hawkular-services' >> /etc/hosts"
Alternatively, if no verification is used, one could use localhost instead of hawkular-services.
2017 09 hawkfx disable validation
Figure 3: HawkFX without validation
Conclusion
Securing communications between a dockerized Hawkular Agent and Hawkular Services is possible with self-signed certificates. Connecting clients is also possible with the additional step of providing the certificate.
from Hawkular Blog
> [QE] - JMAN4-188 - Secure link from CloudForms to MW Provider
> -------------------------------------------------------------
>
> Key: HAWKULARQE-33
> URL: https://issues.jboss.org/browse/HAWKULARQE-33
> Project: Hawkular QE
> Issue Type: Task
> Reporter: Hayk Hovsepyan
> Assignee: Prachi Yadav
> Priority: Critical
> Labels: mm-integration-hawkular-qe
>
> See [JMAN4-188|https://issues.jboss.org/browse/JMAN4-188]
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 7 months
[JBoss JIRA] (DROOLS-1740) Read configuration of defualt KieBase and KieSession from top level kmodule
by Mario Fusco (JIRA)
Mario Fusco created DROOLS-1740:
-----------------------------------
Summary: Read configuration of defualt KieBase and KieSession from top level kmodule
Key: DROOLS-1740
URL: https://issues.jboss.org/browse/DROOLS-1740
Project: Drools
Issue Type: Enhancement
Components: core engine
Reporter: Mario Fusco
Assignee: Mario Fusco
When a kmodule depends on another one and both kmodule declares their own default KieBase and KieSession the engine disable all defaults printing a warning like the following
{code}
Found more than one default KieBase: disabling all. KieBases will be accessible only by name
{code}
In this case it is required to keep the default of the top level kmodule.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 7 months
[JBoss JIRA] (WFLY-9014) Second remote EJB client connection prevents server from shutting down (Elytron, Remoting)
by David Lloyd (JIRA)
[ https://issues.jboss.org/browse/WFLY-9014?page=com.atlassian.jira.plugin.... ]
David Lloyd resolved WFLY-9014.
-------------------------------
Fix Version/s: 11.0.0.Final
Resolution: Done
> Second remote EJB client connection prevents server from shutting down (Elytron, Remoting)
> -------------------------------------------------------------------------------------------
>
> Key: WFLY-9014
> URL: https://issues.jboss.org/browse/WFLY-9014
> Project: WildFly
> Issue Type: Bug
> Components: EJB, Remoting, Security
> Reporter: Josef Cacek
> Assignee: David Lloyd
> Fix For: 11.0.0.Final
>
>
> Second call of EJB client causes server stuck in some cases. When this happens administrator is not able to shutdown or reload such a server.
> *Scenario details:*
> Elytron is configured as security provider on servers and Elytron client API is used for authentication. Protected stateless beans are used.
> {noformat}
> EJB Client -> Entry bean on Server1 -> WhoAmI bean on Server2
> {noformat}
> * Client uses {{AuthenticationConfiguration}} with valid credentials for Server1; Client calls the Entry bean twice - the difference is in arguments provided. The arguments are subsequently used for authentication from Server1 to Server2 (using AuthenticationConfiguration):
> ** first call uses null for both username and password
> ** second call uses valid credentials for Server2
> * Entry bean on Server1 calls WhoAmI bean on Server2 with credentials which was provided as method arguments;
> * WhoAmI bean on Server2 just returns caller principal
> The Client is executed twice, the second call doesn't finish
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 7 months
[JBoss JIRA] (WFCORE-3238) Unable to use principal name with spaces within the simple-permission-mapper (Elytron)
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFCORE-3238?page=com.atlassian.jira.plugi... ]
Brian Stansberry updated WFCORE-3238:
-------------------------------------
Fix Version/s: 3.0.2.Final
(was: 3.0.4.Final)
Correcting the Fix Version. There have been no changes in 3.x since we release 3.0.3 so it can't be 3.0.4, and 3.0.3 vs 3.0.2 was just a remoting bump to pick up a fix unrelated to this.
> Unable to use principal name with spaces within the simple-permission-mapper (Elytron)
> --------------------------------------------------------------------------------------
>
> Key: WFCORE-3238
> URL: https://issues.jboss.org/browse/WFCORE-3238
> Project: WildFly Core
> Issue Type: Bug
> Components: Security
> Reporter: Josef Cacek
> Assignee: Darran Lofthouse
> Priority: Critical
> Fix For: 3.0.2.Final
>
>
> The {{simple-permission-mapper}} fails to handle principal names which include space character in them.
> The problem seems to be caused by storing the names into one attribute (space-separated values) in Elytron subsystem schema.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 7 months