From mazz at redhat.com  Tue Sep  5 09:35:09 2017
From: mazz at redhat.com (John Mazzitelli)
Date: Tue, 5 Sep 2017 09:35:09 -0400 (EDT)
Subject: [Hawkular-dev] docker clean up
In-Reply-To: <743261329.7401541.1487478199702.JavaMail.zimbra@redhat.com>
References: <1906147875.7401485.1487477734685.JavaMail.zimbra@redhat.com>
	<743261329.7401541.1487478199702.JavaMail.zimbra@redhat.com>
Message-ID: <1902985695.8618833.1504618509196.JavaMail.zimbra@redhat.com>

Just a little set of tricks that might be helpful to others.

If you want to clean up your docker:

* Delete all containers: docker ps -q -a | xargs docker rm
* Delete all images: docker rmi $(docker images --filter dangling=true)
* Another way of removing all images is: docker images -q | xargs docker rmi
* If images have dependent children, forced removal is via the -f flag: docker images -q | xargs docker rmi -f
* Delete untagged images: docker rmi $(docker images | grep "^<none>" | awk '{print $3}')

From theute at redhat.com  Wed Sep  6 17:35:50 2017
From: theute at redhat.com (Thomas Heute)
Date: Wed, 06 Sep 2017 21:35:50 +0000
Subject: [Hawkular-dev] New Hawkular Blog Post: Hawkular Alerts with
	OpenTracing
Message-ID: <59b06a368873c_567b46b324994db@mini-queue-02.resque.ife.mail>

New Hawkular blog post from noreply at hawkular.org (John Mazzitelli): http://ift.tt/2xc8MUQ

Two recent blogs discuss how OpenTracing instrumentation can be used to collect application metrics:

http://ift.tt/2rX1NbW

http://ift.tt/2vWrA6Y

A further interesting integration can be the addition of Hawkular Alerts to the environment.

As the previous blog and demo discuss, Hawkular Alerts is a generic, federated alerts system that can trigger events, alerts, and notifications from different, independent systems such as Prometheus, ElasticSearch, and Kafka.

Here we can combine the two. Let?s follow the directions for the OpenTracing demo (using the Jaeger implementation) and add Hawkular Alerts.

What this can show is OpenTracing application metrics triggering alerts when (as in this example) OpenTracing spans encounter a larger-than-expected error rates.

(Note: these instructions assume you are using Kubernetes / Minikube - see the Hawkular OpenTracing blogs linked above for more details on these instructions)

START KUBERNETES

Here we start minikube giving it enough resources to run all of the pods necessary for this demo. We also start up a browser pointing to the Kubernetes dashboard, so you can follow the progress of the remaining instructions.

minikube start --cpus 4 --memory 8192

minikube dashboard

DEPLOY PROMETHEUS

kubectl create -f http://ift.tt/2wHvySK

kubectl create -f http://ift.tt/2tgfO8q

(Note: the last command might not work depending on your version - if you get errors, download a copy of prometheus-kubernetes.yml and edit it, changing ?v1alpha1? to ?v1?)

DEPLOY JAEGER

kubectl create -f http://ift.tt/2tfYiRY

The following will build and deploy the Jaeger example code that will produce the OpenTracing data for the demo:

mkdir -p ${HOME}/opentracing ; cd ${HOME}/opentracing

git clone git at github.com:objectiser/opentracing-prometheus-example.git

cd opentracing-prometheus-example/simple

eval $(minikube docker-env)

mvn clean install docker:build

kubectl create -f services-kubernetes.yml

(Note: The last command might not work depending on your version - if you get errors, edit services-kubernetes.yml, changing ?v1alpha1? to ?v1?)

DEPLOY HAWKULAR-ALERTS AND CREATE ALERT TRIGGER

The following will deploy Hawkular Alerts and create the trigger definition that will trigger an alert when the Jaeger OpenTracing data indicates an error rate that is over 30%

kubectl create -f http://ift.tt/2w86alD

Next use minikube service hawkular-alerts --url to determine the Hawkular Alerts URL and point your browser to the path ?/hawkular/alerts/ui? at that URL (i.e. http://host:port/hawkular/alerts/ui).

>From the browser page running the Hawkular Alerts UI, enter a tenant name in the top right text field (?my-organization? for example) and click the ?Change? button.

Navigate to the ?Triggers? page (found in the left-hand nav menu).

Click the kabob menu icon at the top and select ?New Trigger?.

In the text area, enter the following to define a new trigger that will trigger alerts when the Prometheus query shows that there is a 30% error rate or greater in the accountmgr or ordermgr servers:

{
   &quot;trigger&quot;:{
      &quot;id&quot;:&quot;jaeger-prom-trigger&quot;,
      &quot;name&quot;:&quot;High Error Rate&quot;,
      &quot;description&quot;:&quot;Data indicates high error rate&quot;,
      &quot;severity&quot;:&quot;HIGH&quot;,
      &quot;enabled&quot;:true,
      &quot;autoDisable&quot;:false,
      &quot;tags&quot;:{
         &quot;prometheus&quot;:&quot;Test&quot;
      },
      &quot;context&quot;:{
         &quot;prometheus.url&quot;:&quot;http://prometheus:9090&quot;
      }
   },
   &quot;conditions&quot;:[
      {
         &quot;type&quot;:&quot;EXTERNAL&quot;,
         &quot;alerterId&quot;:&quot;prometheus&quot;,
         &quot;dataId&quot;:&quot;prometheus-test&quot;,
         &quot;expression&quot;:&quot;(sum(increase(span_count{error=\&quot;true\&quot;,span_kind=\&quot;server\&quot;}[1m])) without (pod,instance,job,namespace,endpoint,transaction,error,operation,span_kind) / sum(increase(span_count{span_kind=\&quot;server\&quot;}[1m])) without (pod,instance,job,namespace,endpoint,transaction,error,operation,span_kind)) &gt; 0.3&quot;
      }
   ]
}

Figure 1: Create New Alert Trigger

Figure 2: Alert Trigger

Now navigate back to the ?Dashboard? page (again via the left-hand nav menu). From this Dashboard page, look for alerts when they are triggered. We?ll next start generating the data that will trigger these alerts.

GENERATE SOME SAMPLE OPEN TRACING APPLICATION DATA

export ORDERMGR=$(minikube service ordermgr --url)

${HOME}/opentracing/opentracing-prometheus-example/simple/genorders.sh

Once the data starts to be collected, you will see alerts in the Hawkular Alerts UI as error rates become over 30% in the past minute (as per the Prometheus query).

Figure 3: Alerts Dashboard

Figure 4: Alert

If you look at the alerts information in the Hawkular Alerts UI, you?ll see the conditions that triggered the alerts. For example, one such alert could look like this:

Time: 2017-09-01 17:41:17 -0400
External[prometheus]: prometheus-test[Event [tenantId=my-organization,
id=1a81471d-340d-4dba-abe9-5b991326dc80, ctime=1504302077288, category=prometheus,
dataId=prometheus-test, dataSource=none, text=[1.504302077286E9, 0.3333333333333333],
context={service=ordermgr, version=0.0.1}, tags={}, trigger=null]] matches
[(sum(increase(span_count{error=&quot;true&quot;,span_kind=&quot;server&quot;}[1m])) without
(pod,instance,job,namespace,endpoint,transaction,error,operation,span_kind) /
sum(increase(span_count{span_kind=&quot;server&quot;}[1m])) without
(pod,instance,job,namespace,endpoint,transaction,error,operation,span_kind)) &gt; 0.3]

Notice the ?ordermgr? service (version &quot;0.0.1&quot;) had an error rate of 0.3333 (33%) which caused the alert since it is above the allowed 30% threshold.

At this point, the Hawkular Alerts UI provides the ability for system admins to log notes about the issue, acknowledge the alert and mark the alert resolved if the underlying issue has been fixed. These lifecycle functions (also available as REST operations) are just part of the value add of Hawkular-Alerts.

You could do more complex things such as only trigger this alert if this Prometheus query generated results AND some other condition was true (say, ElasticSearch logs match a particular pattern, or if a Kafka topic had certain data). This demo merely scratches the surface, but does show how Hawkular Alerts can be used to work with OpenTracing to provide additional capabilities that may be found useful by system administrators and IT support personnel.

from Hawkular Blog
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170906/c675abf7/attachment.html 

From jkremser at redhat.com  Fri Sep  8 09:36:26 2017
From: jkremser at redhat.com (Jiri Kremser)
Date: Fri, 8 Sep 2017 15:36:26 +0200
Subject: [Hawkular-dev] docker clean up
In-Reply-To: <1902985695.8618833.1504618509196.JavaMail.zimbra@redhat.com>
References: <1906147875.7401485.1487477734685.JavaMail.zimbra@redhat.com>
	<743261329.7401541.1487478199702.JavaMail.zimbra@redhat.com>
	<1902985695.8618833.1504618509196.JavaMail.zimbra@redhat.com>
Message-ID: <CAE7VWz-ROXvC3k-O6GBUssP41ndh7Mk1FxHgMKzK1+a_kM-W5Q@mail.gmail.com>

https://github.com/Jiri-Kremser/dotfiles/blob/master/.bashrc#L403

:D

On Tue, Sep 5, 2017 at 3:35 PM, John Mazzitelli <mazz at redhat.com> wrote:

> Just a little set of tricks that might be helpful to others.
>
> If you want to clean up your docker:
>
> * Delete all containers: docker ps -q -a | xargs docker rm
> * Delete all images: docker rmi $(docker images --filter dangling=true)
> * Another way of removing all images is: docker images -q | xargs docker
> rmi
> * If images have dependent children, forced removal is via the -f flag:
> docker images -q | xargs docker rmi -f
> * Delete untagged images: docker rmi $(docker images | grep "^<none>" |
> awk '{print $3}')
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170908/cc2c94fb/attachment-0001.html 

From jtakvori at redhat.com  Fri Sep  8 09:42:49 2017
From: jtakvori at redhat.com (Joel Takvorian)
Date: Fri, 8 Sep 2017 15:42:49 +0200
Subject: [Hawkular-dev]  Grafana datasource release: 1.1.0
Message-ID: <CAJo5TF=xSgpp5FBHFcvkKiF7z5imnerwJfO9C2-3j=MuPfE5-Q@mail.gmail.com>

The Hawkular grafana datasource has been released today to 1.1.0.
This release contains:


   - Allow per-query tenant configuration [feat. #62 / PR #85]
   - Annotations can now be configured out of Availability metrics [PR #85]
   and Hawkular Alerts events [PR #88] in addition to string metrics
   - Issue #86 / PR #87 : allow dot character in tag names


IMPORTANT INFORMATION:
Because now this datasource is able to fetch not only Hawkular Metrics but
also Hawkular Alerts, the server URL in the datasource configuration must
be updated:
*http://myserver:123/hawkular/metrics
<http://myserver:123/hawkular/metrics>* must be changed to
*http://myserver:123/hawkular
<http://myserver:123/hawkular>*

Release link:
https://github.com/hawkular/hawkular-grafana-datasource/releases/tag/v1.1.0

DockerHub registry updated as well.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170908/0db78226/attachment.html 

From theute at redhat.com  Wed Sep 20 04:52:10 2017
From: theute at redhat.com (Thomas Heute)
Date: Wed, 20 Sep 2017 08:52:10 +0000
Subject: [Hawkular-dev] New Hawkular Blog Post: Connecting Hawkular Agent to
 Hawkular Services over SSL
Message-ID: <59c22c3ab1f68_48b1f2733033856@mini-queue-04.resque.ife.mail>

New Hawkular blog post from noreply at hawkular.org (Josejulio Mart?nez): http://ift.tt/2fiy8ZH

SSL provides identification of the communicating parties, confidentiality and integrity of the information being shared.

In production environments, network eavesdropping could be a concern. You can use SSL to provide a secure communication channel between servers.

Hawkular Services expects Hawkular Agents to connect, push information and listen for commands. These commands also include server credentials that could be vulnerable to a man-in-the-middle attack.

A previous article shows how to configure your Hawkular Services with SSL and have Hawkular Agents connect to it trough SSL. This guide will show how is done for Dockerized Hawkular Services Agents.

Preparing your certificates

Before starting we need to prepare the certificates that we are going to use. Your public and private key for Hawkular Services need to be on PEM or PKC12 format. For this guide we will use PEM.

We can create self-signed certificates (in PEM format) using:

keytool -genkey -keystore hawkular.keystore -alias hawkular -dname &quot;CN=hawkular-services&quot; -keyalg RSA -storepass hawkular -keypass hawkular -validity 36500 -ext san=ip:127.0.0.1,dns:hawkular-services
keytool -importkeystore -srckeystore hawkular.keystore  -destkeystore hawkular.p12 -deststoretype PKCS12 -srcalias hawkular -deststorepass hawkular -destkeypass hawkular -srcstorepass hawkular
openssl pkcs12 -in hawkular.p12 -password pass:hawkular -nokeys -out hawkular-services-public.pem
openssl pkcs12 -in hawkular.p12 -password pass:hawkular -nodes -nocerts -out hawkular-services-private.key

Important

When creating the certificates, remember to include the host that Hawkular will use (as Common Name(CN) and Subject Alternative Name(SAN)), else the certificate validation will fail. Through this guide, host will be set to hawkular-services.

Warning

Always include san=ip:127.0.0.1 in your certificate, as this will be used internally by Hawkular Services.

Starting Cassandra

Hawkular Services requires a cassandra instance, you can start one by doing:

docker run --name hawkular-cassandra -e CASSANDRA_START_RPC=true -d cassandra:3.0.12

Starting Hawkular Services on SSL

Now that there is a cassandra ready, Hawkular Services can be started, it will be linked to the cassandra instance (hawkular-cassandra)

docker pull hawkular/hawkular-services
docker run --name hawkular-services --link=hawkular-cassandra -e CASSANDRA_NODES=hawkular-cassandra -e HAWKULAR_HOSTNAME=hawkular-services -e HAWKULAR_USE_SSL=true -p 8443:8443 -v `pwd`/hawkular-services-private.key:/client-secrets/hawkular-services-private.key:z -v `pwd`/hawkular-services-public.pem:/client-secrets/hawkular-services-public.pem:z hawkular/hawkular-services

Note

To avoid guessing the host, we explicitly set it to hawkular-services using HAWKULAR_HOSTNAME environmental variable.

Warning

If you don?t specify any certificate, Hawkular services will create one, but we won?t be able to connect agents unless we export the certificate.

Starting a Hawkular Agent

By now there should be a Hawkular Services listening on default secure port 8443, if any agent wants to connect, it will need to know and trust its certificate. If you are self-signing your certificate, you will need to pass Hawkular Service?s certificate when starting the agent.

docker pull hawkular/wildfly-hawkular-javaagent
docker run --name hawkular-agent-01 --link=hawkular-services -e HAWKULAR_SERVER_PROTOCOL=https -e HAWKULAR_SERVER_ADDR=hawkular-services -e HAWKULAR_SERVER_PORT=8443 -v `pwd`/hawkular-services-public.pem:/client-secrets/hawkular-services-public.pem:z hawkular/wildfly-hawkular-javaagent

Note

Host must match with the one specified in the certificate, thus setting HAWKULAR_SERVER_ADDR to hawkular-services is required. Update as needed if using other value.

Testing the setup

Any Hawkular client that supports connecting using SSL can be used. Curl, ManageIQ and HawkFX will be show below.

Note

If using self-signed certificates, the client machine needs to trust Hawkular Services certificate or the client should support to specify which certificate to trust. See here for more info.

Curl

Simplest way to test, we only need to tell curl how to resolve hawkular-services and pass the public certificate.

curl -H &quot;Hawkular-Tenant: hawkular&quot; --resolve &quot;hawkular-services:8443:127.0.0.1&quot; --cacert hawkular-services-prd -X GET https://hawkular-services:8443/hawkular/metrics/metrics

It will output metrics stored on Hawkular Services.

ManageIQ

It can be quickly tested using the dockerized ManageIQ as follow:

docker run --link=hawkular-services --privileged -d -p 8444:443 manageiq/manageiq:fine-3

Once ManageIQ has started, add a Middleware Provider, there is three secured ways to do that:

SSL ???Requires that the client box trusts Hawkular Services certificate.

SSL trusting custom CA???Requires to provide the certificate to trust.

SSL without validation???No validation is performed.

We will focus on (2) and (3) as (1) will require to trust the certificates on the machine itself.

Before going onto details, navigate to https://localhost:8444/, login with default username admin and password smartvm. Click on Middleware ? Configuration ? Add a New Middleware Provider.

SSL trusting custom CA

We need to select SSL trusting custom CA in Security Protocol and copy the certificate from the contents of hawkular-services-public.pem. Fill the Hostname with the one used in the certificate and complete the required information.

Figure 1: Middleware Provider SSL trusting custom ca

SSL without validation

We need to select SSL without validation in Security Protocol. No validation regarding the certificate is made with this option, only fill the required information.

Figure 2: Middleware Provider SSL without validation

HawkFX

One can connect using HawkFx by either installing the certificate on the machine or disabling the verification as show in the image. You will also need to update your /etc/hosts to resolve hawkular-services.

sudo su -c &quot;echo '127.0.0.1 hawkular-services' &gt;&gt; /etc/hosts&quot;

Alternatively, if no verification is used, one could use localhost instead of hawkular-services.

Figure 3: HawkFX without validation

Conclusion

Securing communications between a dockerized Hawkular Agent and Hawkular Services is possible with self-signed certificates. Connecting clients is also possible with the additional step of providing the certificate.

from Hawkular Blog
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170920/db21e943/attachment.html 

From abonas at redhat.com  Thu Sep 28 05:40:26 2017
From: abonas at redhat.com (Alissa Bonas)
Date: Thu, 28 Sep 2017 12:40:26 +0300
Subject: [Hawkular-dev] docker clean up
In-Reply-To: <1902985695.8618833.1504618509196.JavaMail.zimbra@redhat.com>
References: <1906147875.7401485.1487477734685.JavaMail.zimbra@redhat.com>
	<743261329.7401541.1487478199702.JavaMail.zimbra@redhat.com>
	<1902985695.8618833.1504618509196.JavaMail.zimbra@redhat.com>
Message-ID: <CAOJV2uBHAkAZ27v3OO+4=Ld_63CDL+Av_AGuh5Nt55Ws+770zA@mail.gmail.com>

On Tue, Sep 5, 2017 at 4:35 PM, John Mazzitelli <mazz at redhat.com> wrote:

> Just a little set of tricks that might be helpful to others.
>
> If you want to clean up your docker:
>
> * Delete all containers: docker ps -q -a | xargs docker rm
>

you'd probably want 'docker stop...' before 'docker rm...'
and as Jiri's file mentions, cleaning networks could be relevant too.



> * Delete all images: docker rmi $(docker images --filter dangling=true)
> * Another way of removing all images is: docker images -q | xargs docker
> rmi
> * If images have dependent children, forced removal is via the -f flag:
> docker images -q | xargs docker rmi -f
> * Delete untagged images: docker rmi $(docker images | grep "^<none>" |
> awk '{print $3}')
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170928/8c9d0872/attachment-0001.html 

From mazz at redhat.com  Thu Sep 28 11:29:22 2017
From: mazz at redhat.com (John Mazzitelli)
Date: Thu, 28 Sep 2017 11:29:22 -0400 (EDT)
Subject: [Hawkular-dev] logging in java agents is not trivial in WildFly /
	EAP
In-Reply-To: <546374569.23935587.1506610503950.JavaMail.zimbra@redhat.com>
Message-ID: <1947258991.23966844.1506612562805.JavaMail.zimbra@redhat.com>

<tl;dr>
No matter if a java agent uses JBoss Logging or simply relies on the JRE's own Java logging, it still won't work in WildFly/EAP unless you set additional things to environment variables in standalone.conf. So no matter what logging is used, this means problems still need to be overcome when putting java agents in host controllers for EAP 7.0 domain mode.
</tl;dr>

I am looking at how Java Agents should log messages when attached to WildFly or EAP. I want to see if there is a way to implement a Java Agent and NOT have to set any special JAVA_OPTS values to pass to the VM to get it to work.

Turns out, it is not a trivial issue.

I wrote a silly little test agent [1] just to see what would happen if it simply logs a message using *java* logging and spins a thread logging a message every second again using *java* logging. So, no extra third party logging libraries, just using java.util.logging classes that are in the JRE already.

Adding just -javaagent to JAVA_OPTS (in standalone.conf, add something like "-javaagent:/where/it/is/ja.jar=foo=bar") is no good, the server bombs with this error, presumably because my agent already started logging using java logger:

   Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: WFLYLOG0078: The logging subsystem requires the log manager to be org.jboss.logmanager.LogManager. The subsystem has not be initialized and cannot be used. To use JBoss Log Manager you must add the system property "java.util.logging.manager" and set it to "org.jboss.logmanager.LogManager"

Well, catch-22, because if I set java.util.logging.manager=org.jboss.logmanager.LogManager in JAVA_OPTS, this happens when the test agent tries to log:

   Could not load Logmanager "org.jboss.logmanager.LogManager"
   java.lang.ClassNotFoundException: org.jboss.logmanager.LogManager

which then falls back to java logging, which then causes the server to bomb again because it wants JBoss Logging.

If I set this in standalone.conf:

   JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,org.jboss.logmanager"

the java agent still gets this error:

   Could not load Logmanager "org.jboss.logmanager.LogManager"
   java.lang.ClassNotFoundException: org.jboss.logmanager.LogManager

which is then followed by jboss modules error which causes the server to again to fail to start:

   WARNING: Failed to load the specified log manager class org.jboss.logmanager.LogManager
   Exception in thread "main" java.lang.NoClassDefFoundError: org/jboss/logmanager/Level
      ...
	at org.jboss.modules.Module.run(Module.java:320)

The ONLY way to get the server to start properly with this java agent installed (EVEN WITH the agent using ONLY java logging), you have to do all of the above plus add this to JAVA_OPTS:

-Xbootclasspath/p:$JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.0.4.Final.jar 

So, in short, you need to add "-Djava.util.logging.manager=org.jboss.logmanager.LogManager" to JAVA_OPTS, you need to add ",org.jboss.logmanager" to JBOSS_MODULES_SYSTEM_PKGS, and you need to add that -Xbootclasspath to JAVA_OPTS just to get a javaagent to work even if the javaagent simply uses nothing more than standard java logging.

---

[1] $ cat JavaAgent.java
import java.util.logging.*;
public class JavaAgent {
    private static final Logger log = Logger.getLogger(JavaAgent.class.getName());
    public static void premain(String args) {
        log.severe("log in premain: " + args);
        new Thread(new Runnable() {
            public void run() {
                while (true) {
                    try {
                        Thread.sleep(1000);
                        log.severe("log in premain loop: " + args);
                    } catch (Exception e) {
                        return;
                    }
                }
            }
        }).start();
    }
}

To build it:

$ echo "Premain-Class: JavaAgent" > manifest.mf
$ javac JavaAgent.java
$ jar cvfm ja.jar manifest.mf JavaAgent*


From theute at redhat.com  Fri Sep 29 04:35:20 2017
From: theute at redhat.com (Thomas Heute)
Date: Fri, 29 Sep 2017 10:35:20 +0200
Subject: [Hawkular-dev] logging in java agents is not trivial in WildFly
	/ EAP
In-Reply-To: <1947258991.23966844.1506612562805.JavaMail.zimbra@redhat.com>
References: <546374569.23935587.1506610503950.JavaMail.zimbra@redhat.com>
	<1947258991.23966844.1506612562805.JavaMail.zimbra@redhat.com>
Message-ID: <CA+aN=Moiooab9OKKu46jhHqORh0_MrAEiiTAYYCPgDgWW3epcA@mail.gmail.com>

Adding Dimitris and Brian (It relates to
https://issues.jboss.org/browse/WFCORE-2526 )

(Mazz I have a question below)

On Thu, Sep 28, 2017 at 5:29 PM, John Mazzitelli <mazz at redhat.com> wrote:

> <tl;dr>
> No matter if a java agent uses JBoss Logging or simply relies on the JRE's
> own Java logging, it still won't work in WildFly/EAP unless you set
> additional things to environment variables in standalone.conf. So no matter
> what logging is used, this means problems still need to be overcome when
> putting java agents in host controllers for EAP 7.0 domain mode.
> </tl;dr>
>
> I am looking at how Java Agents should log messages when attached to
> WildFly or EAP. I want to see if there is a way to implement a Java Agent
> and NOT have to set any special JAVA_OPTS values to pass to the VM to get
> it to work.
>
> Turns out, it is not a trivial issue.
>
> I wrote a silly little test agent [1] just to see what would happen if it
> simply logs a message using *java* logging and spins a thread logging a
> message every second again using *java* logging. So, no extra third party
> logging libraries, just using java.util.logging classes that are in the JRE
> already.
>
> Adding just -javaagent to JAVA_OPTS (in standalone.conf, add something
> like "-javaagent:/where/it/is/ja.jar=foo=bar") is no good, the server
> bombs with this error, presumably because my agent already started logging
> using java logger:
>
>    Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException:
> WFLYLOG0078: The logging subsystem requires the log manager to be
> org.jboss.logmanager.LogManager. The subsystem has not be initialized and
> cannot be used. To use JBoss Log Manager you must add the system property
> "java.util.logging.manager" and set it to "org.jboss.logmanager.
> LogManager"
>
> Well, catch-22, because if I set java.util.logging.manager=org.jboss.logmanager.LogManager
> in JAVA_OPTS, this happens when the test agent tries to log:
>
>    Could not load Logmanager "org.jboss.logmanager.LogManager"
>    java.lang.ClassNotFoundException: org.jboss.logmanager.LogManager
>
> which then falls back to java logging, which then causes the server to
> bomb again because it wants JBoss Logging.
>
> If I set this in standalone.conf:
>
>    JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,org.jboss.logmanager"
>
> the java agent still gets this error:
>
>    Could not load Logmanager "org.jboss.logmanager.LogManager"
>    java.lang.ClassNotFoundException: org.jboss.logmanager.LogManager
>
> which is then followed by jboss modules error which causes the server to
> again to fail to start:
>
>    WARNING: Failed to load the specified log manager class
> org.jboss.logmanager.LogManager
>    Exception in thread "main" java.lang.NoClassDefFoundError:
> org/jboss/logmanager/Level
>       ...
>         at org.jboss.modules.Module.run(Module.java:320)
>
> The ONLY way to get the server to start properly with this java agent
> installed (EVEN WITH the agent using ONLY java logging), you have to do all
> of the above plus add this to JAVA_OPTS:
>
> -Xbootclasspath/p:$JBOSS_HOME/modules/system/layers/base/
> org/jboss/logmanager/main/jboss-logmanager-2.0.4.Final.jar
>
> So, in short, you need to add "-Djava.util.logging.manager=
> org.jboss.logmanager.LogManager" to JAVA_OPTS, you need to add
> ",org.jboss.logmanager" to JBOSS_MODULES_SYSTEM_PKGS, and you need to add
> that -Xbootclasspath to JAVA_OPTS just to get a javaagent to work even if
> the javaagent simply uses nothing more than standard java logging.
>

That's for WF-Core 2.x (as in EAP 7.0), how about WF-Core 3 (as in EAP 7.1)
? IIRC you also need some of those ugliness ?

Thomas



>
> ---
>
> [1] $ cat JavaAgent.java
> import java.util.logging.*;
> public class JavaAgent {
>     private static final Logger log = Logger.getLogger(JavaAgent.
> class.getName());
>     public static void premain(String args) {
>         log.severe("log in premain: " + args);
>         new Thread(new Runnable() {
>             public void run() {
>                 while (true) {
>                     try {
>                         Thread.sleep(1000);
>                         log.severe("log in premain loop: " + args);
>                     } catch (Exception e) {
>                         return;
>                     }
>                 }
>             }
>         }).start();
>     }
> }
>
> To build it:
>
> $ echo "Premain-Class: JavaAgent" > manifest.mf
> $ javac JavaAgent.java
> $ jar cvfm ja.jar manifest.mf JavaAgent*
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170929/075dedaf/attachment.html 

From brian.stansberry at redhat.com  Fri Sep 29 10:44:25 2017
From: brian.stansberry at redhat.com (Brian Stansberry)
Date: Fri, 29 Sep 2017 09:44:25 -0500
Subject: [Hawkular-dev] logging in java agents is not trivial in WildFly
	/ EAP
In-Reply-To: <CA+aN=Moiooab9OKKu46jhHqORh0_MrAEiiTAYYCPgDgWW3epcA@mail.gmail.com>
References: <546374569.23935587.1506610503950.JavaMail.zimbra@redhat.com>
	<1947258991.23966844.1506612562805.JavaMail.zimbra@redhat.com>
	<CA+aN=Moiooab9OKKu46jhHqORh0_MrAEiiTAYYCPgDgWW3epcA@mail.gmail.com>
Message-ID: <CAPhbRDb7s1K_X6Fj4vK6wQSh7FY7Kdzo4UPrzhNJrTd4g4hAtw@mail.gmail.com>

I've copied James Perkins, who is the WildFly logging component lead.

Brian Stansberry
Manager, Senior Principal Software Engineer
Red Hat

On Fri, Sep 29, 2017 at 3:35 AM, Thomas Heute <theute at redhat.com> wrote:

> Adding Dimitris and Brian (It relates to https://issues.jboss.org/
> browse/WFCORE-2526 )
>
> (Mazz I have a question below)
>
> On Thu, Sep 28, 2017 at 5:29 PM, John Mazzitelli <mazz at redhat.com> wrote:
>
>> <tl;dr>
>> No matter if a java agent uses JBoss Logging or simply relies on the
>> JRE's own Java logging, it still won't work in WildFly/EAP unless you set
>> additional things to environment variables in standalone.conf. So no matter
>> what logging is used, this means problems still need to be overcome when
>> putting java agents in host controllers for EAP 7.0 domain mode.
>> </tl;dr>
>>
>> I am looking at how Java Agents should log messages when attached to
>> WildFly or EAP. I want to see if there is a way to implement a Java Agent
>> and NOT have to set any special JAVA_OPTS values to pass to the VM to get
>> it to work.
>>
>> Turns out, it is not a trivial issue.
>>
>> I wrote a silly little test agent [1] just to see what would happen if it
>> simply logs a message using *java* logging and spins a thread logging a
>> message every second again using *java* logging. So, no extra third party
>> logging libraries, just using java.util.logging classes that are in the JRE
>> already.
>>
>> Adding just -javaagent to JAVA_OPTS (in standalone.conf, add something
>> like "-javaagent:/where/it/is/ja.jar=foo=bar") is no good, the server
>> bombs with this error, presumably because my agent already started logging
>> using java logger:
>>
>>    Caused by: java.util.concurrent.ExecutionException:
>> java.lang.IllegalStateException: WFLYLOG0078: The logging subsystem
>> requires the log manager to be org.jboss.logmanager.LogManager. The
>> subsystem has not be initialized and cannot be used. To use JBoss Log
>> Manager you must add the system property "java.util.logging.manager" and
>> set it to "org.jboss.logmanager.LogManager"
>>
>> Well, catch-22, because if I set java.util.logging.manager=org.jboss.logmanager.LogManager
>> in JAVA_OPTS, this happens when the test agent tries to log:
>>
>>    Could not load Logmanager "org.jboss.logmanager.LogManager"
>>    java.lang.ClassNotFoundException: org.jboss.logmanager.LogManager
>>
>> which then falls back to java logging, which then causes the server to
>> bomb again because it wants JBoss Logging.
>>
>> If I set this in standalone.conf:
>>
>>    JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,org.jboss.logmanager"
>>
>> the java agent still gets this error:
>>
>>    Could not load Logmanager "org.jboss.logmanager.LogManager"
>>    java.lang.ClassNotFoundException: org.jboss.logmanager.LogManager
>>
>> which is then followed by jboss modules error which causes the server to
>> again to fail to start:
>>
>>    WARNING: Failed to load the specified log manager class
>> org.jboss.logmanager.LogManager
>>    Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/jboss/logmanager/Level
>>       ...
>>         at org.jboss.modules.Module.run(Module.java:320)
>>
>> The ONLY way to get the server to start properly with this java agent
>> installed (EVEN WITH the agent using ONLY java logging), you have to do all
>> of the above plus add this to JAVA_OPTS:
>>
>> -Xbootclasspath/p:$JBOSS_HOME/modules/system/layers/base/org
>> /jboss/logmanager/main/jboss-logmanager-2.0.4.Final.jar
>>
>> So, in short, you need to add "-Djava.util.logging.manager=o
>> rg.jboss.logmanager.LogManager" to JAVA_OPTS, you need to add
>> ",org.jboss.logmanager" to JBOSS_MODULES_SYSTEM_PKGS, and you need to add
>> that -Xbootclasspath to JAVA_OPTS just to get a javaagent to work even if
>> the javaagent simply uses nothing more than standard java logging.
>>
>
> That's for WF-Core 2.x (as in EAP 7.0), how about WF-Core 3 (as in EAP
> 7.1) ? IIRC you also need some of those ugliness ?
>
> Thomas
>
>
>
>>
>> ---
>>
>> [1] $ cat JavaAgent.java
>> import java.util.logging.*;
>> public class JavaAgent {
>>     private static final Logger log = Logger.getLogger(JavaAgent.cla
>> ss.getName());
>>     public static void premain(String args) {
>>         log.severe("log in premain: " + args);
>>         new Thread(new Runnable() {
>>             public void run() {
>>                 while (true) {
>>                     try {
>>                         Thread.sleep(1000);
>>                         log.severe("log in premain loop: " + args);
>>                     } catch (Exception e) {
>>                         return;
>>                     }
>>                 }
>>             }
>>         }).start();
>>     }
>> }
>>
>> To build it:
>>
>> $ echo "Premain-Class: JavaAgent" > manifest.mf
>> $ javac JavaAgent.java
>> $ jar cvfm ja.jar manifest.mf JavaAgent*
>>
>> _______________________________________________
>> hawkular-dev mailing list
>> hawkular-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170929/7812737a/attachment-0001.html 

From jperkins at redhat.com  Fri Sep 29 12:05:33 2017
From: jperkins at redhat.com (James Perkins)
Date: Fri, 29 Sep 2017 09:05:33 -0700
Subject: [Hawkular-dev] logging in java agents is not trivial in WildFly
	/ EAP
In-Reply-To: <CAPhbRDb7s1K_X6Fj4vK6wQSh7FY7Kdzo4UPrzhNJrTd4g4hAtw@mail.gmail.com>
References: <546374569.23935587.1506610503950.JavaMail.zimbra@redhat.com>
	<1947258991.23966844.1506612562805.JavaMail.zimbra@redhat.com>
	<CA+aN=Moiooab9OKKu46jhHqORh0_MrAEiiTAYYCPgDgWW3epcA@mail.gmail.com>
	<CAPhbRDb7s1K_X6Fj4vK6wQSh7FY7Kdzo4UPrzhNJrTd4g4hAtw@mail.gmail.com>
Message-ID: <CADGwLpiBWRAgLRkD2LTSt4Xaxf=yQb6fcA4cT0qYr7T+6qLQkA@mail.gmail.com>

Unfortunately there's just nothing that can be done about that. The log
manager has to be set before any logger is initialized. Since it's an agent
the log manager needs to be set on the boot class path. This isn't an issue
with WildFly/EAP it's just the way JUL works. Any container using a
different log manager other than JUL would have this issue.

Side note. It's better now to use -Xbootclasspath/a: as the /p option was
removed in Java 9, but /a was not.

On Fri, Sep 29, 2017 at 7:44 AM, Brian Stansberry <
brian.stansberry at redhat.com> wrote:

> I've copied James Perkins, who is the WildFly logging component lead.
>
> Brian Stansberry
> Manager, Senior Principal Software Engineer
> Red Hat
>
> On Fri, Sep 29, 2017 at 3:35 AM, Thomas Heute <theute at redhat.com> wrote:
>
>> Adding Dimitris and Brian (It relates to https://issues.jboss.org/br
>> owse/WFCORE-2526 )
>>
>> (Mazz I have a question below)
>>
>> On Thu, Sep 28, 2017 at 5:29 PM, John Mazzitelli <mazz at redhat.com> wrote:
>>
>>> <tl;dr>
>>> No matter if a java agent uses JBoss Logging or simply relies on the
>>> JRE's own Java logging, it still won't work in WildFly/EAP unless you set
>>> additional things to environment variables in standalone.conf. So no matter
>>> what logging is used, this means problems still need to be overcome when
>>> putting java agents in host controllers for EAP 7.0 domain mode.
>>> </tl;dr>
>>>
>>> I am looking at how Java Agents should log messages when attached to
>>> WildFly or EAP. I want to see if there is a way to implement a Java Agent
>>> and NOT have to set any special JAVA_OPTS values to pass to the VM to get
>>> it to work.
>>>
>>> Turns out, it is not a trivial issue.
>>>
>>> I wrote a silly little test agent [1] just to see what would happen if
>>> it simply logs a message using *java* logging and spins a thread logging a
>>> message every second again using *java* logging. So, no extra third party
>>> logging libraries, just using java.util.logging classes that are in the JRE
>>> already.
>>>
>>> Adding just -javaagent to JAVA_OPTS (in standalone.conf, add something
>>> like "-javaagent:/where/it/is/ja.jar=foo=bar") is no good, the server
>>> bombs with this error, presumably because my agent already started logging
>>> using java logger:
>>>
>>>    Caused by: java.util.concurrent.ExecutionException:
>>> java.lang.IllegalStateException: WFLYLOG0078: The logging subsystem
>>> requires the log manager to be org.jboss.logmanager.LogManager. The
>>> subsystem has not be initialized and cannot be used. To use JBoss Log
>>> Manager you must add the system property "java.util.logging.manager" and
>>> set it to "org.jboss.logmanager.LogManager"
>>>
>>> Well, catch-22, because if I set java.util.logging.manager=org.jboss.logmanager.LogManager
>>> in JAVA_OPTS, this happens when the test agent tries to log:
>>>
>>>    Could not load Logmanager "org.jboss.logmanager.LogManager"
>>>    java.lang.ClassNotFoundException: org.jboss.logmanager.LogManager
>>>
>>> which then falls back to java logging, which then causes the server to
>>> bomb again because it wants JBoss Logging.
>>>
>>> If I set this in standalone.conf:
>>>
>>>    JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,org.jboss.logmanager"
>>>
>>> the java agent still gets this error:
>>>
>>>    Could not load Logmanager "org.jboss.logmanager.LogManager"
>>>    java.lang.ClassNotFoundException: org.jboss.logmanager.LogManager
>>>
>>> which is then followed by jboss modules error which causes the server to
>>> again to fail to start:
>>>
>>>    WARNING: Failed to load the specified log manager class
>>> org.jboss.logmanager.LogManager
>>>    Exception in thread "main" java.lang.NoClassDefFoundError:
>>> org/jboss/logmanager/Level
>>>       ...
>>>         at org.jboss.modules.Module.run(Module.java:320)
>>>
>>> The ONLY way to get the server to start properly with this java agent
>>> installed (EVEN WITH the agent using ONLY java logging), you have to do all
>>> of the above plus add this to JAVA_OPTS:
>>>
>>> -Xbootclasspath/p:$JBOSS_HOME/modules/system/layers/base/org
>>> /jboss/logmanager/main/jboss-logmanager-2.0.4.Final.jar
>>>
>>> So, in short, you need to add "-Djava.util.logging.manager=o
>>> rg.jboss.logmanager.LogManager" to JAVA_OPTS, you need to add
>>> ",org.jboss.logmanager" to JBOSS_MODULES_SYSTEM_PKGS, and you need to add
>>> that -Xbootclasspath to JAVA_OPTS just to get a javaagent to work even if
>>> the javaagent simply uses nothing more than standard java logging.
>>>
>>
>> That's for WF-Core 2.x (as in EAP 7.0), how about WF-Core 3 (as in EAP
>> 7.1) ? IIRC you also need some of those ugliness ?
>>
>> Thomas
>>
>>
>>
>>>
>>> ---
>>>
>>> [1] $ cat JavaAgent.java
>>> import java.util.logging.*;
>>> public class JavaAgent {
>>>     private static final Logger log = Logger.getLogger(JavaAgent.cla
>>> ss.getName());
>>>     public static void premain(String args) {
>>>         log.severe("log in premain: " + args);
>>>         new Thread(new Runnable() {
>>>             public void run() {
>>>                 while (true) {
>>>                     try {
>>>                         Thread.sleep(1000);
>>>                         log.severe("log in premain loop: " + args);
>>>                     } catch (Exception e) {
>>>                         return;
>>>                     }
>>>                 }
>>>             }
>>>         }).start();
>>>     }
>>> }
>>>
>>> To build it:
>>>
>>> $ echo "Premain-Class: JavaAgent" > manifest.mf
>>> $ javac JavaAgent.java
>>> $ jar cvfm ja.jar manifest.mf JavaAgent*
>>>
>>> _______________________________________________
>>> hawkular-dev mailing list
>>> hawkular-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>>
>>
>>
>


-- 
James R. Perkins
JBoss by Red Hat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170929/392fa943/attachment.html