[Hawkular-dev] problem with jmx exporter reading an attribute we need for availability

[John Mazzitelli]

Ok, well, i hate to leave it like this but - this is so friggin' odd that I can only say "it is not my fault" :)

First - note someone changed jmx exporter code because they saw the same thing as we are - that's what this comment is about:

https://github.com/prometheus/jmx_exporter/blob/master/collector/src/main/java/io/prometheus/jmx/JmxScraper.java#L110

See: https://github.com/prometheus/jmx_exporter/issues/89

But the fix is not 100%.

If you grab these three files - you can see it yourself: https://github.com/jmazzitelli/test/tree/master/javaagent
(get Makefile and the two .java files and then run "make download-wildfly unzip-wildfly compile run")

You see i use the same API as the jmx exporter here: https://github.com/jmazzitelli/test/blob/master/javaagent/TestJavaAgent.java#L67

and that is what you will see:

16:31:05,290 INFO  [stdout] (Test Java Agent Thread) =============================================================
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) FIND INFORMATION ABOUT MBEAN: jboss.as:management-root=server
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) =============================================================
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) isRegistered:
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) true
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) getMBeanInfo:
16:31:05,291 INFO  [stdout] (Test Java Agent Thread)   description: The root node of the server-level management model.
16:31:05,291 INFO  [stdout] (Test Java Agent Thread)   #attributes: 19
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) getAttribute:
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) serverState=reload-required
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) queryNames:
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) []
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) queryMBeans:
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) []
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) queryNames(null, null):
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) FOUND IT: jboss.as:management-root=server
16:31:05,291 INFO  [stdout] (Test Java Agent Thread) =============================================================


You will see SOME JMX APIs can see the MBean, queryMBeans and queryNames canNOT (note the empty arrays []).

But notice getMBeanInfo CAN see it - I can even get the attribute value from that! (you can see it is in reload-required state)

But again, queryMBeans returns nothing.

Oddly, queryNames(null, null) DOES return it in the list (see the "FOUND IT" line). It is only if I specifically ask for it does it fail in the query API.

The end result - JMX Exporter (at least sometimes) is not able to get "Server Availability" because it can't get this MBean.

For some odd reason it can get it sometimes - but it seems when it can't, it will never get it.