[keycloak-user] Keycloak user/authorization/realmRevisionsCache and Prometheus JMX exporter: rebalancingEnabled attribute missing

[Schuster Sebastian (INST/ESY1)]

Hi everybody,

We are currently trying to get Infinispan Cache Metrics via JMX out of Keycloak and into Prometheus. After including the Prometheus JXM Exporter (https://github.com/prometheus/jmx_exporter) and going through a configuration spree, we get warnings in the Keycloak log indicating that the attribute
“rebalancingEnabled” is null/not set for the Infinispan Caches userRevisions, authorizationRevisions, and realmRevisions. The error looks like the following:


07:32:34,276 DEBUG [org.infinispan.jmx.ResourceDMBean] (pool-1-thread-3) Exception while reading value of attribute rebalancingEnabled: java.lang.reflect.InvocationTargetException
07:32:34,276 WARN  [org.infinispan.jmx.ResourceDMBean] (pool-1-thread-3) ISPN000036: Did not find attribute rebalancingEnabled
07:32:34,336 WARN  [org.infinispan.topology.CacheTopologyControlCommand] (pool-1-thread-3) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=authorizationRevisions, type=POLICY_GET_STATUS, sender=keycloak-mssql-6599fb56cb-vhll5, joinInfo=null, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=0}: java.lang.NullPointerException
            at org.infinispan.topology.ClusterTopologyManagerImpl.isRebalancingEnabled(ClusterTopologyManagerImpl.java:628)
            at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:197)
            at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:153)
            at org.infinispan.topology.LocalTopologyManagerImpl.executeOnCoordinator(LocalTopologyManagerImpl.java:606)
            at org.infinispan.topology.LocalTopologyManagerImpl.isCacheRebalancingEnabled(LocalTopologyManagerImpl.java:540)
            at org.infinispan.cache.impl.CacheImpl.isRebalancingEnabled(CacheImpl.java:998)
            at sun.reflect.GeneratedMethodAccessor554.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at org.infinispan.jmx.ResourceDMBean$InvokableSetterBasedMBeanAttributeInfo.invoke(ResourceDMBean.java:394)
            at org.infinispan.jmx.ResourceDMBean.getNamedAttribute(ResourceDMBean.java:298)
            at org.infinispan.jmx.ResourceDMBean.getAttributes(ResourceDMBean.java:197)
            at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttributes(DefaultMBeanServerInterceptor.java:709)
            at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttributes(JmxMBeanServer.java:705)
            at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.scrapeBean(JmxScraper.java:151)
            at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:117)
            at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector.collect(JmxCollector.java:456)
            at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
            at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:216)
            at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:137)
            at io.prometheus.jmx.shaded.io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22)
            at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer$HTTPMetricHandler.handle(HTTPServer.java:59)
            at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
            at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83)
            at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82)
            at sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675)
            at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
            at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:647)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)

These caches are created programmatically, as far as I can see so there is no easy way to change their configuration. Did anybody have a similar problem or can shed some light why these caches miss the “rebalancingEnabled” attribute?

Thanks and best regards,
Sebastian

Mit freundlichen Grüßen / Best regards

Dr.-Ing. Sebastian Schuster

Engineering and Support (INST/ESY1)
Bosch Software Innovations GmbH | Ullsteinstr. 128 | 12109 Berlin | GERMANY | www.bosch-si.com<http://www.bosch-si.com>
Tel. +49 30 726112-485 | Fax +49 30 726112-100 | Sebastian.Schuster at bosch-si.com<mailto:Sebastian.Schuster at bosch-si.com>

Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B
Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung: Dr. Stefan Ferber, Michael Hahn