[jboss-jira] [JBoss JIRA] (WFLY-12504) Memoryleak in Metrics-Endpoint
Brian Stansberry (Jira)
issues at jboss.org
Sat Sep 7 09:10:00 EDT 2019
[ https://issues.jboss.org/browse/WFLY-12504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brian Stansberry reassigned WFLY-12504:
---------------------------------------
Assignee: Richard Opalka (was: Jeff Mesnil)
[~ropalka] I can reproduce this with current WildFly master so it's not a case of a re-report of something already fixed. And it looks like WFLY-12567 is not completely fixed.
This is easy to reproduce; just start WF using the standalone-full-ha.xml config, take a heap dump, point a browser at http://localhost:9990/metrics, keep refreshing for a while, take another dump, and compare.
I attached a jvisualvm comparision like that and it's showing thousands of additional ServiceRegistrationImpl instances.
I randomly picked one and I believe it's a good example and attached a screenshot showing its state. Its name (not shown in the png) is a UUID so highly likely it's one installed by PassiveServiceSupplier. I see that it's state has 'removed=false' and 'pendingDependents=1'. I see that 'removed' will only be set to 'true' if SRI.clear() is called when pendingDependents == 0, and if it doesn't get set to true the SRI will not be removed from the ServiceContainerImpl.registry map.
I *think* the problem here is that ServiceContainerImpl.getOrCreateRegistration calls SRI.addPendingDependent both for cases where it is handling the installation of the service itself, and for cases where it is recording a new dependent for a registration. But the pendingDependents counter only gets decremented when a addDependent gets called; i.e. when dealing with the new dependent. The effect is the initial increment of pendingDependents, when service is being created, never gets an offsetting decrement and AFAICT no SRI will ever be removed.
> Memoryleak in Metrics-Endpoint
> ------------------------------
>
> Key: WFLY-12504
> URL: https://issues.jboss.org/browse/WFLY-12504
> Project: WildFly
> Issue Type: Bug
> Components: MP Metrics
> Affects Versions: 17.0.1.Final
> Environment: Docker Image jboss/keycloak:6.0.1
> Reporter: Mathias Werlitz
> Assignee: Richard Opalka
> Priority: Blocker
> Labels: blocker-WF18
> Attachments: Screen Shot 2019-09-07 at 7.27.24 AM.png, Screen Shot 2019-09-07 at 7.29.02 AM.png
>
>
> I noticed there seems to be a memory leak when calling the /metrics endpoint (with Prometheus). I don't know if this relates to the Keycloak metrics or the general Wildfly base metrics.
> Every request to /metrics seems to consume some more heap memory until a OOME occurs. No other requests are done.
> If the metrics endpoint isn't used at all the server works fine.
> {noformat}
> 15:14:50,951 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([("deployment" => "keycloak-server.war")]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.HashMap.resize(HashMap.java:704)
> at java.util.HashMap.putVal(HashMap.java:629)
> at java.util.HashMap.put(HashMap.java:612)
> at java.util.HashSet.add(HashSet.java:220)
> at org.jboss.as.controller.PathAddress.pathAddress(PathAddress.java:95)
> at org.jboss.as.controller.AbstractOperationContext$OperationId.<init>(AbstractOperationContext.java:1571)
> at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1326)
> at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1321)
> at org.jboss.as.controller.operations.global.ReadAttributeHandler$AuthorizeAttributeReadHandler.doExecuteInternal(ReadAttributeHandler.java:244)
> at org.jboss.as.controller.operations.global.ReadAttributeHandler$AuthorizeAttributeReadHandler.execute(ReadAttributeHandler.java:229)
> at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
> at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
> at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
> at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1412)
> at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:423)
> at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerImpl$$Lambda$553/974545249.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity$$Lambda$554/2003274744.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:255)
> at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient$$Lambda$552/516471669.apply(Unknown Source)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient.executeInModelControllerCl(ModelControllerClientFactoryImpl.java:259)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient.executeOperation(ModelControllerClientFactoryImpl.java:157)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$1$$Lambda$546/98454422.apply(Unknown Source)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.lambda$executeInVm$0(ModelControllerClientFactoryImpl.java:351)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$$Lambda$550/226565561.run(Unknown Source)
> at org.jboss.as.controller.access.InVmAccess.runInVm(InVmAccess.java:85)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.executeInVm(ModelControllerClientFactoryImpl.java:351)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.access$000(ModelControllerClientFactoryImpl.java:59)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$1.executeOperation(ModelControllerClientFactoryImpl.java:82)
> at org.jboss.as.controller.LocalModelControllerClient.execute(LocalModelControllerClient.java:54)
> 15:14:50,955 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) WFLYDS0012: Scan of /opt/jboss/keycloak/standalone/deployments threw Exception: java.lang.RuntimeException: WFLYDS0036: Deployment model operation failed. undefined
> at org.jboss.as.server.deployment.scanner.DefaultDeploymentOperations.getUnrelatedDeployments(DefaultDeploymentOperations.java:109)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1692)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1640)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:589)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:493)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$DeploymentScanRunnable.run(FileSystemDeploymentService.java:255)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at org.jboss.threads.JBossThread.run(JBossThread.java:485)
> 15:14:52,319 ERROR [org.jboss.as.controller.management-operation] (management I/O-1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "pbcast.NAKACK2")
> ]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> at sun.reflect.generics.parser.SignatureParser.parsePackageNameAndSimpleClassTypeSignature(SignatureParser.java:331)
> at sun.reflect.generics.parser.SignatureParser.parseClassTypeSignature(SignatureParser.java:310)
> at sun.reflect.generics.parser.SignatureParser.parseFieldTypeSignature(SignatureParser.java:289)
> at sun.reflect.generics.parser.SignatureParser.parseFieldTypeSignature(SignatureParser.java:283)
> at sun.reflect.generics.parser.SignatureParser.parseTypeSignature(SignatureParser.java:485)
> at sun.reflect.generics.parser.SignatureParser.parseTypeSig(SignatureParser.java:188)
> at sun.reflect.annotation.AnnotationParser.parseSig(AnnotationParser.java:436)
> at sun.reflect.annotation.AnnotationParser.parseAnnotation2(AnnotationParser.java:241)
> at sun.reflect.annotation.AnnotationParser.parseAnnotations2(AnnotationParser.java:120)
> at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:72)
> at java.lang.reflect.Field.declaredAnnotations(Field.java:1150)
> at java.lang.reflect.Field.declaredAnnotations(Field.java:1148)
> at java.lang.reflect.Field.getAnnotation(Field.java:1120)
> at org.jgroups.stack.Configurator.resolveAndAssignField(Configurator.java:926)
> at org.jgroups.stack.Configurator.resolveAndAssignFields(Configurator.java:921)
> at org.jgroups.stack.Protocol.setProperties(Protocol.java:133)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator.lambda$createProtocol$0(AbstractProtocolConfigurationServiceConfigurator.java:119)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator$$Lambda$590/1559509553.run(Unknown Source)
> at org.wildfly.security.manager.WildFlySecurityManager.doUnchecked(WildFlySecurityManager.java:849)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator.createProtocol(AbstractProtocolConfigurationServiceConfigurator.java:125)
> at org.jboss.as.clustering.jgroups.subsystem.ChannelRuntimeResourceRegistration.findProtocol(ChannelRuntimeResourceRegistration.java:87)
> at org.jboss.as.clustering.jgroups.subsystem.ProtocolMetricsHandler.executeRuntimeStep(ProtocolMetricsHandler.java:244)
> at org.jboss.as.controller.AbstractRuntimeOnlyHandler$1.execute(AbstractRuntimeOnlyHandler.java:59)
> at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
> at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
> at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
> at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1412)
> at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:423)
> at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerImpl$$Lambda$553/974545249.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity$$Lambda$554/2003274744.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)
> 15:14:52,322 ERROR [io.undertow.request] (management I/O-1) UT005071: Undertow request failed HttpServerExchange{ GET /metrics request {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8], Connection=[keep-alive], Accept-Language=[de,en-US;q=0.7,en;q=0.3], Cache-Control=[max-age=0], Accept-Encoding=[gzip, deflate], User-Agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0], Upgrade-Insecure-Requests=[1], Host=[192.168.99.100:9990]} response {Access-Control-Allow-Origin=[*], Access-Control-Allow-Headers=[origin, content-type, accept, authorization], Access-Control-Allow-Credentials=[true], Content-Type=[text/plain], Access-Control-Allow-Methods=[GET, POST, PUT, DELETE, OPTIONS, HEAD], Access-Control-Max-Age=[1209600]}}: java.lang.IllegalStateException: WFLYMETRICS0003: Unable to read attribute xmit_table_num_purges on [
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "pbcast.NAKACK2")
> ]: "WFLYCTL0158: Operation handler failed: java.lang.OutOfMemoryError: GC overhead limit exceeded".
> at org.wildfly.extension.microprofile.metrics.MetricCollector.lambda$collectResourceMetrics0$0(MetricCollector.java:143)
> at org.wildfly.extension.microprofile.metrics.PrometheusCollector.collect(PrometheusCollector.java:51)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:147)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:168)
> at io.prometheus.client.CollectorRegistry.metricFamilySamples(CollectorRegistry.java:130)
> at org.wildfly.extension.microprofile.metrics.MetricsContextService$1.handleRequest(MetricsContextService.java:100)
> at org.jboss.as.domain.http.server.security.RealmReadinessHandler.handleRequest(RealmReadinessHandler.java:51)
> at org.jboss.as.domain.http.server.security.ServerErrorReadinessHandler.handleRequest(ServerErrorReadinessHandler.java:35)
> at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:91)
> at io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:211)
> at io.undertow.server.handlers.cache.CacheHandler.handleRequest(CacheHandler.java:92)
> at io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:78)
> at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
> at org.jboss.as.domain.http.server.ManagementHttpRequestHandler.handleRequest(ManagementHttpRequestHandler.java:57)
> at org.jboss.as.domain.http.server.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:75)
> at org.jboss.as.domain.http.server.ManagementHttpServer$UpgradeFixHandler.handleRequest(ManagementHttpServer.java:662)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:364)
> at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
> at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
> 15:14:58,673 ERROR [org.jboss.as.controller.management-operation] (management I/O-2) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "UDP")
> ]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> 15:14:58,675 ERROR [io.undertow.request] (management I/O-2) UT005071: Undertow request failed HttpServerExchange{ GET /metrics request {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8], Connection=[keep-alive], Accept-Language=[de,en-US;q=0.7,en;q=0.3], Cache-Control=[max-age=0], Accept-Encoding=[gzip, deflate], User-Agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0], Upgrade-Insecure-Requests=[1], Host=[192.168.99.100:9990]} response {Access-Control-Allow-Origin=[*], Access-Control-Allow-Headers=[origin, content-type, accept, authorization], Access-Control-Allow-Credentials=[true], Content-Type=[text/plain], Access-Control-Allow-Methods=[GET, POST, PUT, DELETE, OPTIONS, HEAD], Access-Control-Max-Age=[1209600]}}: java.lang.IllegalStateException: WFLYMETRICS0003: Unable to read attribute max_bundle_size on [
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "UDP")
> ]: "WFLYCTL0158: Operation handler failed: java.lang.OutOfMemoryError: GC overhead limit exceeded".
> at org.wildfly.extension.microprofile.metrics.MetricCollector.lambda$collectResourceMetrics0$0(MetricCollector.java:143)
> at org.wildfly.extension.microprofile.metrics.PrometheusCollector.collect(PrometheusCollector.java:51)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:147)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:168)
> at io.prometheus.client.CollectorRegistry.metricFamilySamples(CollectorRegistry.java:130)
> at org.wildfly.extension.microprofile.metrics.MetricsContextService$1.handleRequest(MetricsContextService.java:100)
> at org.jboss.as.domain.http.server.security.RealmReadinessHandler.handleRequest(RealmReadinessHandler.java:51)
> at org.jboss.as.domain.http.server.security.ServerErrorReadinessHandler.handleRequest(ServerErrorReadinessHandler.java:35)
> at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:91)
> at io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:211)
> at io.undertow.server.handlers.cache.CacheHandler.handleRequest(CacheHandler.java:92)
> at io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:78)
> at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
> at org.jboss.as.domain.http.server.ManagementHttpRequestHandler.handleRequest(ManagementHttpRequestHandler.java:57)
> at org.jboss.as.domain.http.server.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:75)
> at org.jboss.as.domain.http.server.ManagementHttpServer$UpgradeFixHandler.handleRequest(ManagementHttpServer.java:662)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:364)
> at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:162)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:100)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:57)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:291)
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.nio.QueuedNioTcpServer$1.run(QueuedNioTcpServer.java:131)
> at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612)
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:479)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
More information about the jboss-jira
mailing list