[JBoss JIRA] (WFLY-13025) Deprecate token=simple and token=digest
by Paul Ferraro (Jira)
[ https://issues.redhat.com/browse/WFLY-13025?page=com.atlassian.jira.plugi... ]
Paul Ferraro closed WFLY-13025.
-------------------------------
Resolution: Rejected
Actually, /token=simple and /token=digest do not use SimpleToken nor MD5Token, but rather use custom token classes.
> Deprecate token=simple and token=digest
> ---------------------------------------
>
> Key: WFLY-13025
> URL: https://issues.redhat.com/browse/WFLY-13025
> Project: WildFly
> Issue Type: Task
> Components: Clustering
> Affects Versions: 19.0.0.Beta1
> Reporter: Paul Ferraro
> Assignee: Paul Ferraro
> Priority: Major
>
> AuthToken and SimpleToken are removed in JGroups 5.0.0.Alpha1. In preparation, we need to deprecate these resources from WildFly.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 5 months
- 1
- 0
[JBoss JIRA] (WFLY-13047) Flooding "not found in retransmission table" in server log too much in cluster and making application slow.
by Paul Ferraro (Jira)
[ https://issues.redhat.com/browse/WFLY-13047?page=com.atlassian.jira.plugi... ]
Paul Ferraro closed WFLY-13047.
-------------------------------
Resolution: Out of Date
This was already fixed by https://github.com/wildfly/wildfly/commit/106ed3000ad120ff586905c0e6b5768...
> Flooding "not found in retransmission table" in server log too much in cluster and making application slow.
> -----------------------------------------------------------------------------------------------------------
>
> Key: WFLY-13047
> URL: https://issues.redhat.com/browse/WFLY-13047
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 11.0.0.Final
> Reporter: Dheeraj Mishra
> Assignee: Paul Ferraro
> Priority: Major
> Attachments: log3.zip, standalone.xml
>
>
> We have 24 JVM cluster using multicast and observing below error too much in server.log and it is making application slower. attaching logs for your reference.
> 2020-02-02 02:22:07,839 WARN [org.jboss.as.clustering.jgroups.protocol.NAKACK2] (thread-1) JGRP000041: sol120_7433: message sol120_7403::1425 not found in retransmission table
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 5 months
- 1
- 0
[JBoss JIRA] (WFLY-13069) CoarseDatabasePersistenceWebFailoverTestCase fails intermittently
by Brian Stansberry (Jira)
[ https://issues.redhat.com/browse/WFLY-13069?page=com.atlassian.jira.plugi... ]
Brian Stansberry updated WFLY-13069:
------------------------------------
Component/s: Clustering
> CoarseDatabasePersistenceWebFailoverTestCase fails intermittently
> -----------------------------------------------------------------
>
> Key: WFLY-13069
> URL: https://issues.redhat.com/browse/WFLY-13069
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Reporter: Yeray Borges
> Priority: Major
>
> The following jobs kicked off by different PRs hit failures on CoarseDatabasePersistenceWebFailoverTestCase:
> https://ci.wildfly.org/viewLog.html?buildId=185412&tab=buildResultsDiv&bu...
> https://ci.wildfly.org/viewLog.html?buildId=185397&tab=buildResultsDiv&bu...
> https://ci.wildfly.org/viewLog.html?buildId=184962&tab=buildResultsDiv&bu...
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 5 months
- 1
- 0
[JBoss JIRA] (WFLY-12504) Memoryleak in Metrics-Endpoint
by Brian Stansberry (Jira)
[ https://issues.redhat.com/browse/WFLY-12504?page=com.atlassian.jira.plugi... ]
Brian Stansberry commented on WFLY-12504:
-----------------------------------------
[~thradec] If you are still seeing problems, please file a new issue. Thanks!
> Memoryleak in Metrics-Endpoint
> ------------------------------
>
> Key: WFLY-12504
> URL: https://issues.redhat.com/browse/WFLY-12504
> Project: WildFly
> Issue Type: Bug
> Components: MP Metrics
> Affects Versions: 17.0.1.Final
> Environment: Docker Image jboss/keycloak:6.0.1
> Reporter: Mathias Werlitz
> Assignee: Richard Opalka
> Priority: Blocker
> Labels: blocker-WF18, fixed_needs_core
> Fix For: 18.0.0.Final
>
> Attachments: Screen Shot 2019-09-07 at 7.27.24 AM.png, Screen Shot 2019-09-07 at 7.29.02 AM.png
>
>
> I noticed there seems to be a memory leak when calling the /metrics endpoint (with Prometheus). I don't know if this relates to the Keycloak metrics or the general Wildfly base metrics.
> Every request to /metrics seems to consume some more heap memory until a OOME occurs. No other requests are done.
> If the metrics endpoint isn't used at all the server works fine.
> {noformat}
> 15:14:50,951 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([("deployment" => "keycloak-server.war")]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.HashMap.resize(HashMap.java:704)
> at java.util.HashMap.putVal(HashMap.java:629)
> at java.util.HashMap.put(HashMap.java:612)
> at java.util.HashSet.add(HashSet.java:220)
> at org.jboss.as.controller.PathAddress.pathAddress(PathAddress.java:95)
> at org.jboss.as.controller.AbstractOperationContext$OperationId.<init>(AbstractOperationContext.java:1571)
> at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1326)
> at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1321)
> at org.jboss.as.controller.operations.global.ReadAttributeHandler$AuthorizeAttributeReadHandler.doExecuteInternal(ReadAttributeHandler.java:244)
> at org.jboss.as.controller.operations.global.ReadAttributeHandler$AuthorizeAttributeReadHandler.execute(ReadAttributeHandler.java:229)
> at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
> at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
> at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
> at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1412)
> at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:423)
> at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerImpl$$Lambda$553/974545249.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity$$Lambda$554/2003274744.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:255)
> at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient$$Lambda$552/516471669.apply(Unknown Source)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient.executeInModelControllerCl(ModelControllerClientFactoryImpl.java:259)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient.executeOperation(ModelControllerClientFactoryImpl.java:157)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$1$$Lambda$546/98454422.apply(Unknown Source)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.lambda$executeInVm$0(ModelControllerClientFactoryImpl.java:351)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$$Lambda$550/226565561.run(Unknown Source)
> at org.jboss.as.controller.access.InVmAccess.runInVm(InVmAccess.java:85)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.executeInVm(ModelControllerClientFactoryImpl.java:351)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.access$000(ModelControllerClientFactoryImpl.java:59)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$1.executeOperation(ModelControllerClientFactoryImpl.java:82)
> at org.jboss.as.controller.LocalModelControllerClient.execute(LocalModelControllerClient.java:54)
> 15:14:50,955 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) WFLYDS0012: Scan of /opt/jboss/keycloak/standalone/deployments threw Exception: java.lang.RuntimeException: WFLYDS0036: Deployment model operation failed. undefined
> at org.jboss.as.server.deployment.scanner.DefaultDeploymentOperations.getUnrelatedDeployments(DefaultDeploymentOperations.java:109)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1692)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1640)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:589)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:493)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$DeploymentScanRunnable.run(FileSystemDeploymentService.java:255)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at org.jboss.threads.JBossThread.run(JBossThread.java:485)
> 15:14:52,319 ERROR [org.jboss.as.controller.management-operation] (management I/O-1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "pbcast.NAKACK2")
> ]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> at sun.reflect.generics.parser.SignatureParser.parsePackageNameAndSimpleClassTypeSignature(SignatureParser.java:331)
> at sun.reflect.generics.parser.SignatureParser.parseClassTypeSignature(SignatureParser.java:310)
> at sun.reflect.generics.parser.SignatureParser.parseFieldTypeSignature(SignatureParser.java:289)
> at sun.reflect.generics.parser.SignatureParser.parseFieldTypeSignature(SignatureParser.java:283)
> at sun.reflect.generics.parser.SignatureParser.parseTypeSignature(SignatureParser.java:485)
> at sun.reflect.generics.parser.SignatureParser.parseTypeSig(SignatureParser.java:188)
> at sun.reflect.annotation.AnnotationParser.parseSig(AnnotationParser.java:436)
> at sun.reflect.annotation.AnnotationParser.parseAnnotation2(AnnotationParser.java:241)
> at sun.reflect.annotation.AnnotationParser.parseAnnotations2(AnnotationParser.java:120)
> at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:72)
> at java.lang.reflect.Field.declaredAnnotations(Field.java:1150)
> at java.lang.reflect.Field.declaredAnnotations(Field.java:1148)
> at java.lang.reflect.Field.getAnnotation(Field.java:1120)
> at org.jgroups.stack.Configurator.resolveAndAssignField(Configurator.java:926)
> at org.jgroups.stack.Configurator.resolveAndAssignFields(Configurator.java:921)
> at org.jgroups.stack.Protocol.setProperties(Protocol.java:133)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator.lambda$createProtocol$0(AbstractProtocolConfigurationServiceConfigurator.java:119)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator$$Lambda$590/1559509553.run(Unknown Source)
> at org.wildfly.security.manager.WildFlySecurityManager.doUnchecked(WildFlySecurityManager.java:849)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator.createProtocol(AbstractProtocolConfigurationServiceConfigurator.java:125)
> at org.jboss.as.clustering.jgroups.subsystem.ChannelRuntimeResourceRegistration.findProtocol(ChannelRuntimeResourceRegistration.java:87)
> at org.jboss.as.clustering.jgroups.subsystem.ProtocolMetricsHandler.executeRuntimeStep(ProtocolMetricsHandler.java:244)
> at org.jboss.as.controller.AbstractRuntimeOnlyHandler$1.execute(AbstractRuntimeOnlyHandler.java:59)
> at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
> at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
> at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
> at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1412)
> at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:423)
> at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerImpl$$Lambda$553/974545249.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity$$Lambda$554/2003274744.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)
> 15:14:52,322 ERROR [io.undertow.request] (management I/O-1) UT005071: Undertow request failed HttpServerExchange{ GET /metrics request {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8], Connection=[keep-alive], Accept-Language=[de,en-US;q=0.7,en;q=0.3], Cache-Control=[max-age=0], Accept-Encoding=[gzip, deflate], User-Agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0], Upgrade-Insecure-Requests=[1], Host=[192.168.99.100:9990]} response {Access-Control-Allow-Origin=[*], Access-Control-Allow-Headers=[origin, content-type, accept, authorization], Access-Control-Allow-Credentials=[true], Content-Type=[text/plain], Access-Control-Allow-Methods=[GET, POST, PUT, DELETE, OPTIONS, HEAD], Access-Control-Max-Age=[1209600]}}: java.lang.IllegalStateException: WFLYMETRICS0003: Unable to read attribute xmit_table_num_purges on [
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "pbcast.NAKACK2")
> ]: "WFLYCTL0158: Operation handler failed: java.lang.OutOfMemoryError: GC overhead limit exceeded".
> at org.wildfly.extension.microprofile.metrics.MetricCollector.lambda$collectResourceMetrics0$0(MetricCollector.java:143)
> at org.wildfly.extension.microprofile.metrics.PrometheusCollector.collect(PrometheusCollector.java:51)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:147)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:168)
> at io.prometheus.client.CollectorRegistry.metricFamilySamples(CollectorRegistry.java:130)
> at org.wildfly.extension.microprofile.metrics.MetricsContextService$1.handleRequest(MetricsContextService.java:100)
> at org.jboss.as.domain.http.server.security.RealmReadinessHandler.handleRequest(RealmReadinessHandler.java:51)
> at org.jboss.as.domain.http.server.security.ServerErrorReadinessHandler.handleRequest(ServerErrorReadinessHandler.java:35)
> at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:91)
> at io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:211)
> at io.undertow.server.handlers.cache.CacheHandler.handleRequest(CacheHandler.java:92)
> at io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:78)
> at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
> at org.jboss.as.domain.http.server.ManagementHttpRequestHandler.handleRequest(ManagementHttpRequestHandler.java:57)
> at org.jboss.as.domain.http.server.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:75)
> at org.jboss.as.domain.http.server.ManagementHttpServer$UpgradeFixHandler.handleRequest(ManagementHttpServer.java:662)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:364)
> at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
> at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
> 15:14:58,673 ERROR [org.jboss.as.controller.management-operation] (management I/O-2) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "UDP")
> ]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> 15:14:58,675 ERROR [io.undertow.request] (management I/O-2) UT005071: Undertow request failed HttpServerExchange{ GET /metrics request {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8], Connection=[keep-alive], Accept-Language=[de,en-US;q=0.7,en;q=0.3], Cache-Control=[max-age=0], Accept-Encoding=[gzip, deflate], User-Agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0], Upgrade-Insecure-Requests=[1], Host=[192.168.99.100:9990]} response {Access-Control-Allow-Origin=[*], Access-Control-Allow-Headers=[origin, content-type, accept, authorization], Access-Control-Allow-Credentials=[true], Content-Type=[text/plain], Access-Control-Allow-Methods=[GET, POST, PUT, DELETE, OPTIONS, HEAD], Access-Control-Max-Age=[1209600]}}: java.lang.IllegalStateException: WFLYMETRICS0003: Unable to read attribute max_bundle_size on [
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "UDP")
> ]: "WFLYCTL0158: Operation handler failed: java.lang.OutOfMemoryError: GC overhead limit exceeded".
> at org.wildfly.extension.microprofile.metrics.MetricCollector.lambda$collectResourceMetrics0$0(MetricCollector.java:143)
> at org.wildfly.extension.microprofile.metrics.PrometheusCollector.collect(PrometheusCollector.java:51)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:147)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:168)
> at io.prometheus.client.CollectorRegistry.metricFamilySamples(CollectorRegistry.java:130)
> at org.wildfly.extension.microprofile.metrics.MetricsContextService$1.handleRequest(MetricsContextService.java:100)
> at org.jboss.as.domain.http.server.security.RealmReadinessHandler.handleRequest(RealmReadinessHandler.java:51)
> at org.jboss.as.domain.http.server.security.ServerErrorReadinessHandler.handleRequest(ServerErrorReadinessHandler.java:35)
> at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:91)
> at io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:211)
> at io.undertow.server.handlers.cache.CacheHandler.handleRequest(CacheHandler.java:92)
> at io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:78)
> at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
> at org.jboss.as.domain.http.server.ManagementHttpRequestHandler.handleRequest(ManagementHttpRequestHandler.java:57)
> at org.jboss.as.domain.http.server.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:75)
> at org.jboss.as.domain.http.server.ManagementHttpServer$UpgradeFixHandler.handleRequest(ManagementHttpServer.java:662)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:364)
> at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:162)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:100)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:57)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:291)
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.nio.QueuedNioTcpServer$1.run(QueuedNioTcpServer.java:131)
> at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612)
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:479)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 5 months
- 1
- 0
[JBoss JIRA] (WFLY-12504) Memoryleak in Metrics-Endpoint
by Brian Stansberry (Jira)
[ https://issues.redhat.com/browse/WFLY-12504?page=com.atlassian.jira.plugi... ]
Brian Stansberry closed WFLY-12504.
-----------------------------------
> Memoryleak in Metrics-Endpoint
> ------------------------------
>
> Key: WFLY-12504
> URL: https://issues.redhat.com/browse/WFLY-12504
> Project: WildFly
> Issue Type: Bug
> Components: MP Metrics
> Affects Versions: 17.0.1.Final
> Environment: Docker Image jboss/keycloak:6.0.1
> Reporter: Mathias Werlitz
> Assignee: Richard Opalka
> Priority: Blocker
> Labels: blocker-WF18, fixed_needs_core
> Fix For: 18.0.0.Final
>
> Attachments: Screen Shot 2019-09-07 at 7.27.24 AM.png, Screen Shot 2019-09-07 at 7.29.02 AM.png
>
>
> I noticed there seems to be a memory leak when calling the /metrics endpoint (with Prometheus). I don't know if this relates to the Keycloak metrics or the general Wildfly base metrics.
> Every request to /metrics seems to consume some more heap memory until a OOME occurs. No other requests are done.
> If the metrics endpoint isn't used at all the server works fine.
> {noformat}
> 15:14:50,951 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([("deployment" => "keycloak-server.war")]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.HashMap.resize(HashMap.java:704)
> at java.util.HashMap.putVal(HashMap.java:629)
> at java.util.HashMap.put(HashMap.java:612)
> at java.util.HashSet.add(HashSet.java:220)
> at org.jboss.as.controller.PathAddress.pathAddress(PathAddress.java:95)
> at org.jboss.as.controller.AbstractOperationContext$OperationId.<init>(AbstractOperationContext.java:1571)
> at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1326)
> at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1321)
> at org.jboss.as.controller.operations.global.ReadAttributeHandler$AuthorizeAttributeReadHandler.doExecuteInternal(ReadAttributeHandler.java:244)
> at org.jboss.as.controller.operations.global.ReadAttributeHandler$AuthorizeAttributeReadHandler.execute(ReadAttributeHandler.java:229)
> at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
> at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
> at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
> at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1412)
> at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:423)
> at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerImpl$$Lambda$553/974545249.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity$$Lambda$554/2003274744.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:255)
> at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient$$Lambda$552/516471669.apply(Unknown Source)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient.executeInModelControllerCl(ModelControllerClientFactoryImpl.java:259)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient.executeOperation(ModelControllerClientFactoryImpl.java:157)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$1$$Lambda$546/98454422.apply(Unknown Source)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.lambda$executeInVm$0(ModelControllerClientFactoryImpl.java:351)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$$Lambda$550/226565561.run(Unknown Source)
> at org.jboss.as.controller.access.InVmAccess.runInVm(InVmAccess.java:85)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.executeInVm(ModelControllerClientFactoryImpl.java:351)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.access$000(ModelControllerClientFactoryImpl.java:59)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$1.executeOperation(ModelControllerClientFactoryImpl.java:82)
> at org.jboss.as.controller.LocalModelControllerClient.execute(LocalModelControllerClient.java:54)
> 15:14:50,955 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) WFLYDS0012: Scan of /opt/jboss/keycloak/standalone/deployments threw Exception: java.lang.RuntimeException: WFLYDS0036: Deployment model operation failed. undefined
> at org.jboss.as.server.deployment.scanner.DefaultDeploymentOperations.getUnrelatedDeployments(DefaultDeploymentOperations.java:109)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1692)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1640)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:589)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:493)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$DeploymentScanRunnable.run(FileSystemDeploymentService.java:255)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at org.jboss.threads.JBossThread.run(JBossThread.java:485)
> 15:14:52,319 ERROR [org.jboss.as.controller.management-operation] (management I/O-1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "pbcast.NAKACK2")
> ]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> at sun.reflect.generics.parser.SignatureParser.parsePackageNameAndSimpleClassTypeSignature(SignatureParser.java:331)
> at sun.reflect.generics.parser.SignatureParser.parseClassTypeSignature(SignatureParser.java:310)
> at sun.reflect.generics.parser.SignatureParser.parseFieldTypeSignature(SignatureParser.java:289)
> at sun.reflect.generics.parser.SignatureParser.parseFieldTypeSignature(SignatureParser.java:283)
> at sun.reflect.generics.parser.SignatureParser.parseTypeSignature(SignatureParser.java:485)
> at sun.reflect.generics.parser.SignatureParser.parseTypeSig(SignatureParser.java:188)
> at sun.reflect.annotation.AnnotationParser.parseSig(AnnotationParser.java:436)
> at sun.reflect.annotation.AnnotationParser.parseAnnotation2(AnnotationParser.java:241)
> at sun.reflect.annotation.AnnotationParser.parseAnnotations2(AnnotationParser.java:120)
> at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:72)
> at java.lang.reflect.Field.declaredAnnotations(Field.java:1150)
> at java.lang.reflect.Field.declaredAnnotations(Field.java:1148)
> at java.lang.reflect.Field.getAnnotation(Field.java:1120)
> at org.jgroups.stack.Configurator.resolveAndAssignField(Configurator.java:926)
> at org.jgroups.stack.Configurator.resolveAndAssignFields(Configurator.java:921)
> at org.jgroups.stack.Protocol.setProperties(Protocol.java:133)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator.lambda$createProtocol$0(AbstractProtocolConfigurationServiceConfigurator.java:119)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator$$Lambda$590/1559509553.run(Unknown Source)
> at org.wildfly.security.manager.WildFlySecurityManager.doUnchecked(WildFlySecurityManager.java:849)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator.createProtocol(AbstractProtocolConfigurationServiceConfigurator.java:125)
> at org.jboss.as.clustering.jgroups.subsystem.ChannelRuntimeResourceRegistration.findProtocol(ChannelRuntimeResourceRegistration.java:87)
> at org.jboss.as.clustering.jgroups.subsystem.ProtocolMetricsHandler.executeRuntimeStep(ProtocolMetricsHandler.java:244)
> at org.jboss.as.controller.AbstractRuntimeOnlyHandler$1.execute(AbstractRuntimeOnlyHandler.java:59)
> at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
> at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
> at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
> at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1412)
> at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:423)
> at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerImpl$$Lambda$553/974545249.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity$$Lambda$554/2003274744.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)
> 15:14:52,322 ERROR [io.undertow.request] (management I/O-1) UT005071: Undertow request failed HttpServerExchange{ GET /metrics request {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8], Connection=[keep-alive], Accept-Language=[de,en-US;q=0.7,en;q=0.3], Cache-Control=[max-age=0], Accept-Encoding=[gzip, deflate], User-Agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0], Upgrade-Insecure-Requests=[1], Host=[192.168.99.100:9990]} response {Access-Control-Allow-Origin=[*], Access-Control-Allow-Headers=[origin, content-type, accept, authorization], Access-Control-Allow-Credentials=[true], Content-Type=[text/plain], Access-Control-Allow-Methods=[GET, POST, PUT, DELETE, OPTIONS, HEAD], Access-Control-Max-Age=[1209600]}}: java.lang.IllegalStateException: WFLYMETRICS0003: Unable to read attribute xmit_table_num_purges on [
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "pbcast.NAKACK2")
> ]: "WFLYCTL0158: Operation handler failed: java.lang.OutOfMemoryError: GC overhead limit exceeded".
> at org.wildfly.extension.microprofile.metrics.MetricCollector.lambda$collectResourceMetrics0$0(MetricCollector.java:143)
> at org.wildfly.extension.microprofile.metrics.PrometheusCollector.collect(PrometheusCollector.java:51)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:147)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:168)
> at io.prometheus.client.CollectorRegistry.metricFamilySamples(CollectorRegistry.java:130)
> at org.wildfly.extension.microprofile.metrics.MetricsContextService$1.handleRequest(MetricsContextService.java:100)
> at org.jboss.as.domain.http.server.security.RealmReadinessHandler.handleRequest(RealmReadinessHandler.java:51)
> at org.jboss.as.domain.http.server.security.ServerErrorReadinessHandler.handleRequest(ServerErrorReadinessHandler.java:35)
> at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:91)
> at io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:211)
> at io.undertow.server.handlers.cache.CacheHandler.handleRequest(CacheHandler.java:92)
> at io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:78)
> at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
> at org.jboss.as.domain.http.server.ManagementHttpRequestHandler.handleRequest(ManagementHttpRequestHandler.java:57)
> at org.jboss.as.domain.http.server.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:75)
> at org.jboss.as.domain.http.server.ManagementHttpServer$UpgradeFixHandler.handleRequest(ManagementHttpServer.java:662)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:364)
> at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
> at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
> 15:14:58,673 ERROR [org.jboss.as.controller.management-operation] (management I/O-2) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "UDP")
> ]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> 15:14:58,675 ERROR [io.undertow.request] (management I/O-2) UT005071: Undertow request failed HttpServerExchange{ GET /metrics request {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8], Connection=[keep-alive], Accept-Language=[de,en-US;q=0.7,en;q=0.3], Cache-Control=[max-age=0], Accept-Encoding=[gzip, deflate], User-Agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0], Upgrade-Insecure-Requests=[1], Host=[192.168.99.100:9990]} response {Access-Control-Allow-Origin=[*], Access-Control-Allow-Headers=[origin, content-type, accept, authorization], Access-Control-Allow-Credentials=[true], Content-Type=[text/plain], Access-Control-Allow-Methods=[GET, POST, PUT, DELETE, OPTIONS, HEAD], Access-Control-Max-Age=[1209600]}}: java.lang.IllegalStateException: WFLYMETRICS0003: Unable to read attribute max_bundle_size on [
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "UDP")
> ]: "WFLYCTL0158: Operation handler failed: java.lang.OutOfMemoryError: GC overhead limit exceeded".
> at org.wildfly.extension.microprofile.metrics.MetricCollector.lambda$collectResourceMetrics0$0(MetricCollector.java:143)
> at org.wildfly.extension.microprofile.metrics.PrometheusCollector.collect(PrometheusCollector.java:51)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:147)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:168)
> at io.prometheus.client.CollectorRegistry.metricFamilySamples(CollectorRegistry.java:130)
> at org.wildfly.extension.microprofile.metrics.MetricsContextService$1.handleRequest(MetricsContextService.java:100)
> at org.jboss.as.domain.http.server.security.RealmReadinessHandler.handleRequest(RealmReadinessHandler.java:51)
> at org.jboss.as.domain.http.server.security.ServerErrorReadinessHandler.handleRequest(ServerErrorReadinessHandler.java:35)
> at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:91)
> at io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:211)
> at io.undertow.server.handlers.cache.CacheHandler.handleRequest(CacheHandler.java:92)
> at io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:78)
> at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
> at org.jboss.as.domain.http.server.ManagementHttpRequestHandler.handleRequest(ManagementHttpRequestHandler.java:57)
> at org.jboss.as.domain.http.server.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:75)
> at org.jboss.as.domain.http.server.ManagementHttpServer$UpgradeFixHandler.handleRequest(ManagementHttpServer.java:662)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:364)
> at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:162)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:100)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:57)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:291)
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.nio.QueuedNioTcpServer$1.run(QueuedNioTcpServer.java:131)
> at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612)
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:479)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 5 months
- 1
- 0
[JBoss JIRA] (WFLY-12504) Memoryleak in Metrics-Endpoint
by Brian Stansberry (Jira)
[ https://issues.redhat.com/browse/WFLY-12504?page=com.atlassian.jira.plugi... ]
Brian Stansberry resolved WFLY-12504.
-------------------------------------
Fix Version/s: 18.0.0.Final
Resolution: Done
> Memoryleak in Metrics-Endpoint
> ------------------------------
>
> Key: WFLY-12504
> URL: https://issues.redhat.com/browse/WFLY-12504
> Project: WildFly
> Issue Type: Bug
> Components: MP Metrics
> Affects Versions: 17.0.1.Final
> Environment: Docker Image jboss/keycloak:6.0.1
> Reporter: Mathias Werlitz
> Assignee: Richard Opalka
> Priority: Blocker
> Labels: blocker-WF18, fixed_needs_core
> Fix For: 18.0.0.Final
>
> Attachments: Screen Shot 2019-09-07 at 7.27.24 AM.png, Screen Shot 2019-09-07 at 7.29.02 AM.png
>
>
> I noticed there seems to be a memory leak when calling the /metrics endpoint (with Prometheus). I don't know if this relates to the Keycloak metrics or the general Wildfly base metrics.
> Every request to /metrics seems to consume some more heap memory until a OOME occurs. No other requests are done.
> If the metrics endpoint isn't used at all the server works fine.
> {noformat}
> 15:14:50,951 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([("deployment" => "keycloak-server.war")]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.HashMap.resize(HashMap.java:704)
> at java.util.HashMap.putVal(HashMap.java:629)
> at java.util.HashMap.put(HashMap.java:612)
> at java.util.HashSet.add(HashSet.java:220)
> at org.jboss.as.controller.PathAddress.pathAddress(PathAddress.java:95)
> at org.jboss.as.controller.AbstractOperationContext$OperationId.<init>(AbstractOperationContext.java:1571)
> at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1326)
> at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1321)
> at org.jboss.as.controller.operations.global.ReadAttributeHandler$AuthorizeAttributeReadHandler.doExecuteInternal(ReadAttributeHandler.java:244)
> at org.jboss.as.controller.operations.global.ReadAttributeHandler$AuthorizeAttributeReadHandler.execute(ReadAttributeHandler.java:229)
> at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
> at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
> at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
> at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1412)
> at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:423)
> at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerImpl$$Lambda$553/974545249.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity$$Lambda$554/2003274744.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:255)
> at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient$$Lambda$552/516471669.apply(Unknown Source)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient.executeInModelControllerCl(ModelControllerClientFactoryImpl.java:259)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient.executeOperation(ModelControllerClientFactoryImpl.java:157)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$1$$Lambda$546/98454422.apply(Unknown Source)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.lambda$executeInVm$0(ModelControllerClientFactoryImpl.java:351)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$$Lambda$550/226565561.run(Unknown Source)
> at org.jboss.as.controller.access.InVmAccess.runInVm(InVmAccess.java:85)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.executeInVm(ModelControllerClientFactoryImpl.java:351)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.access$000(ModelControllerClientFactoryImpl.java:59)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$1.executeOperation(ModelControllerClientFactoryImpl.java:82)
> at org.jboss.as.controller.LocalModelControllerClient.execute(LocalModelControllerClient.java:54)
> 15:14:50,955 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) WFLYDS0012: Scan of /opt/jboss/keycloak/standalone/deployments threw Exception: java.lang.RuntimeException: WFLYDS0036: Deployment model operation failed. undefined
> at org.jboss.as.server.deployment.scanner.DefaultDeploymentOperations.getUnrelatedDeployments(DefaultDeploymentOperations.java:109)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1692)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1640)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:589)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:493)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$DeploymentScanRunnable.run(FileSystemDeploymentService.java:255)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at org.jboss.threads.JBossThread.run(JBossThread.java:485)
> 15:14:52,319 ERROR [org.jboss.as.controller.management-operation] (management I/O-1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "pbcast.NAKACK2")
> ]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> at sun.reflect.generics.parser.SignatureParser.parsePackageNameAndSimpleClassTypeSignature(SignatureParser.java:331)
> at sun.reflect.generics.parser.SignatureParser.parseClassTypeSignature(SignatureParser.java:310)
> at sun.reflect.generics.parser.SignatureParser.parseFieldTypeSignature(SignatureParser.java:289)
> at sun.reflect.generics.parser.SignatureParser.parseFieldTypeSignature(SignatureParser.java:283)
> at sun.reflect.generics.parser.SignatureParser.parseTypeSignature(SignatureParser.java:485)
> at sun.reflect.generics.parser.SignatureParser.parseTypeSig(SignatureParser.java:188)
> at sun.reflect.annotation.AnnotationParser.parseSig(AnnotationParser.java:436)
> at sun.reflect.annotation.AnnotationParser.parseAnnotation2(AnnotationParser.java:241)
> at sun.reflect.annotation.AnnotationParser.parseAnnotations2(AnnotationParser.java:120)
> at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:72)
> at java.lang.reflect.Field.declaredAnnotations(Field.java:1150)
> at java.lang.reflect.Field.declaredAnnotations(Field.java:1148)
> at java.lang.reflect.Field.getAnnotation(Field.java:1120)
> at org.jgroups.stack.Configurator.resolveAndAssignField(Configurator.java:926)
> at org.jgroups.stack.Configurator.resolveAndAssignFields(Configurator.java:921)
> at org.jgroups.stack.Protocol.setProperties(Protocol.java:133)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator.lambda$createProtocol$0(AbstractProtocolConfigurationServiceConfigurator.java:119)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator$$Lambda$590/1559509553.run(Unknown Source)
> at org.wildfly.security.manager.WildFlySecurityManager.doUnchecked(WildFlySecurityManager.java:849)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator.createProtocol(AbstractProtocolConfigurationServiceConfigurator.java:125)
> at org.jboss.as.clustering.jgroups.subsystem.ChannelRuntimeResourceRegistration.findProtocol(ChannelRuntimeResourceRegistration.java:87)
> at org.jboss.as.clustering.jgroups.subsystem.ProtocolMetricsHandler.executeRuntimeStep(ProtocolMetricsHandler.java:244)
> at org.jboss.as.controller.AbstractRuntimeOnlyHandler$1.execute(AbstractRuntimeOnlyHandler.java:59)
> at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
> at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
> at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
> at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1412)
> at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:423)
> at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerImpl$$Lambda$553/974545249.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity$$Lambda$554/2003274744.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)
> 15:14:52,322 ERROR [io.undertow.request] (management I/O-1) UT005071: Undertow request failed HttpServerExchange{ GET /metrics request {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8], Connection=[keep-alive], Accept-Language=[de,en-US;q=0.7,en;q=0.3], Cache-Control=[max-age=0], Accept-Encoding=[gzip, deflate], User-Agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0], Upgrade-Insecure-Requests=[1], Host=[192.168.99.100:9990]} response {Access-Control-Allow-Origin=[*], Access-Control-Allow-Headers=[origin, content-type, accept, authorization], Access-Control-Allow-Credentials=[true], Content-Type=[text/plain], Access-Control-Allow-Methods=[GET, POST, PUT, DELETE, OPTIONS, HEAD], Access-Control-Max-Age=[1209600]}}: java.lang.IllegalStateException: WFLYMETRICS0003: Unable to read attribute xmit_table_num_purges on [
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "pbcast.NAKACK2")
> ]: "WFLYCTL0158: Operation handler failed: java.lang.OutOfMemoryError: GC overhead limit exceeded".
> at org.wildfly.extension.microprofile.metrics.MetricCollector.lambda$collectResourceMetrics0$0(MetricCollector.java:143)
> at org.wildfly.extension.microprofile.metrics.PrometheusCollector.collect(PrometheusCollector.java:51)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:147)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:168)
> at io.prometheus.client.CollectorRegistry.metricFamilySamples(CollectorRegistry.java:130)
> at org.wildfly.extension.microprofile.metrics.MetricsContextService$1.handleRequest(MetricsContextService.java:100)
> at org.jboss.as.domain.http.server.security.RealmReadinessHandler.handleRequest(RealmReadinessHandler.java:51)
> at org.jboss.as.domain.http.server.security.ServerErrorReadinessHandler.handleRequest(ServerErrorReadinessHandler.java:35)
> at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:91)
> at io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:211)
> at io.undertow.server.handlers.cache.CacheHandler.handleRequest(CacheHandler.java:92)
> at io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:78)
> at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
> at org.jboss.as.domain.http.server.ManagementHttpRequestHandler.handleRequest(ManagementHttpRequestHandler.java:57)
> at org.jboss.as.domain.http.server.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:75)
> at org.jboss.as.domain.http.server.ManagementHttpServer$UpgradeFixHandler.handleRequest(ManagementHttpServer.java:662)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:364)
> at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
> at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
> 15:14:58,673 ERROR [org.jboss.as.controller.management-operation] (management I/O-2) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "UDP")
> ]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> 15:14:58,675 ERROR [io.undertow.request] (management I/O-2) UT005071: Undertow request failed HttpServerExchange{ GET /metrics request {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8], Connection=[keep-alive], Accept-Language=[de,en-US;q=0.7,en;q=0.3], Cache-Control=[max-age=0], Accept-Encoding=[gzip, deflate], User-Agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0], Upgrade-Insecure-Requests=[1], Host=[192.168.99.100:9990]} response {Access-Control-Allow-Origin=[*], Access-Control-Allow-Headers=[origin, content-type, accept, authorization], Access-Control-Allow-Credentials=[true], Content-Type=[text/plain], Access-Control-Allow-Methods=[GET, POST, PUT, DELETE, OPTIONS, HEAD], Access-Control-Max-Age=[1209600]}}: java.lang.IllegalStateException: WFLYMETRICS0003: Unable to read attribute max_bundle_size on [
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "UDP")
> ]: "WFLYCTL0158: Operation handler failed: java.lang.OutOfMemoryError: GC overhead limit exceeded".
> at org.wildfly.extension.microprofile.metrics.MetricCollector.lambda$collectResourceMetrics0$0(MetricCollector.java:143)
> at org.wildfly.extension.microprofile.metrics.PrometheusCollector.collect(PrometheusCollector.java:51)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:147)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:168)
> at io.prometheus.client.CollectorRegistry.metricFamilySamples(CollectorRegistry.java:130)
> at org.wildfly.extension.microprofile.metrics.MetricsContextService$1.handleRequest(MetricsContextService.java:100)
> at org.jboss.as.domain.http.server.security.RealmReadinessHandler.handleRequest(RealmReadinessHandler.java:51)
> at org.jboss.as.domain.http.server.security.ServerErrorReadinessHandler.handleRequest(ServerErrorReadinessHandler.java:35)
> at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:91)
> at io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:211)
> at io.undertow.server.handlers.cache.CacheHandler.handleRequest(CacheHandler.java:92)
> at io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:78)
> at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
> at org.jboss.as.domain.http.server.ManagementHttpRequestHandler.handleRequest(ManagementHttpRequestHandler.java:57)
> at org.jboss.as.domain.http.server.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:75)
> at org.jboss.as.domain.http.server.ManagementHttpServer$UpgradeFixHandler.handleRequest(ManagementHttpServer.java:662)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:364)
> at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:162)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:100)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:57)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:291)
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.nio.QueuedNioTcpServer$1.run(QueuedNioTcpServer.java:131)
> at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612)
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:479)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 5 months
- 1
- 0
[JBoss JIRA] (WFLY-12504) Memoryleak in Metrics-Endpoint
by Brian Stansberry (Jira)
[ https://issues.redhat.com/browse/WFLY-12504?page=com.atlassian.jira.plugi... ]
Brian Stansberry reopened WFLY-12504:
-------------------------------------
> Memoryleak in Metrics-Endpoint
> ------------------------------
>
> Key: WFLY-12504
> URL: https://issues.redhat.com/browse/WFLY-12504
> Project: WildFly
> Issue Type: Bug
> Components: MP Metrics
> Affects Versions: 17.0.1.Final
> Environment: Docker Image jboss/keycloak:6.0.1
> Reporter: Mathias Werlitz
> Assignee: Richard Opalka
> Priority: Blocker
> Labels: blocker-WF18, fixed_needs_core
> Fix For: 18.0.0.Final
>
> Attachments: Screen Shot 2019-09-07 at 7.27.24 AM.png, Screen Shot 2019-09-07 at 7.29.02 AM.png
>
>
> I noticed there seems to be a memory leak when calling the /metrics endpoint (with Prometheus). I don't know if this relates to the Keycloak metrics or the general Wildfly base metrics.
> Every request to /metrics seems to consume some more heap memory until a OOME occurs. No other requests are done.
> If the metrics endpoint isn't used at all the server works fine.
> {noformat}
> 15:14:50,951 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([("deployment" => "keycloak-server.war")]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.HashMap.resize(HashMap.java:704)
> at java.util.HashMap.putVal(HashMap.java:629)
> at java.util.HashMap.put(HashMap.java:612)
> at java.util.HashSet.add(HashSet.java:220)
> at org.jboss.as.controller.PathAddress.pathAddress(PathAddress.java:95)
> at org.jboss.as.controller.AbstractOperationContext$OperationId.<init>(AbstractOperationContext.java:1571)
> at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1326)
> at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1321)
> at org.jboss.as.controller.operations.global.ReadAttributeHandler$AuthorizeAttributeReadHandler.doExecuteInternal(ReadAttributeHandler.java:244)
> at org.jboss.as.controller.operations.global.ReadAttributeHandler$AuthorizeAttributeReadHandler.execute(ReadAttributeHandler.java:229)
> at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
> at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
> at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
> at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1412)
> at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:423)
> at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerImpl$$Lambda$553/974545249.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity$$Lambda$554/2003274744.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:255)
> at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient$$Lambda$552/516471669.apply(Unknown Source)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient.executeInModelControllerCl(ModelControllerClientFactoryImpl.java:259)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient.executeOperation(ModelControllerClientFactoryImpl.java:157)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$1$$Lambda$546/98454422.apply(Unknown Source)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.lambda$executeInVm$0(ModelControllerClientFactoryImpl.java:351)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$$Lambda$550/226565561.run(Unknown Source)
> at org.jboss.as.controller.access.InVmAccess.runInVm(InVmAccess.java:85)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.executeInVm(ModelControllerClientFactoryImpl.java:351)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.access$000(ModelControllerClientFactoryImpl.java:59)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$1.executeOperation(ModelControllerClientFactoryImpl.java:82)
> at org.jboss.as.controller.LocalModelControllerClient.execute(LocalModelControllerClient.java:54)
> 15:14:50,955 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) WFLYDS0012: Scan of /opt/jboss/keycloak/standalone/deployments threw Exception: java.lang.RuntimeException: WFLYDS0036: Deployment model operation failed. undefined
> at org.jboss.as.server.deployment.scanner.DefaultDeploymentOperations.getUnrelatedDeployments(DefaultDeploymentOperations.java:109)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1692)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1640)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:589)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:493)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$DeploymentScanRunnable.run(FileSystemDeploymentService.java:255)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at org.jboss.threads.JBossThread.run(JBossThread.java:485)
> 15:14:52,319 ERROR [org.jboss.as.controller.management-operation] (management I/O-1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "pbcast.NAKACK2")
> ]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> at sun.reflect.generics.parser.SignatureParser.parsePackageNameAndSimpleClassTypeSignature(SignatureParser.java:331)
> at sun.reflect.generics.parser.SignatureParser.parseClassTypeSignature(SignatureParser.java:310)
> at sun.reflect.generics.parser.SignatureParser.parseFieldTypeSignature(SignatureParser.java:289)
> at sun.reflect.generics.parser.SignatureParser.parseFieldTypeSignature(SignatureParser.java:283)
> at sun.reflect.generics.parser.SignatureParser.parseTypeSignature(SignatureParser.java:485)
> at sun.reflect.generics.parser.SignatureParser.parseTypeSig(SignatureParser.java:188)
> at sun.reflect.annotation.AnnotationParser.parseSig(AnnotationParser.java:436)
> at sun.reflect.annotation.AnnotationParser.parseAnnotation2(AnnotationParser.java:241)
> at sun.reflect.annotation.AnnotationParser.parseAnnotations2(AnnotationParser.java:120)
> at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:72)
> at java.lang.reflect.Field.declaredAnnotations(Field.java:1150)
> at java.lang.reflect.Field.declaredAnnotations(Field.java:1148)
> at java.lang.reflect.Field.getAnnotation(Field.java:1120)
> at org.jgroups.stack.Configurator.resolveAndAssignField(Configurator.java:926)
> at org.jgroups.stack.Configurator.resolveAndAssignFields(Configurator.java:921)
> at org.jgroups.stack.Protocol.setProperties(Protocol.java:133)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator.lambda$createProtocol$0(AbstractProtocolConfigurationServiceConfigurator.java:119)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator$$Lambda$590/1559509553.run(Unknown Source)
> at org.wildfly.security.manager.WildFlySecurityManager.doUnchecked(WildFlySecurityManager.java:849)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator.createProtocol(AbstractProtocolConfigurationServiceConfigurator.java:125)
> at org.jboss.as.clustering.jgroups.subsystem.ChannelRuntimeResourceRegistration.findProtocol(ChannelRuntimeResourceRegistration.java:87)
> at org.jboss.as.clustering.jgroups.subsystem.ProtocolMetricsHandler.executeRuntimeStep(ProtocolMetricsHandler.java:244)
> at org.jboss.as.controller.AbstractRuntimeOnlyHandler$1.execute(AbstractRuntimeOnlyHandler.java:59)
> at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
> at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
> at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
> at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1412)
> at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:423)
> at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerImpl$$Lambda$553/974545249.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity$$Lambda$554/2003274744.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)
> 15:14:52,322 ERROR [io.undertow.request] (management I/O-1) UT005071: Undertow request failed HttpServerExchange{ GET /metrics request {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8], Connection=[keep-alive], Accept-Language=[de,en-US;q=0.7,en;q=0.3], Cache-Control=[max-age=0], Accept-Encoding=[gzip, deflate], User-Agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0], Upgrade-Insecure-Requests=[1], Host=[192.168.99.100:9990]} response {Access-Control-Allow-Origin=[*], Access-Control-Allow-Headers=[origin, content-type, accept, authorization], Access-Control-Allow-Credentials=[true], Content-Type=[text/plain], Access-Control-Allow-Methods=[GET, POST, PUT, DELETE, OPTIONS, HEAD], Access-Control-Max-Age=[1209600]}}: java.lang.IllegalStateException: WFLYMETRICS0003: Unable to read attribute xmit_table_num_purges on [
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "pbcast.NAKACK2")
> ]: "WFLYCTL0158: Operation handler failed: java.lang.OutOfMemoryError: GC overhead limit exceeded".
> at org.wildfly.extension.microprofile.metrics.MetricCollector.lambda$collectResourceMetrics0$0(MetricCollector.java:143)
> at org.wildfly.extension.microprofile.metrics.PrometheusCollector.collect(PrometheusCollector.java:51)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:147)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:168)
> at io.prometheus.client.CollectorRegistry.metricFamilySamples(CollectorRegistry.java:130)
> at org.wildfly.extension.microprofile.metrics.MetricsContextService$1.handleRequest(MetricsContextService.java:100)
> at org.jboss.as.domain.http.server.security.RealmReadinessHandler.handleRequest(RealmReadinessHandler.java:51)
> at org.jboss.as.domain.http.server.security.ServerErrorReadinessHandler.handleRequest(ServerErrorReadinessHandler.java:35)
> at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:91)
> at io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:211)
> at io.undertow.server.handlers.cache.CacheHandler.handleRequest(CacheHandler.java:92)
> at io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:78)
> at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
> at org.jboss.as.domain.http.server.ManagementHttpRequestHandler.handleRequest(ManagementHttpRequestHandler.java:57)
> at org.jboss.as.domain.http.server.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:75)
> at org.jboss.as.domain.http.server.ManagementHttpServer$UpgradeFixHandler.handleRequest(ManagementHttpServer.java:662)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:364)
> at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
> at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
> 15:14:58,673 ERROR [org.jboss.as.controller.management-operation] (management I/O-2) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "UDP")
> ]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> 15:14:58,675 ERROR [io.undertow.request] (management I/O-2) UT005071: Undertow request failed HttpServerExchange{ GET /metrics request {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8], Connection=[keep-alive], Accept-Language=[de,en-US;q=0.7,en;q=0.3], Cache-Control=[max-age=0], Accept-Encoding=[gzip, deflate], User-Agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0], Upgrade-Insecure-Requests=[1], Host=[192.168.99.100:9990]} response {Access-Control-Allow-Origin=[*], Access-Control-Allow-Headers=[origin, content-type, accept, authorization], Access-Control-Allow-Credentials=[true], Content-Type=[text/plain], Access-Control-Allow-Methods=[GET, POST, PUT, DELETE, OPTIONS, HEAD], Access-Control-Max-Age=[1209600]}}: java.lang.IllegalStateException: WFLYMETRICS0003: Unable to read attribute max_bundle_size on [
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "UDP")
> ]: "WFLYCTL0158: Operation handler failed: java.lang.OutOfMemoryError: GC overhead limit exceeded".
> at org.wildfly.extension.microprofile.metrics.MetricCollector.lambda$collectResourceMetrics0$0(MetricCollector.java:143)
> at org.wildfly.extension.microprofile.metrics.PrometheusCollector.collect(PrometheusCollector.java:51)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:147)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:168)
> at io.prometheus.client.CollectorRegistry.metricFamilySamples(CollectorRegistry.java:130)
> at org.wildfly.extension.microprofile.metrics.MetricsContextService$1.handleRequest(MetricsContextService.java:100)
> at org.jboss.as.domain.http.server.security.RealmReadinessHandler.handleRequest(RealmReadinessHandler.java:51)
> at org.jboss.as.domain.http.server.security.ServerErrorReadinessHandler.handleRequest(ServerErrorReadinessHandler.java:35)
> at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:91)
> at io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:211)
> at io.undertow.server.handlers.cache.CacheHandler.handleRequest(CacheHandler.java:92)
> at io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:78)
> at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
> at org.jboss.as.domain.http.server.ManagementHttpRequestHandler.handleRequest(ManagementHttpRequestHandler.java:57)
> at org.jboss.as.domain.http.server.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:75)
> at org.jboss.as.domain.http.server.ManagementHttpServer$UpgradeFixHandler.handleRequest(ManagementHttpServer.java:662)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:364)
> at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:162)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:100)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:57)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:291)
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.nio.QueuedNioTcpServer$1.run(QueuedNioTcpServer.java:131)
> at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612)
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:479)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 5 months
- 1
- 0
[JBoss JIRA] (WFLY-13070) WARN about using NoopTracer by default
by Brian Stansberry (Jira)
[ https://issues.redhat.com/browse/WFLY-13070?page=com.atlassian.jira.plugi... ]
Brian Stansberry updated WFLY-13070:
------------------------------------
Summary: WARN about using NoopTracer by default (was: Use NoopTracer by default)
> WARN about using NoopTracer by default
> --------------------------------------
>
> Key: WFLY-13070
> URL: https://issues.redhat.com/browse/WFLY-13070
> Project: WildFly
> Issue Type: Bug
> Components: Application Client, MP OpenTracing
> Affects Versions: 19.0.0.Beta2
> Reporter: Simon Priadka
> Assignee: Emmanuel Hugonnet
> Priority: Major
>
> When deploying an [app-client quickstart|https://github.com/wildfly/quickstart/tree/master/app-client], following lines are present in the server log:
> {code}
> 16:24:38,057 WARN [org.wildfly.microprofile.opentracing.smallrye] (MSC service thread 1-8) WFLYTRAC0003: Could not determine the service name and can't therefore use Jaeger Tracer. Using NoopTracer.
> 16:24:38,058 WARN [org.wildfly.microprofile.opentracing.smallrye] (MSC service thread 1-2) WFLYTRAC0003: Could not determine the service name and can't therefore use Jaeger Tracer. Using NoopTracer.
> {code}
> As I haven't done anything to the MP configuration, I don't think seeing a WARN message about failing to create a resource I have very little or no interest in, should be present in the logs.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 5 months
- 1
- 0
[JBoss JIRA] (WFLY-13070) Use NoopTracer by default
by Brian Stansberry (Jira)
[ https://issues.redhat.com/browse/WFLY-13070?page=com.atlassian.jira.plugi... ]
Brian Stansberry reassigned WFLY-13070:
---------------------------------------
Component/s: Application Client
(was: MP Config)
Assignee: Emmanuel Hugonnet (was: Jeff Mesnil)
> Use NoopTracer by default
> -------------------------
>
> Key: WFLY-13070
> URL: https://issues.redhat.com/browse/WFLY-13070
> Project: WildFly
> Issue Type: Bug
> Components: Application Client, MP OpenTracing
> Affects Versions: 19.0.0.Beta2
> Reporter: Simon Priadka
> Assignee: Emmanuel Hugonnet
> Priority: Major
>
> When deploying an [app-client quickstart|https://github.com/wildfly/quickstart/tree/master/app-client], following lines are present in the server log:
> {code}
> 16:24:38,057 WARN [org.wildfly.microprofile.opentracing.smallrye] (MSC service thread 1-8) WFLYTRAC0003: Could not determine the service name and can't therefore use Jaeger Tracer. Using NoopTracer.
> 16:24:38,058 WARN [org.wildfly.microprofile.opentracing.smallrye] (MSC service thread 1-2) WFLYTRAC0003: Could not determine the service name and can't therefore use Jaeger Tracer. Using NoopTracer.
> {code}
> As I haven't done anything to the MP configuration, I don't think seeing a WARN message about failing to create a resource I have very little or no interest in, should be present in the logs.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 5 months
- 1
- 0
[JBoss JIRA] (WFLY-13047) Flooding "not found in retransmission table" in server log too much in cluster and making application slow.
by Brian Stansberry (Jira)
[ https://issues.redhat.com/browse/WFLY-13047?page=com.atlassian.jira.plugi... ]
Brian Stansberry reassigned WFLY-13047:
---------------------------------------
Component/s: Clustering
(was: EE)
Priority: Major (was: Blocker)
Assignee: Paul Ferraro (was: Brian Stansberry)
> Flooding "not found in retransmission table" in server log too much in cluster and making application slow.
> -----------------------------------------------------------------------------------------------------------
>
> Key: WFLY-13047
> URL: https://issues.redhat.com/browse/WFLY-13047
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 11.0.0.Final
> Reporter: Dheeraj Mishra
> Assignee: Paul Ferraro
> Priority: Major
> Attachments: log3.zip, standalone.xml
>
>
> We have 24 JVM cluster using multicast and observing below error too much in server.log and it is making application slower. attaching logs for your reference.
> 2020-02-02 02:22:07,839 WARN [org.jboss.as.clustering.jgroups.protocol.NAKACK2] (thread-1) JGRP000041: sol120_7433: message sol120_7403::1425 not found in retransmission table
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 5 months
- 1
- 0