[JBoss JIRA] (WFLY-12513) Invalid logger exclude in org.wildfly.clustering.web.hotrod module.xml
by Brian Stansberry (Jira)
Brian Stansberry created WFLY-12513:
---------------------------------------
Summary: Invalid logger exclude in org.wildfly.clustering.web.hotrod module.xml
Key: WFLY-12513
URL: https://issues.jboss.org/browse/WFLY-12513
Project: WildFly
Issue Type: Bug
Components: Clustering
Reporter: Brian Stansberry
Assignee: Paul Ferraro
This looks like a copy-and-paste error from a different file. The Logger in this module is not in that package.
{code}
<exports>
<exclude path="org/wildfly/clustering/web/infinispan/logging"/>
</exports>
{code}
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 8 months
[JBoss JIRA] (WFLY-12503) Turn off the default surefire execution when the 'elytron' and 'security.manager' profiles are used
by Brian Stansberry (Jira)
[ https://issues.jboss.org/browse/WFLY-12503?page=com.atlassian.jira.plugin... ]
Brian Stansberry edited comment on WFLY-12503 at 9/7/19 12:43 PM:
------------------------------------------------------------------
This is problematic for -Delytron because for some reason adding the profile to the root pom adjusting the surefire plugin execution causes the execution of various 'process-test-resource' phase maven goals to happen in a different order. The testsuite/integration/elytron module has a number of such goals and reordering them breaks the ultimately configuration against which the tests run, leading to failures.
Breaking out the -Dsecurity.manager commit into a separate JIRA/PR would get the benefit of that bit, but I have other priorities so I'm putting this on the shelf for now.
was (Author: brian.stansberry):
This is problematic for -Delytron because for some reason adding the profile to the root pom adjusting the surefire plugin execution causes the execution of various 'process-test-resource' phase maven goals to happen in a different order. The testsuite/integration/elytron module has a number of such goals and reordering them breaks the ultimately configuration against which the tests run, leading to failures.
> Turn off the default surefire execution when the 'elytron' and 'security.manager' profiles are used
> ---------------------------------------------------------------------------------------------------
>
> Key: WFLY-12503
> URL: https://issues.jboss.org/browse/WFLY-12503
> Project: WildFly
> Issue Type: Task
> Components: Test Suite
> Reporter: Brian Stansberry
> Assignee: Brian Stansberry
> Priority: Major
>
> We have CI runs for PRs that differ from the basic ones by using -Delytron instead of -DallTests to turn on profiles that select a subset of the testsuite and to specially configure things for that subset. Same for -Dsecurity.manager.
> Expand on this by having -Delytron and -Dsecurity.manager also turn on profiles that turn off the default surefire execution, so surefire only runs for those maven modules where a profile turns it back on. There's no point, for example, running the 'undertow' maven module tests in these CI runs.
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 8 months
[JBoss JIRA] (WFLY-12504) Memoryleak in Metrics-Endpoint
by Brian Stansberry (Jira)
[ https://issues.jboss.org/browse/WFLY-12504?page=com.atlassian.jira.plugin... ]
Brian Stansberry commented on WFLY-12504:
-----------------------------------------
[~pferraro] [~jmesnil] FYI.
> Memoryleak in Metrics-Endpoint
> ------------------------------
>
> Key: WFLY-12504
> URL: https://issues.jboss.org/browse/WFLY-12504
> Project: WildFly
> Issue Type: Bug
> Components: MP Metrics
> Affects Versions: 17.0.1.Final
> Environment: Docker Image jboss/keycloak:6.0.1
> Reporter: Mathias Werlitz
> Assignee: Richard Opalka
> Priority: Blocker
> Labels: blocker-WF18
> Attachments: Screen Shot 2019-09-07 at 7.27.24 AM.png, Screen Shot 2019-09-07 at 7.29.02 AM.png
>
>
> I noticed there seems to be a memory leak when calling the /metrics endpoint (with Prometheus). I don't know if this relates to the Keycloak metrics or the general Wildfly base metrics.
> Every request to /metrics seems to consume some more heap memory until a OOME occurs. No other requests are done.
> If the metrics endpoint isn't used at all the server works fine.
> {noformat}
> 15:14:50,951 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([("deployment" => "keycloak-server.war")]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.HashMap.resize(HashMap.java:704)
> at java.util.HashMap.putVal(HashMap.java:629)
> at java.util.HashMap.put(HashMap.java:612)
> at java.util.HashSet.add(HashSet.java:220)
> at org.jboss.as.controller.PathAddress.pathAddress(PathAddress.java:95)
> at org.jboss.as.controller.AbstractOperationContext$OperationId.<init>(AbstractOperationContext.java:1571)
> at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1326)
> at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1321)
> at org.jboss.as.controller.operations.global.ReadAttributeHandler$AuthorizeAttributeReadHandler.doExecuteInternal(ReadAttributeHandler.java:244)
> at org.jboss.as.controller.operations.global.ReadAttributeHandler$AuthorizeAttributeReadHandler.execute(ReadAttributeHandler.java:229)
> at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
> at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
> at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
> at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1412)
> at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:423)
> at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerImpl$$Lambda$553/974545249.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity$$Lambda$554/2003274744.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:255)
> at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient$$Lambda$552/516471669.apply(Unknown Source)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient.executeInModelControllerCl(ModelControllerClientFactoryImpl.java:259)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient.executeOperation(ModelControllerClientFactoryImpl.java:157)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$1$$Lambda$546/98454422.apply(Unknown Source)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.lambda$executeInVm$0(ModelControllerClientFactoryImpl.java:351)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$$Lambda$550/226565561.run(Unknown Source)
> at org.jboss.as.controller.access.InVmAccess.runInVm(InVmAccess.java:85)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.executeInVm(ModelControllerClientFactoryImpl.java:351)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.access$000(ModelControllerClientFactoryImpl.java:59)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$1.executeOperation(ModelControllerClientFactoryImpl.java:82)
> at org.jboss.as.controller.LocalModelControllerClient.execute(LocalModelControllerClient.java:54)
> 15:14:50,955 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) WFLYDS0012: Scan of /opt/jboss/keycloak/standalone/deployments threw Exception: java.lang.RuntimeException: WFLYDS0036: Deployment model operation failed. undefined
> at org.jboss.as.server.deployment.scanner.DefaultDeploymentOperations.getUnrelatedDeployments(DefaultDeploymentOperations.java:109)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1692)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1640)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:589)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:493)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$DeploymentScanRunnable.run(FileSystemDeploymentService.java:255)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at org.jboss.threads.JBossThread.run(JBossThread.java:485)
> 15:14:52,319 ERROR [org.jboss.as.controller.management-operation] (management I/O-1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "pbcast.NAKACK2")
> ]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> at sun.reflect.generics.parser.SignatureParser.parsePackageNameAndSimpleClassTypeSignature(SignatureParser.java:331)
> at sun.reflect.generics.parser.SignatureParser.parseClassTypeSignature(SignatureParser.java:310)
> at sun.reflect.generics.parser.SignatureParser.parseFieldTypeSignature(SignatureParser.java:289)
> at sun.reflect.generics.parser.SignatureParser.parseFieldTypeSignature(SignatureParser.java:283)
> at sun.reflect.generics.parser.SignatureParser.parseTypeSignature(SignatureParser.java:485)
> at sun.reflect.generics.parser.SignatureParser.parseTypeSig(SignatureParser.java:188)
> at sun.reflect.annotation.AnnotationParser.parseSig(AnnotationParser.java:436)
> at sun.reflect.annotation.AnnotationParser.parseAnnotation2(AnnotationParser.java:241)
> at sun.reflect.annotation.AnnotationParser.parseAnnotations2(AnnotationParser.java:120)
> at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:72)
> at java.lang.reflect.Field.declaredAnnotations(Field.java:1150)
> at java.lang.reflect.Field.declaredAnnotations(Field.java:1148)
> at java.lang.reflect.Field.getAnnotation(Field.java:1120)
> at org.jgroups.stack.Configurator.resolveAndAssignField(Configurator.java:926)
> at org.jgroups.stack.Configurator.resolveAndAssignFields(Configurator.java:921)
> at org.jgroups.stack.Protocol.setProperties(Protocol.java:133)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator.lambda$createProtocol$0(AbstractProtocolConfigurationServiceConfigurator.java:119)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator$$Lambda$590/1559509553.run(Unknown Source)
> at org.wildfly.security.manager.WildFlySecurityManager.doUnchecked(WildFlySecurityManager.java:849)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator.createProtocol(AbstractProtocolConfigurationServiceConfigurator.java:125)
> at org.jboss.as.clustering.jgroups.subsystem.ChannelRuntimeResourceRegistration.findProtocol(ChannelRuntimeResourceRegistration.java:87)
> at org.jboss.as.clustering.jgroups.subsystem.ProtocolMetricsHandler.executeRuntimeStep(ProtocolMetricsHandler.java:244)
> at org.jboss.as.controller.AbstractRuntimeOnlyHandler$1.execute(AbstractRuntimeOnlyHandler.java:59)
> at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
> at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
> at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
> at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1412)
> at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:423)
> at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerImpl$$Lambda$553/974545249.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity$$Lambda$554/2003274744.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)
> 15:14:52,322 ERROR [io.undertow.request] (management I/O-1) UT005071: Undertow request failed HttpServerExchange{ GET /metrics request {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8], Connection=[keep-alive], Accept-Language=[de,en-US;q=0.7,en;q=0.3], Cache-Control=[max-age=0], Accept-Encoding=[gzip, deflate], User-Agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0], Upgrade-Insecure-Requests=[1], Host=[192.168.99.100:9990]} response {Access-Control-Allow-Origin=[*], Access-Control-Allow-Headers=[origin, content-type, accept, authorization], Access-Control-Allow-Credentials=[true], Content-Type=[text/plain], Access-Control-Allow-Methods=[GET, POST, PUT, DELETE, OPTIONS, HEAD], Access-Control-Max-Age=[1209600]}}: java.lang.IllegalStateException: WFLYMETRICS0003: Unable to read attribute xmit_table_num_purges on [
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "pbcast.NAKACK2")
> ]: "WFLYCTL0158: Operation handler failed: java.lang.OutOfMemoryError: GC overhead limit exceeded".
> at org.wildfly.extension.microprofile.metrics.MetricCollector.lambda$collectResourceMetrics0$0(MetricCollector.java:143)
> at org.wildfly.extension.microprofile.metrics.PrometheusCollector.collect(PrometheusCollector.java:51)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:147)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:168)
> at io.prometheus.client.CollectorRegistry.metricFamilySamples(CollectorRegistry.java:130)
> at org.wildfly.extension.microprofile.metrics.MetricsContextService$1.handleRequest(MetricsContextService.java:100)
> at org.jboss.as.domain.http.server.security.RealmReadinessHandler.handleRequest(RealmReadinessHandler.java:51)
> at org.jboss.as.domain.http.server.security.ServerErrorReadinessHandler.handleRequest(ServerErrorReadinessHandler.java:35)
> at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:91)
> at io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:211)
> at io.undertow.server.handlers.cache.CacheHandler.handleRequest(CacheHandler.java:92)
> at io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:78)
> at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
> at org.jboss.as.domain.http.server.ManagementHttpRequestHandler.handleRequest(ManagementHttpRequestHandler.java:57)
> at org.jboss.as.domain.http.server.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:75)
> at org.jboss.as.domain.http.server.ManagementHttpServer$UpgradeFixHandler.handleRequest(ManagementHttpServer.java:662)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:364)
> at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
> at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
> 15:14:58,673 ERROR [org.jboss.as.controller.management-operation] (management I/O-2) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "UDP")
> ]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> 15:14:58,675 ERROR [io.undertow.request] (management I/O-2) UT005071: Undertow request failed HttpServerExchange{ GET /metrics request {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8], Connection=[keep-alive], Accept-Language=[de,en-US;q=0.7,en;q=0.3], Cache-Control=[max-age=0], Accept-Encoding=[gzip, deflate], User-Agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0], Upgrade-Insecure-Requests=[1], Host=[192.168.99.100:9990]} response {Access-Control-Allow-Origin=[*], Access-Control-Allow-Headers=[origin, content-type, accept, authorization], Access-Control-Allow-Credentials=[true], Content-Type=[text/plain], Access-Control-Allow-Methods=[GET, POST, PUT, DELETE, OPTIONS, HEAD], Access-Control-Max-Age=[1209600]}}: java.lang.IllegalStateException: WFLYMETRICS0003: Unable to read attribute max_bundle_size on [
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "UDP")
> ]: "WFLYCTL0158: Operation handler failed: java.lang.OutOfMemoryError: GC overhead limit exceeded".
> at org.wildfly.extension.microprofile.metrics.MetricCollector.lambda$collectResourceMetrics0$0(MetricCollector.java:143)
> at org.wildfly.extension.microprofile.metrics.PrometheusCollector.collect(PrometheusCollector.java:51)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:147)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:168)
> at io.prometheus.client.CollectorRegistry.metricFamilySamples(CollectorRegistry.java:130)
> at org.wildfly.extension.microprofile.metrics.MetricsContextService$1.handleRequest(MetricsContextService.java:100)
> at org.jboss.as.domain.http.server.security.RealmReadinessHandler.handleRequest(RealmReadinessHandler.java:51)
> at org.jboss.as.domain.http.server.security.ServerErrorReadinessHandler.handleRequest(ServerErrorReadinessHandler.java:35)
> at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:91)
> at io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:211)
> at io.undertow.server.handlers.cache.CacheHandler.handleRequest(CacheHandler.java:92)
> at io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:78)
> at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
> at org.jboss.as.domain.http.server.ManagementHttpRequestHandler.handleRequest(ManagementHttpRequestHandler.java:57)
> at org.jboss.as.domain.http.server.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:75)
> at org.jboss.as.domain.http.server.ManagementHttpServer$UpgradeFixHandler.handleRequest(ManagementHttpServer.java:662)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:364)
> at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:162)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:100)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:57)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:291)
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.nio.QueuedNioTcpServer$1.run(QueuedNioTcpServer.java:131)
> at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612)
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:479)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 8 months
[JBoss JIRA] (WFLY-12504) Memoryleak in Metrics-Endpoint
by Brian Stansberry (Jira)
[ https://issues.jboss.org/browse/WFLY-12504?page=com.atlassian.jira.plugin... ]
Brian Stansberry reassigned WFLY-12504:
---------------------------------------
Assignee: Richard Opalka (was: Jeff Mesnil)
[~ropalka] I can reproduce this with current WildFly master so it's not a case of a re-report of something already fixed. And it looks like WFLY-12567 is not completely fixed.
This is easy to reproduce; just start WF using the standalone-full-ha.xml config, take a heap dump, point a browser at http://localhost:9990/metrics, keep refreshing for a while, take another dump, and compare.
I attached a jvisualvm comparision like that and it's showing thousands of additional ServiceRegistrationImpl instances.
I randomly picked one and I believe it's a good example and attached a screenshot showing its state. Its name (not shown in the png) is a UUID so highly likely it's one installed by PassiveServiceSupplier. I see that it's state has 'removed=false' and 'pendingDependents=1'. I see that 'removed' will only be set to 'true' if SRI.clear() is called when pendingDependents == 0, and if it doesn't get set to true the SRI will not be removed from the ServiceContainerImpl.registry map.
I *think* the problem here is that ServiceContainerImpl.getOrCreateRegistration calls SRI.addPendingDependent both for cases where it is handling the installation of the service itself, and for cases where it is recording a new dependent for a registration. But the pendingDependents counter only gets decremented when a addDependent gets called; i.e. when dealing with the new dependent. The effect is the initial increment of pendingDependents, when service is being created, never gets an offsetting decrement and AFAICT no SRI will ever be removed.
> Memoryleak in Metrics-Endpoint
> ------------------------------
>
> Key: WFLY-12504
> URL: https://issues.jboss.org/browse/WFLY-12504
> Project: WildFly
> Issue Type: Bug
> Components: MP Metrics
> Affects Versions: 17.0.1.Final
> Environment: Docker Image jboss/keycloak:6.0.1
> Reporter: Mathias Werlitz
> Assignee: Richard Opalka
> Priority: Blocker
> Labels: blocker-WF18
> Attachments: Screen Shot 2019-09-07 at 7.27.24 AM.png, Screen Shot 2019-09-07 at 7.29.02 AM.png
>
>
> I noticed there seems to be a memory leak when calling the /metrics endpoint (with Prometheus). I don't know if this relates to the Keycloak metrics or the general Wildfly base metrics.
> Every request to /metrics seems to consume some more heap memory until a OOME occurs. No other requests are done.
> If the metrics endpoint isn't used at all the server works fine.
> {noformat}
> 15:14:50,951 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([("deployment" => "keycloak-server.war")]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.HashMap.resize(HashMap.java:704)
> at java.util.HashMap.putVal(HashMap.java:629)
> at java.util.HashMap.put(HashMap.java:612)
> at java.util.HashSet.add(HashSet.java:220)
> at org.jboss.as.controller.PathAddress.pathAddress(PathAddress.java:95)
> at org.jboss.as.controller.AbstractOperationContext$OperationId.<init>(AbstractOperationContext.java:1571)
> at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1326)
> at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1321)
> at org.jboss.as.controller.operations.global.ReadAttributeHandler$AuthorizeAttributeReadHandler.doExecuteInternal(ReadAttributeHandler.java:244)
> at org.jboss.as.controller.operations.global.ReadAttributeHandler$AuthorizeAttributeReadHandler.execute(ReadAttributeHandler.java:229)
> at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
> at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
> at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
> at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1412)
> at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:423)
> at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerImpl$$Lambda$553/974545249.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity$$Lambda$554/2003274744.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:255)
> at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient$$Lambda$552/516471669.apply(Unknown Source)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient.executeInModelControllerCl(ModelControllerClientFactoryImpl.java:259)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient.executeOperation(ModelControllerClientFactoryImpl.java:157)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$1$$Lambda$546/98454422.apply(Unknown Source)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.lambda$executeInVm$0(ModelControllerClientFactoryImpl.java:351)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$$Lambda$550/226565561.run(Unknown Source)
> at org.jboss.as.controller.access.InVmAccess.runInVm(InVmAccess.java:85)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.executeInVm(ModelControllerClientFactoryImpl.java:351)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.access$000(ModelControllerClientFactoryImpl.java:59)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$1.executeOperation(ModelControllerClientFactoryImpl.java:82)
> at org.jboss.as.controller.LocalModelControllerClient.execute(LocalModelControllerClient.java:54)
> 15:14:50,955 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) WFLYDS0012: Scan of /opt/jboss/keycloak/standalone/deployments threw Exception: java.lang.RuntimeException: WFLYDS0036: Deployment model operation failed. undefined
> at org.jboss.as.server.deployment.scanner.DefaultDeploymentOperations.getUnrelatedDeployments(DefaultDeploymentOperations.java:109)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1692)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1640)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:589)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:493)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$DeploymentScanRunnable.run(FileSystemDeploymentService.java:255)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at org.jboss.threads.JBossThread.run(JBossThread.java:485)
> 15:14:52,319 ERROR [org.jboss.as.controller.management-operation] (management I/O-1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "pbcast.NAKACK2")
> ]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> at sun.reflect.generics.parser.SignatureParser.parsePackageNameAndSimpleClassTypeSignature(SignatureParser.java:331)
> at sun.reflect.generics.parser.SignatureParser.parseClassTypeSignature(SignatureParser.java:310)
> at sun.reflect.generics.parser.SignatureParser.parseFieldTypeSignature(SignatureParser.java:289)
> at sun.reflect.generics.parser.SignatureParser.parseFieldTypeSignature(SignatureParser.java:283)
> at sun.reflect.generics.parser.SignatureParser.parseTypeSignature(SignatureParser.java:485)
> at sun.reflect.generics.parser.SignatureParser.parseTypeSig(SignatureParser.java:188)
> at sun.reflect.annotation.AnnotationParser.parseSig(AnnotationParser.java:436)
> at sun.reflect.annotation.AnnotationParser.parseAnnotation2(AnnotationParser.java:241)
> at sun.reflect.annotation.AnnotationParser.parseAnnotations2(AnnotationParser.java:120)
> at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:72)
> at java.lang.reflect.Field.declaredAnnotations(Field.java:1150)
> at java.lang.reflect.Field.declaredAnnotations(Field.java:1148)
> at java.lang.reflect.Field.getAnnotation(Field.java:1120)
> at org.jgroups.stack.Configurator.resolveAndAssignField(Configurator.java:926)
> at org.jgroups.stack.Configurator.resolveAndAssignFields(Configurator.java:921)
> at org.jgroups.stack.Protocol.setProperties(Protocol.java:133)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator.lambda$createProtocol$0(AbstractProtocolConfigurationServiceConfigurator.java:119)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator$$Lambda$590/1559509553.run(Unknown Source)
> at org.wildfly.security.manager.WildFlySecurityManager.doUnchecked(WildFlySecurityManager.java:849)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator.createProtocol(AbstractProtocolConfigurationServiceConfigurator.java:125)
> at org.jboss.as.clustering.jgroups.subsystem.ChannelRuntimeResourceRegistration.findProtocol(ChannelRuntimeResourceRegistration.java:87)
> at org.jboss.as.clustering.jgroups.subsystem.ProtocolMetricsHandler.executeRuntimeStep(ProtocolMetricsHandler.java:244)
> at org.jboss.as.controller.AbstractRuntimeOnlyHandler$1.execute(AbstractRuntimeOnlyHandler.java:59)
> at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
> at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
> at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
> at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1412)
> at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:423)
> at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerImpl$$Lambda$553/974545249.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity$$Lambda$554/2003274744.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)
> 15:14:52,322 ERROR [io.undertow.request] (management I/O-1) UT005071: Undertow request failed HttpServerExchange{ GET /metrics request {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8], Connection=[keep-alive], Accept-Language=[de,en-US;q=0.7,en;q=0.3], Cache-Control=[max-age=0], Accept-Encoding=[gzip, deflate], User-Agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0], Upgrade-Insecure-Requests=[1], Host=[192.168.99.100:9990]} response {Access-Control-Allow-Origin=[*], Access-Control-Allow-Headers=[origin, content-type, accept, authorization], Access-Control-Allow-Credentials=[true], Content-Type=[text/plain], Access-Control-Allow-Methods=[GET, POST, PUT, DELETE, OPTIONS, HEAD], Access-Control-Max-Age=[1209600]}}: java.lang.IllegalStateException: WFLYMETRICS0003: Unable to read attribute xmit_table_num_purges on [
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "pbcast.NAKACK2")
> ]: "WFLYCTL0158: Operation handler failed: java.lang.OutOfMemoryError: GC overhead limit exceeded".
> at org.wildfly.extension.microprofile.metrics.MetricCollector.lambda$collectResourceMetrics0$0(MetricCollector.java:143)
> at org.wildfly.extension.microprofile.metrics.PrometheusCollector.collect(PrometheusCollector.java:51)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:147)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:168)
> at io.prometheus.client.CollectorRegistry.metricFamilySamples(CollectorRegistry.java:130)
> at org.wildfly.extension.microprofile.metrics.MetricsContextService$1.handleRequest(MetricsContextService.java:100)
> at org.jboss.as.domain.http.server.security.RealmReadinessHandler.handleRequest(RealmReadinessHandler.java:51)
> at org.jboss.as.domain.http.server.security.ServerErrorReadinessHandler.handleRequest(ServerErrorReadinessHandler.java:35)
> at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:91)
> at io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:211)
> at io.undertow.server.handlers.cache.CacheHandler.handleRequest(CacheHandler.java:92)
> at io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:78)
> at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
> at org.jboss.as.domain.http.server.ManagementHttpRequestHandler.handleRequest(ManagementHttpRequestHandler.java:57)
> at org.jboss.as.domain.http.server.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:75)
> at org.jboss.as.domain.http.server.ManagementHttpServer$UpgradeFixHandler.handleRequest(ManagementHttpServer.java:662)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:364)
> at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
> at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
> 15:14:58,673 ERROR [org.jboss.as.controller.management-operation] (management I/O-2) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "UDP")
> ]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> 15:14:58,675 ERROR [io.undertow.request] (management I/O-2) UT005071: Undertow request failed HttpServerExchange{ GET /metrics request {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8], Connection=[keep-alive], Accept-Language=[de,en-US;q=0.7,en;q=0.3], Cache-Control=[max-age=0], Accept-Encoding=[gzip, deflate], User-Agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0], Upgrade-Insecure-Requests=[1], Host=[192.168.99.100:9990]} response {Access-Control-Allow-Origin=[*], Access-Control-Allow-Headers=[origin, content-type, accept, authorization], Access-Control-Allow-Credentials=[true], Content-Type=[text/plain], Access-Control-Allow-Methods=[GET, POST, PUT, DELETE, OPTIONS, HEAD], Access-Control-Max-Age=[1209600]}}: java.lang.IllegalStateException: WFLYMETRICS0003: Unable to read attribute max_bundle_size on [
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "UDP")
> ]: "WFLYCTL0158: Operation handler failed: java.lang.OutOfMemoryError: GC overhead limit exceeded".
> at org.wildfly.extension.microprofile.metrics.MetricCollector.lambda$collectResourceMetrics0$0(MetricCollector.java:143)
> at org.wildfly.extension.microprofile.metrics.PrometheusCollector.collect(PrometheusCollector.java:51)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:147)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:168)
> at io.prometheus.client.CollectorRegistry.metricFamilySamples(CollectorRegistry.java:130)
> at org.wildfly.extension.microprofile.metrics.MetricsContextService$1.handleRequest(MetricsContextService.java:100)
> at org.jboss.as.domain.http.server.security.RealmReadinessHandler.handleRequest(RealmReadinessHandler.java:51)
> at org.jboss.as.domain.http.server.security.ServerErrorReadinessHandler.handleRequest(ServerErrorReadinessHandler.java:35)
> at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:91)
> at io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:211)
> at io.undertow.server.handlers.cache.CacheHandler.handleRequest(CacheHandler.java:92)
> at io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:78)
> at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
> at org.jboss.as.domain.http.server.ManagementHttpRequestHandler.handleRequest(ManagementHttpRequestHandler.java:57)
> at org.jboss.as.domain.http.server.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:75)
> at org.jboss.as.domain.http.server.ManagementHttpServer$UpgradeFixHandler.handleRequest(ManagementHttpServer.java:662)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:364)
> at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:162)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:100)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:57)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:291)
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.nio.QueuedNioTcpServer$1.run(QueuedNioTcpServer.java:131)
> at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612)
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:479)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 8 months
[JBoss JIRA] (WFLY-12504) Memoryleak in Metrics-Endpoint
by Brian Stansberry (Jira)
[ https://issues.jboss.org/browse/WFLY-12504?page=com.atlassian.jira.plugin... ]
Brian Stansberry updated WFLY-12504:
------------------------------------
Attachment: Screen Shot 2019-09-07 at 7.27.24 AM.png
Screen Shot 2019-09-07 at 7.29.02 AM.png
> Memoryleak in Metrics-Endpoint
> ------------------------------
>
> Key: WFLY-12504
> URL: https://issues.jboss.org/browse/WFLY-12504
> Project: WildFly
> Issue Type: Bug
> Components: MP Metrics
> Affects Versions: 17.0.1.Final
> Environment: Docker Image jboss/keycloak:6.0.1
> Reporter: Mathias Werlitz
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: blocker-WF18
> Attachments: Screen Shot 2019-09-07 at 7.27.24 AM.png, Screen Shot 2019-09-07 at 7.29.02 AM.png
>
>
> I noticed there seems to be a memory leak when calling the /metrics endpoint (with Prometheus). I don't know if this relates to the Keycloak metrics or the general Wildfly base metrics.
> Every request to /metrics seems to consume some more heap memory until a OOME occurs. No other requests are done.
> If the metrics endpoint isn't used at all the server works fine.
> {noformat}
> 15:14:50,951 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([("deployment" => "keycloak-server.war")]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.HashMap.resize(HashMap.java:704)
> at java.util.HashMap.putVal(HashMap.java:629)
> at java.util.HashMap.put(HashMap.java:612)
> at java.util.HashSet.add(HashSet.java:220)
> at org.jboss.as.controller.PathAddress.pathAddress(PathAddress.java:95)
> at org.jboss.as.controller.AbstractOperationContext$OperationId.<init>(AbstractOperationContext.java:1571)
> at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1326)
> at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1321)
> at org.jboss.as.controller.operations.global.ReadAttributeHandler$AuthorizeAttributeReadHandler.doExecuteInternal(ReadAttributeHandler.java:244)
> at org.jboss.as.controller.operations.global.ReadAttributeHandler$AuthorizeAttributeReadHandler.execute(ReadAttributeHandler.java:229)
> at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
> at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
> at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
> at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1412)
> at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:423)
> at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerImpl$$Lambda$553/974545249.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity$$Lambda$554/2003274744.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:255)
> at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient$$Lambda$552/516471669.apply(Unknown Source)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient.executeInModelControllerCl(ModelControllerClientFactoryImpl.java:259)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient.executeOperation(ModelControllerClientFactoryImpl.java:157)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$1$$Lambda$546/98454422.apply(Unknown Source)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.lambda$executeInVm$0(ModelControllerClientFactoryImpl.java:351)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$$Lambda$550/226565561.run(Unknown Source)
> at org.jboss.as.controller.access.InVmAccess.runInVm(InVmAccess.java:85)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.executeInVm(ModelControllerClientFactoryImpl.java:351)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl.access$000(ModelControllerClientFactoryImpl.java:59)
> at org.jboss.as.controller.ModelControllerClientFactoryImpl$1.executeOperation(ModelControllerClientFactoryImpl.java:82)
> at org.jboss.as.controller.LocalModelControllerClient.execute(LocalModelControllerClient.java:54)
> 15:14:50,955 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) WFLYDS0012: Scan of /opt/jboss/keycloak/standalone/deployments threw Exception: java.lang.RuntimeException: WFLYDS0036: Deployment model operation failed. undefined
> at org.jboss.as.server.deployment.scanner.DefaultDeploymentOperations.getUnrelatedDeployments(DefaultDeploymentOperations.java:109)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1692)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1640)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:589)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:493)
> at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$DeploymentScanRunnable.run(FileSystemDeploymentService.java:255)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at org.jboss.threads.JBossThread.run(JBossThread.java:485)
> 15:14:52,319 ERROR [org.jboss.as.controller.management-operation] (management I/O-1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "pbcast.NAKACK2")
> ]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> at sun.reflect.generics.parser.SignatureParser.parsePackageNameAndSimpleClassTypeSignature(SignatureParser.java:331)
> at sun.reflect.generics.parser.SignatureParser.parseClassTypeSignature(SignatureParser.java:310)
> at sun.reflect.generics.parser.SignatureParser.parseFieldTypeSignature(SignatureParser.java:289)
> at sun.reflect.generics.parser.SignatureParser.parseFieldTypeSignature(SignatureParser.java:283)
> at sun.reflect.generics.parser.SignatureParser.parseTypeSignature(SignatureParser.java:485)
> at sun.reflect.generics.parser.SignatureParser.parseTypeSig(SignatureParser.java:188)
> at sun.reflect.annotation.AnnotationParser.parseSig(AnnotationParser.java:436)
> at sun.reflect.annotation.AnnotationParser.parseAnnotation2(AnnotationParser.java:241)
> at sun.reflect.annotation.AnnotationParser.parseAnnotations2(AnnotationParser.java:120)
> at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:72)
> at java.lang.reflect.Field.declaredAnnotations(Field.java:1150)
> at java.lang.reflect.Field.declaredAnnotations(Field.java:1148)
> at java.lang.reflect.Field.getAnnotation(Field.java:1120)
> at org.jgroups.stack.Configurator.resolveAndAssignField(Configurator.java:926)
> at org.jgroups.stack.Configurator.resolveAndAssignFields(Configurator.java:921)
> at org.jgroups.stack.Protocol.setProperties(Protocol.java:133)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator.lambda$createProtocol$0(AbstractProtocolConfigurationServiceConfigurator.java:119)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator$$Lambda$590/1559509553.run(Unknown Source)
> at org.wildfly.security.manager.WildFlySecurityManager.doUnchecked(WildFlySecurityManager.java:849)
> at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator.createProtocol(AbstractProtocolConfigurationServiceConfigurator.java:125)
> at org.jboss.as.clustering.jgroups.subsystem.ChannelRuntimeResourceRegistration.findProtocol(ChannelRuntimeResourceRegistration.java:87)
> at org.jboss.as.clustering.jgroups.subsystem.ProtocolMetricsHandler.executeRuntimeStep(ProtocolMetricsHandler.java:244)
> at org.jboss.as.controller.AbstractRuntimeOnlyHandler$1.execute(AbstractRuntimeOnlyHandler.java:59)
> at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
> at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
> at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
> at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1412)
> at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:423)
> at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
> at org.jboss.as.controller.ModelControllerImpl$$Lambda$553/974545249.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity$$Lambda$554/2003274744.run(Unknown Source)
> at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)
> 15:14:52,322 ERROR [io.undertow.request] (management I/O-1) UT005071: Undertow request failed HttpServerExchange{ GET /metrics request {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8], Connection=[keep-alive], Accept-Language=[de,en-US;q=0.7,en;q=0.3], Cache-Control=[max-age=0], Accept-Encoding=[gzip, deflate], User-Agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0], Upgrade-Insecure-Requests=[1], Host=[192.168.99.100:9990]} response {Access-Control-Allow-Origin=[*], Access-Control-Allow-Headers=[origin, content-type, accept, authorization], Access-Control-Allow-Credentials=[true], Content-Type=[text/plain], Access-Control-Allow-Methods=[GET, POST, PUT, DELETE, OPTIONS, HEAD], Access-Control-Max-Age=[1209600]}}: java.lang.IllegalStateException: WFLYMETRICS0003: Unable to read attribute xmit_table_num_purges on [
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "pbcast.NAKACK2")
> ]: "WFLYCTL0158: Operation handler failed: java.lang.OutOfMemoryError: GC overhead limit exceeded".
> at org.wildfly.extension.microprofile.metrics.MetricCollector.lambda$collectResourceMetrics0$0(MetricCollector.java:143)
> at org.wildfly.extension.microprofile.metrics.PrometheusCollector.collect(PrometheusCollector.java:51)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:147)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:168)
> at io.prometheus.client.CollectorRegistry.metricFamilySamples(CollectorRegistry.java:130)
> at org.wildfly.extension.microprofile.metrics.MetricsContextService$1.handleRequest(MetricsContextService.java:100)
> at org.jboss.as.domain.http.server.security.RealmReadinessHandler.handleRequest(RealmReadinessHandler.java:51)
> at org.jboss.as.domain.http.server.security.ServerErrorReadinessHandler.handleRequest(ServerErrorReadinessHandler.java:35)
> at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:91)
> at io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:211)
> at io.undertow.server.handlers.cache.CacheHandler.handleRequest(CacheHandler.java:92)
> at io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:78)
> at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
> at org.jboss.as.domain.http.server.ManagementHttpRequestHandler.handleRequest(ManagementHttpRequestHandler.java:57)
> at org.jboss.as.domain.http.server.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:75)
> at org.jboss.as.domain.http.server.ManagementHttpServer$UpgradeFixHandler.handleRequest(ManagementHttpServer.java:662)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:364)
> at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
> at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
> 15:14:58,673 ERROR [org.jboss.as.controller.management-operation] (management I/O-2) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "UDP")
> ]): java.lang.OutOfMemoryError: GC overhead limit exceeded
> 15:14:58,675 ERROR [io.undertow.request] (management I/O-2) UT005071: Undertow request failed HttpServerExchange{ GET /metrics request {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8], Connection=[keep-alive], Accept-Language=[de,en-US;q=0.7,en;q=0.3], Cache-Control=[max-age=0], Accept-Encoding=[gzip, deflate], User-Agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0], Upgrade-Insecure-Requests=[1], Host=[192.168.99.100:9990]} response {Access-Control-Allow-Origin=[*], Access-Control-Allow-Headers=[origin, content-type, accept, authorization], Access-Control-Allow-Credentials=[true], Content-Type=[text/plain], Access-Control-Allow-Methods=[GET, POST, PUT, DELETE, OPTIONS, HEAD], Access-Control-Max-Age=[1209600]}}: java.lang.IllegalStateException: WFLYMETRICS0003: Unable to read attribute max_bundle_size on [
> ("subsystem" => "jgroups"),
> ("channel" => "ee"),
> ("protocol" => "UDP")
> ]: "WFLYCTL0158: Operation handler failed: java.lang.OutOfMemoryError: GC overhead limit exceeded".
> at org.wildfly.extension.microprofile.metrics.MetricCollector.lambda$collectResourceMetrics0$0(MetricCollector.java:143)
> at org.wildfly.extension.microprofile.metrics.PrometheusCollector.collect(PrometheusCollector.java:51)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:147)
> at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:168)
> at io.prometheus.client.CollectorRegistry.metricFamilySamples(CollectorRegistry.java:130)
> at org.wildfly.extension.microprofile.metrics.MetricsContextService$1.handleRequest(MetricsContextService.java:100)
> at org.jboss.as.domain.http.server.security.RealmReadinessHandler.handleRequest(RealmReadinessHandler.java:51)
> at org.jboss.as.domain.http.server.security.ServerErrorReadinessHandler.handleRequest(ServerErrorReadinessHandler.java:35)
> at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:91)
> at io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:211)
> at io.undertow.server.handlers.cache.CacheHandler.handleRequest(CacheHandler.java:92)
> at io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:78)
> at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
> at org.jboss.as.domain.http.server.ManagementHttpRequestHandler.handleRequest(ManagementHttpRequestHandler.java:57)
> at org.jboss.as.domain.http.server.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:75)
> at org.jboss.as.domain.http.server.ManagementHttpServer$UpgradeFixHandler.handleRequest(ManagementHttpServer.java:662)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:364)
> at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255)
> at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:162)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:100)
> at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:57)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:291)
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.nio.QueuedNioTcpServer$1.run(QueuedNioTcpServer.java:131)
> at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612)
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:479)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 8 months