[
https://issues.jboss.org/browse/WFCORE-1270?page=com.atlassian.jira.plugi...
]
Kabir Khan commented on WFCORE-1270:
------------------------------------
An update (although no solution yet).
The OperationTransformerRegistry is created for each server/host when it is registered,
and that all happens in one thread. This happens via
{code}
[Host Controller] 11:47:47,090 ERROR [stderr] (Controller Boot Thread) at
java.lang.Thread.dumpStack(Thread.java:1329)
[Host Controller] 11:47:47,090 ERROR [stderr] (Controller Boot Thread) at
org.jboss.as.controller.registry.GlobalTransformerRegistry.create(GlobalTransformerRegistry.java:167)
[Host Controller] 11:47:47,090 ERROR [stderr] (Controller Boot Thread) at
org.jboss.as.controller.transform.TransformerRegistry.resolveServer(TransformerRegistry.java:183)
[Host Controller] 11:47:47,090 ERROR [stderr] (Controller Boot Thread) at
org.jboss.as.controller.transform.TransformationTargetImpl.create(TransformationTargetImpl.java:94)
[Host Controller] 11:47:47,090 ERROR [stderr] (Controller Boot Thread) at
org.jboss.as.host.controller.ServerInventoryImpl.createManagedServer(ServerInventoryImpl.java:621)
[Host Controller] 11:47:47,090 ERROR [stderr] (Controller Boot Thread) at
org.jboss.as.host.controller.ServerInventoryImpl.startServer(ServerInventoryImpl.java:193)
[Host Controller] 11:47:47,090 ERROR [stderr] (Controller Boot Thread) at
org.jboss.as.host.controller.DomainModelControllerService$DelegatingServerInventory.startServer(DomainModelControllerService.java:1072)
[Host Controller] 11:47:47,090 ERROR [stderr] (Controller Boot Thread) at
org.jboss.as.host.controller.operations.StartServersHandler.cleanStartServers(StartServersHandler.java
{code}
and
{code}
[Host Controller] 11:50:01,928 ERROR [stderr] (Host Controller Service Threads - 30)
java.lang.Exception: Stack trace
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at
java.lang.Thread.dumpStack(Thread.java:1329)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at
org.jboss.as.controller.registry.GlobalTransformerRegistry.create(GlobalTransformerRegistry.java:167)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at
org.jboss.as.controller.transform.TransformerRegistry.resolveHost(TransformerRegistry.java:157)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at
org.jboss.as.controller.transform.TransformationTargetImpl.create(TransformationTargetImpl.java:97)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at
org.jboss.as.host.controller.mgmt.HostControllerRegistrationHandler$HostRegistrationStepHandler.execute(HostControllerRegistrationHandler.java:306)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at
org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:890)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at
org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:659)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at
org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at
org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1344)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at
org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:392)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at
org.jboss.as.controller.AbstractControllerService.internalExecute(AbstractControllerService.java:408)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at
org.jboss.as.host.controller.DomainModelControllerService.access$900(DomainModelControllerService.java:176)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at
org.jboss.as.host.controller.DomainModelControllerService$InternalExecutor.execute(DomainModelControllerService.java:1230)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at
org.jboss.as.host.controller.mgmt.HostControllerRegistrationHandler$RegistrationContext.processRegistration(HostControllerRegistrationHandler.java:424)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at
org.jboss.as.host.controller.mgmt.HostControllerRegistrationHandler$RegistrationContext.access$400(HostControllerRegistrationHandler.java:334)
{code}
The common point is that both servers and hosts hit GlobalTransformerRegistry.java:167 to
get the OperationTransformerRegistry created from the GlobalTransformerRegistry which
contains all versions. The GlobalTransformerRegistry is populated first, on boot of the
DC, and the population of that takes place in _several_ threads, as part of the parallel
boot of extensions. So that is probably where a race is. So I think your PR will merely
mask the problem rather than fix it.
Intermittent NPE registering a slave HC in mixed-domain test suite
------------------------------------------------------------------
Key: WFCORE-1270
URL:
https://issues.jboss.org/browse/WFCORE-1270
Project: WildFly Core
Issue Type: Bug
Components: Domain Management, Test Suite
Affects Versions: 2.0.5.Final
Reporter: Brian Stansberry
Assignee: Kabir Khan
I've occasionally seen test failures in the mixed domain tests when a testsuite
cannot start because the slave HC can't register due to an NPE on the master:
For example:
http://brontes.lab.eng.brq.redhat.com/viewLog.html?buildId=82653&tab=...
Critical log details:
{code}[0m16:16:15,252 INFO [org.jboss.modules] (main) JBoss Modules version
1.3.3.Final-redhat-1
[0m[0m16:16:15,363 INFO [org.jboss.as.process.Host
Controller.status] (main) JBAS012017: Starting process 'Host Controller'
[0m[Host Controller] [0m16:16:15,871 INFO [org.jboss.modules]
(main) JBoss Modules version 1.3.3.Final-redhat-1[0m
[Host Controller] [0m[0m16:16:16,015 INFO [org.jboss.msc] (main)
JBoss MSC version 1.1.5.Final-redhat-1[0m
[Host Controller] [0m[0m16:16:16,068 INFO [org.jboss.as] (MSC
service thread 1-4) JBAS015899: JBoss EAP 6.3.0.GA (AS 7.4.0.Final-redhat-19)
starting[0m
[Host Controller] [0m[0m16:16:16,625 INFO [org.xnio] (MSC
service thread 1-2) XNIO Version 3.0.10.GA-redhat-1[0m
[Host Controller] [0m[0m16:16:16,629 INFO [org.xnio.nio] (MSC
service thread 1-2) XNIO NIO Implementation Version 3.0.10.GA-redhat-1[0m
[Host Controller] [0m[0m16:16:16,657 INFO [org.jboss.remoting]
(MSC service thread 1-2) JBoss Remoting version (unknown)[0m
[Host Controller] [0m[0m16:16:16,705 INFO
[org.jboss.as.remoting] (MSC service thread 1-2) JBAS017100: Listening on
[::1]:19999[0m
INFO [org.jboss.as.test.integration.domain.management.util.DomainLifecycleUtil]
ServerIdentity{name=server-one, host=slave, server-group=other-server-group} status is
starting
[Host Controller] [0m[31m16:16:17,913 ERROR
[org.jboss.as.controller.management-operation] (Host Controller Service Threads - 50)
WFLYCTL0013: Operation ("read-master-domain-model") failed - address: ([]):
java.lang.NullPointerException[0m
[Host Controller] [31m at
org.jboss.as.controller.registry.OperationTransformerRegistry$1.getResourceTransformer(OperationTransformerRegistry.java:83)[0m
[Host Controller] [31m at
org.jboss.as.controller.transform.ResourceTransformationContextImpl.resolveTransformer(ResourceTransformationContextImpl.java:235)[0m
[Host Controller] [31m at
org.jboss.as.controller.transform.ResourceTransformationContextImpl.processChild(ResourceTransformationContextImpl.java:285)[0m
[Host Controller] [31m at
org.jboss.as.controller.transform.ResourceTransformationContextImpl.processChildren(ResourceTransformationContextImpl.java:254)[0m
[Host Controller] [31m at
org.jboss.as.controller.transform.ResourceTransformer$1.transformResource(ResourceTransformer.java:53)[0m
[Host Controller] [31m at
org.jboss.as.controller.transform.description.TransformingDescription$3.invokeNext(TransformingDescription.java:161)[0m
[Host Controller] [31m at
org.jboss.as.controller.transform.description.AttributeTransformationRule.transformResource(AttributeTransformationRule.java:103)[0m
[Host Controller] [31m at
org.jboss.as.controller.transform.description.TransformingDescription.transformResource(TransformingDescription.java:167)[0m
[Host Controller] [31m at
org.jboss.as.controller.transform.ResourceTransformationContextImpl.processChild(ResourceTransformationContextImpl.java:289)[0m
[Host Controller] [31m at
org.jboss.as.controller.transform.ResourceTransformationContextImpl.processChildren(ResourceTransformationContextImpl.java:254)[0m
[Host Controller] [31m at
org.jboss.as.controller.transform.ResourceTransformer$1.transformResource(ResourceTransformer.java:53)[0m
[Host Controller] [31m at
org.jboss.as.controller.transform.TransformersImpl.transformRootResource(TransformersImpl.java:115)[0m
[Host Controller] [31m at
org.jboss.as.domain.controller.operations.ReadMasterDomainModelUtil.readMasterDomainResourcesForInitialConnect(ReadMasterDomainModelUtil.java:88)[0m
[Host Controller] [31m at
org.jboss.as.domain.controller.operations.ReadDomainModelHandler.execute(ReadDomainModelHandler.java:51)[0m
[Host Controller] [31m at
org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:890)[0m
[Host Controller] [31m at
org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:659)[0m
[Host Controller] [31m at
org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)[0m
[Host Controller] [31m at
org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1343)[0m
[Host Controller] [31m at
org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:395)[0m
[Host Controller] [31m at
org.jboss.as.controller.AbstractControllerService.internalExecute(AbstractControllerService.java:408)[0m
[Host Controller] [31m at
org.jboss.as.host.controller.DomainModelControllerService.access$900(DomainModelControllerService.java:175)[0m
[Host Controller] [31m at
org.jboss.as.host.controller.DomainModelControllerService$InternalExecutor.execute(DomainModelControllerService.java:1229)[0m
[Host Controller] [31m at
org.jboss.as.host.controller.mgmt.HostControllerRegistrationHandler$RegistrationContext.processRegistration(HostControllerRegistrationHandler.java:424)[0m
[Host Controller] [31m at
org.jboss.as.host.controller.mgmt.HostControllerRegistrationHandler$RegistrationContext.access$400(HostControllerRegistrationHandler.java:334)[0m
[Host Controller] [31m at
org.jboss.as.host.controller.mgmt.HostControllerRegistrationHandler$InitiateRegistrationHandler$1.execute(HostControllerRegistrationHandler.java:230)[0m
[Host Controller] [31m at
org.jboss.as.protocol.mgmt.AbstractMessageHandler$ManagementRequestContextImpl$1.doExecute(AbstractMessageHandler.java:363)[0m
[Host Controller] [31m at
org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:465)[0m
[Host Controller] [31m at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[0m
[Host Controller] [31m at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[0m
[Host Controller] [31m at
java.lang.Thread.run(Thread.java:745)[0m
[Host Controller] [31m at
org.jboss.threads.JBossThread.run(JBossThread.java:320)[0m
[Host Controller] [31m[0m
[Host Controller] [0m[33m16:16:17,935 WARN
[org.jboss.as.host.controller] (Controller Boot Thread) JBAS010900: Could not connect to
remote domain controller at remote://[::1]:9999 -- 1-$-WFLYCTL0158: Operation handler
failed: java.lang.NullPointerException[0m
[Host Controller] [33m[0m[33m16:16:17,936 WARN
[org.jboss.as.host.controller] (Controller Boot Thread) JBAS016581: No domain controller
discovery options remain.[0m
[Host Controller] [33m[0m[31m16:16:17,939 ERROR
[org.jboss.as.host.controller] (Controller Boot Thread) JBAS010901: Could not connect to
master. Aborting. Error was: java.lang.IllegalStateException: JBAS016519: Tried all domain
controller discovery option(s) but unable to connect[0m
{code}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)