[jboss-jira] [JBoss JIRA] (WFCORE-1270) Intermittent NPE registering a slave HC in mixed-domain test suite
Kabir Khan (JIRA)
issues at jboss.org
Fri Jan 8 07:00:01 EST 2016
[ https://issues.jboss.org/browse/WFCORE-1270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146237#comment-13146237 ]
Kabir Khan commented on WFCORE-1270:
------------------------------------
An update (although no solution yet).
The OperationTransformerRegistry is created for each server/host when it is registered, and that all happens in one thread. This happens via
{code}
[Host Controller] 11:47:47,090 ERROR [stderr] (Controller Boot Thread) at java.lang.Thread.dumpStack(Thread.java:1329)
[Host Controller] 11:47:47,090 ERROR [stderr] (Controller Boot Thread) at org.jboss.as.controller.registry.GlobalTransformerRegistry.create(GlobalTransformerRegistry.java:167)
[Host Controller] 11:47:47,090 ERROR [stderr] (Controller Boot Thread) at org.jboss.as.controller.transform.TransformerRegistry.resolveServer(TransformerRegistry.java:183)
[Host Controller] 11:47:47,090 ERROR [stderr] (Controller Boot Thread) at org.jboss.as.controller.transform.TransformationTargetImpl.create(TransformationTargetImpl.java:94)
[Host Controller] 11:47:47,090 ERROR [stderr] (Controller Boot Thread) at org.jboss.as.host.controller.ServerInventoryImpl.createManagedServer(ServerInventoryImpl.java:621)
[Host Controller] 11:47:47,090 ERROR [stderr] (Controller Boot Thread) at org.jboss.as.host.controller.ServerInventoryImpl.startServer(ServerInventoryImpl.java:193)
[Host Controller] 11:47:47,090 ERROR [stderr] (Controller Boot Thread) at org.jboss.as.host.controller.DomainModelControllerService$DelegatingServerInventory.startServer(DomainModelControllerService.java:1072)
[Host Controller] 11:47:47,090 ERROR [stderr] (Controller Boot Thread) at org.jboss.as.host.controller.operations.StartServersHandler.cleanStartServers(StartServersHandler.java
{code}
and
{code}
[Host Controller] 11:50:01,928 ERROR [stderr] (Host Controller Service Threads - 30) java.lang.Exception: Stack trace
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at java.lang.Thread.dumpStack(Thread.java:1329)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at org.jboss.as.controller.registry.GlobalTransformerRegistry.create(GlobalTransformerRegistry.java:167)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at org.jboss.as.controller.transform.TransformerRegistry.resolveHost(TransformerRegistry.java:157)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at org.jboss.as.controller.transform.TransformationTargetImpl.create(TransformationTargetImpl.java:97)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at org.jboss.as.host.controller.mgmt.HostControllerRegistrationHandler$HostRegistrationStepHandler.execute(HostControllerRegistrationHandler.java:306)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:890)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:659)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1344)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:392)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at org.jboss.as.controller.AbstractControllerService.internalExecute(AbstractControllerService.java:408)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at org.jboss.as.host.controller.DomainModelControllerService.access$900(DomainModelControllerService.java:176)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at org.jboss.as.host.controller.DomainModelControllerService$InternalExecutor.execute(DomainModelControllerService.java:1230)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at org.jboss.as.host.controller.mgmt.HostControllerRegistrationHandler$RegistrationContext.processRegistration(HostControllerRegistrationHandler.java:424)
[Host Controller] 11:50:01,929 ERROR [stderr] (Host Controller Service Threads - 30) at org.jboss.as.host.controller.mgmt.HostControllerRegistrationHandler$RegistrationContext.access$400(HostControllerRegistrationHandler.java:334)
{code}
The common point is that both servers and hosts hit GlobalTransformerRegistry.java:167 to get the OperationTransformerRegistry created from the GlobalTransformerRegistry which contains all versions. The GlobalTransformerRegistry is populated first, on boot of the DC, and the population of that takes place in _several_ threads, as part of the parallel boot of extensions. So that is probably where a race is. So I think your PR will merely mask the problem rather than fix it.
> Intermittent NPE registering a slave HC in mixed-domain test suite
> ------------------------------------------------------------------
>
> Key: WFCORE-1270
> URL: https://issues.jboss.org/browse/WFCORE-1270
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management, Test Suite
> Affects Versions: 2.0.5.Final
> Reporter: Brian Stansberry
> Assignee: Kabir Khan
>
> I've occasionally seen test failures in the mixed domain tests when a testsuite cannot start because the slave HC can't register due to an NPE on the master:
> For example:
> http://brontes.lab.eng.brq.redhat.com/viewLog.html?buildId=82653&tab=buildResultsDiv&buildTypeId=WildFlyCore_PullRequest_WildFlyCoreFullIntegration
> Critical log details:
> {code}[0m16:16:15,252 INFO [org.jboss.modules] (main) JBoss Modules version 1.3.3.Final-redhat-1
> [0m[0m16:16:15,363 INFO [org.jboss.as.process.Host Controller.status] (main) JBAS012017: Starting process 'Host Controller'
> [0m[Host Controller] [0m16:16:15,871 INFO [org.jboss.modules] (main) JBoss Modules version 1.3.3.Final-redhat-1[0m
> [Host Controller] [0m[0m16:16:16,015 INFO [org.jboss.msc] (main) JBoss MSC version 1.1.5.Final-redhat-1[0m
> [Host Controller] [0m[0m16:16:16,068 INFO [org.jboss.as] (MSC service thread 1-4) JBAS015899: JBoss EAP 6.3.0.GA (AS 7.4.0.Final-redhat-19) starting[0m
> [Host Controller] [0m[0m16:16:16,625 INFO [org.xnio] (MSC service thread 1-2) XNIO Version 3.0.10.GA-redhat-1[0m
> [Host Controller] [0m[0m16:16:16,629 INFO [org.xnio.nio] (MSC service thread 1-2) XNIO NIO Implementation Version 3.0.10.GA-redhat-1[0m
> [Host Controller] [0m[0m16:16:16,657 INFO [org.jboss.remoting] (MSC service thread 1-2) JBoss Remoting version (unknown)[0m
> [Host Controller] [0m[0m16:16:16,705 INFO [org.jboss.as.remoting] (MSC service thread 1-2) JBAS017100: Listening on [::1]:19999[0m
> INFO [org.jboss.as.test.integration.domain.management.util.DomainLifecycleUtil] ServerIdentity{name=server-one, host=slave, server-group=other-server-group} status is starting
> [Host Controller] [0m[31m16:16:17,913 ERROR [org.jboss.as.controller.management-operation] (Host Controller Service Threads - 50) WFLYCTL0013: Operation ("read-master-domain-model") failed - address: ([]): java.lang.NullPointerException[0m
> [Host Controller] [31m at org.jboss.as.controller.registry.OperationTransformerRegistry$1.getResourceTransformer(OperationTransformerRegistry.java:83)[0m
> [Host Controller] [31m at org.jboss.as.controller.transform.ResourceTransformationContextImpl.resolveTransformer(ResourceTransformationContextImpl.java:235)[0m
> [Host Controller] [31m at org.jboss.as.controller.transform.ResourceTransformationContextImpl.processChild(ResourceTransformationContextImpl.java:285)[0m
> [Host Controller] [31m at org.jboss.as.controller.transform.ResourceTransformationContextImpl.processChildren(ResourceTransformationContextImpl.java:254)[0m
> [Host Controller] [31m at org.jboss.as.controller.transform.ResourceTransformer$1.transformResource(ResourceTransformer.java:53)[0m
> [Host Controller] [31m at org.jboss.as.controller.transform.description.TransformingDescription$3.invokeNext(TransformingDescription.java:161)[0m
> [Host Controller] [31m at org.jboss.as.controller.transform.description.AttributeTransformationRule.transformResource(AttributeTransformationRule.java:103)[0m
> [Host Controller] [31m at org.jboss.as.controller.transform.description.TransformingDescription.transformResource(TransformingDescription.java:167)[0m
> [Host Controller] [31m at org.jboss.as.controller.transform.ResourceTransformationContextImpl.processChild(ResourceTransformationContextImpl.java:289)[0m
> [Host Controller] [31m at org.jboss.as.controller.transform.ResourceTransformationContextImpl.processChildren(ResourceTransformationContextImpl.java:254)[0m
> [Host Controller] [31m at org.jboss.as.controller.transform.ResourceTransformer$1.transformResource(ResourceTransformer.java:53)[0m
> [Host Controller] [31m at org.jboss.as.controller.transform.TransformersImpl.transformRootResource(TransformersImpl.java:115)[0m
> [Host Controller] [31m at org.jboss.as.domain.controller.operations.ReadMasterDomainModelUtil.readMasterDomainResourcesForInitialConnect(ReadMasterDomainModelUtil.java:88)[0m
> [Host Controller] [31m at org.jboss.as.domain.controller.operations.ReadDomainModelHandler.execute(ReadDomainModelHandler.java:51)[0m
> [Host Controller] [31m at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:890)[0m
> [Host Controller] [31m at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:659)[0m
> [Host Controller] [31m at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)[0m
> [Host Controller] [31m at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1343)[0m
> [Host Controller] [31m at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:395)[0m
> [Host Controller] [31m at org.jboss.as.controller.AbstractControllerService.internalExecute(AbstractControllerService.java:408)[0m
> [Host Controller] [31m at org.jboss.as.host.controller.DomainModelControllerService.access$900(DomainModelControllerService.java:175)[0m
> [Host Controller] [31m at org.jboss.as.host.controller.DomainModelControllerService$InternalExecutor.execute(DomainModelControllerService.java:1229)[0m
> [Host Controller] [31m at org.jboss.as.host.controller.mgmt.HostControllerRegistrationHandler$RegistrationContext.processRegistration(HostControllerRegistrationHandler.java:424)[0m
> [Host Controller] [31m at org.jboss.as.host.controller.mgmt.HostControllerRegistrationHandler$RegistrationContext.access$400(HostControllerRegistrationHandler.java:334)[0m
> [Host Controller] [31m at org.jboss.as.host.controller.mgmt.HostControllerRegistrationHandler$InitiateRegistrationHandler$1.execute(HostControllerRegistrationHandler.java:230)[0m
> [Host Controller] [31m at org.jboss.as.protocol.mgmt.AbstractMessageHandler$ManagementRequestContextImpl$1.doExecute(AbstractMessageHandler.java:363)[0m
> [Host Controller] [31m at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:465)[0m
> [Host Controller] [31m at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[0m
> [Host Controller] [31m at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[0m
> [Host Controller] [31m at java.lang.Thread.run(Thread.java:745)[0m
> [Host Controller] [31m at org.jboss.threads.JBossThread.run(JBossThread.java:320)[0m
> [Host Controller] [31m[0m
> [Host Controller] [0m[33m16:16:17,935 WARN [org.jboss.as.host.controller] (Controller Boot Thread) JBAS010900: Could not connect to remote domain controller at remote://[::1]:9999 -- 1-$-WFLYCTL0158: Operation handler failed: java.lang.NullPointerException[0m
> [Host Controller] [33m[0m[33m16:16:17,936 WARN [org.jboss.as.host.controller] (Controller Boot Thread) JBAS016581: No domain controller discovery options remain.[0m
> [Host Controller] [33m[0m[31m16:16:17,939 ERROR [org.jboss.as.host.controller] (Controller Boot Thread) JBAS010901: Could not connect to master. Aborting. Error was: java.lang.IllegalStateException: JBAS016519: Tried all domain controller discovery option(s) but unable to connect[0m
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
More information about the jboss-jira
mailing list