[jboss-jira] [JBoss JIRA] (WFLY-1966) Hangs in DomainControllerMigrationTestCase
Brian Stansberry (JIRA)
jira-events at lists.jboss.org
Thu Aug 29 22:39:03 EDT 2013
[ https://issues.jboss.org/browse/WFLY-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800428#comment-12800428 ]
Brian Stansberry commented on WFLY-1966:
----------------------------------------
Problem is due to an issue unmarshalling the Subject on the slave HC. See
http://brontes.lab.eng.brq.redhat.com/repository/download/WF_DomainOnlyLinux/1823:id/testsuite/domain/target/domains/DomainControllerMigrationTestCase/failover3/log/host-controller.log
04:20:47,552 INFO [org.jboss.as.protocol] (Remoting "failover-h3:MANAGEMENT" task-16) handling org.jboss.as.protocol.mgmt.ManagementRequestHeader at f9ed49 using org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler at 498870
04:20:47,557 INFO [org.jboss.as.controller.management-operation] (Remoting "failover-h3:MANAGEMENT" task-16) Requested subject for 1166537054
04:20:47,564 INFO [org.jboss.as.protocol.connection] (Remoting "failover-h3:MANAGEMENT" task-1) received response for 4
04:20:47,565 INFO [org.jboss.as.protocol] (Remoting "failover-h3:MANAGEMENT" task-1) handling org.jboss.as.protocol.mgmt.ManagementResponseHeader at 736a7e using org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$GetSubjectResponseHandler at 154abc4
04:20:47,654 ERROR [org.jboss.as.protocol] (Remoting "failover-h3:MANAGEMENT" task-1) failed handling org.jboss.as.protocol.mgmt.ManagementResponseHeader at 736a7e using org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$GetSubjectResponseHandler at 154abc4: java.io.IOException: JBAS013469: Unable to unmarshall Subject received for request.
at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$GetSubjectResponseHandler.handleRequest(TransactionalProtocolOperationHandler.java:184) [wildfly-controller-8.0.0.Beta1-SNAPSHOT.jar:8.0.0.Beta1-SNAPSHOT]
at org.jboss.as.protocol.mgmt.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:258) [wildfly-protocol-8.0.0.Beta1-SNAPSHOT.jar:8.0.0.Beta1-SNAPSHOT]
at org.jboss.as.protocol.mgmt.AbstractMessageHandler.handleRequest(AbstractMessageHandler.java:222) [wildfly-protocol-8.0.0.Beta1-SNAPSHOT.jar:8.0.0.Beta1-SNAPSHOT]
at org.jboss.as.protocol.mgmt.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:116) [wildfly-protocol-8.0.0.Beta1-SNAPSHOT.jar:8.0.0.Beta1-SNAPSHOT]
at org.jboss.as.protocol.mgmt.ManagementChannelReceiver$1.handleMessage(ManagementChannelReceiver.java:56) [wildfly-protocol-8.0.0.Beta1-SNAPSHOT.jar:8.0.0.Beta1-SNAPSHOT]
at org.jboss.as.protocol.mgmt.ManagementChannelReceiver.handleMessage(ManagementChannelReceiver.java:84) [wildfly-protocol-8.0.0.Beta1-SNAPSHOT.jar:8.0.0.Beta1-SNAPSHOT]
at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:451) [jboss-remoting-4.0.0.Beta1.jar:4.0.0.Beta1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
Caused by: java.lang.ClassNotFoundException: org.jboss.as.core.security.RealmUser from [Module "org.jboss.as.host-controller:main" from local module loader @1160a60 (finder: local module finder @3514a (roots: /opt/buildAgent/work/e3fca795beba699e/testsuite/domain/../../build/target/wildfly-8.0.0.Beta1-SNAPSHOT/modules,/opt/buildAgent/work/e3fca795beba699e/testsuite/domain/../../build/target/wildfly-8.0.0.Beta1-SNAPSHOT/modules/system/layers/base))]
at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:197) [jboss-modules.jar:1.3.0.Beta3]
at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:443) [jboss-modules.jar:1.3.0.Beta3]
at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:431) [jboss-modules.jar:1.3.0.Beta3]
at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:373) [jboss-modules.jar:1.3.0.Beta3]
at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:118) [jboss-modules.jar:1.3.0.Beta3]
at java.lang.Class.forName0(Native Method) [rt.jar:1.7.0_25]
at java.lang.Class.forName(Class.java:270) [rt.jar:1.7.0_25]
at org.jboss.marshalling.AbstractClassResolver.loadClass(AbstractClassResolver.java:135) [jboss-marshalling-1.3.18.GA.jar:1.3.18.GA]
at org.jboss.marshalling.AbstractClassResolver.resolveClass(AbstractClassResolver.java:116) [jboss-marshalling-1.3.18.GA.jar:1.3.18.GA]
at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:947)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1259)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadCollectionObject(RiverUnmarshaller.java:184)
at org.jboss.marshalling.river.RiverUnmarshaller.readCollectionData(RiverUnmarshaller.java:777)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:662)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [jboss-marshalling-1.3.18.GA.jar:1.3.18.GA]
at org.jboss.marshalling.river.RiverObjectInputStream.readFields(RiverObjectInputStream.java:129)
at javax.security.auth.Subject$SecureSet.readObject(Subject.java:1297) [rt.jar:1.7.0_25]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]
at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]
at org.jboss.marshalling.reflect.SerializableClass.callReadObject(SerializableClass.java:218) [jboss-marshalling-1.3.18.GA.jar:1.3.18.GA]
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1629)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1290)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1732)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1648)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1612)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1290)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1732)
at org.jboss.marshalling.river.RiverObjectInputStream.defaultReadObject(RiverObjectInputStream.java:73)
at javax.security.auth.Subject.readObject(Subject.java:947) [rt.jar:1.7.0_25]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]
at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]
at org.jboss.marshalling.reflect.SerializableClass.callReadObject(SerializableClass.java:218) [jboss-marshalling-1.3.18.GA.jar:1.3.18.GA]
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1629)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1290)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:72) [jboss-marshalling-1.3.18.GA.jar:1.3.18.GA]
at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$GetSubjectResponseHandler.handleRequest(TransactionalProtocolOperationHandler.java:182) [wildfly-controller-8.0.0.Beta1-SNAPSHOT.jar:8.0.0.Beta1-SNAPSHOT]
... 9 more
I have no idea why that is intermittent.
Another issue is why the catch of that exception didn't result in proper cleanup.
> Hangs in DomainControllerMigrationTestCase
> ------------------------------------------
>
> Key: WFLY-1966
> URL: https://issues.jboss.org/browse/WFLY-1966
> Project: WildFly
> Issue Type: Bug
> Components: Domain Management
> Reporter: Brian Stansberry
> Fix For: 8.0.0.Beta1
>
> Attachments: thread-dump.txt
>
>
> DomainControllerMigrationTestCase is intermittently hanging recently.
> I will attach thread dumps from a recent hang.
> The test driver is hanging here:
> "main" prio=10 tid=0xb7605c00 nid=0x7656 in Object.wait() [0xb7786000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xa99833f8> (a org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl)
> at java.lang.Object.wait(Object.java:503)
> at org.jboss.threads.AsyncFutureTask.await(AsyncFutureTask.java:192)
> - eliminated <0xa99833f8> (a org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl)
> at org.jboss.threads.AsyncFutureTask.get(AsyncFutureTask.java:266)
> - locked <0xa99833f8> (a org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl)
> at org.jboss.as.controller.client.impl.AbstractDelegatingAsyncFuture.get(AbstractDelegatingAsyncFuture.java:100)
> at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:127)
> at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:76)
> at org.jboss.as.controller.client.helpers.domain.impl.DomainClientImpl.execute(DomainClientImpl.java:86)
> at org.jboss.as.test.integration.domain.management.util.DomainLifecycleUtil.executeForResult(DomainLifecycleUtil.java:580)
> at org.jboss.as.test.integration.domain.DomainControllerMigrationTestCase.testDCFailover(DomainControllerMigrationTestCase.java:206)
> The master HC process is stuck here:
> "management-handler-thread - 2" prio=10 tid=0x8be50400 nid=0x1a92 waiting on condition [0x87604000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xaa032fd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.jboss.as.controller.remote.BlockingQueueOperationListener.retrievePreparedOperation(BlockingQueueOperationListener.java:84)
> at org.jboss.as.domain.controller.operations.coordination.DomainSlaveHandler.execute(DomainSlaveHandler.java:117)
> at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:608)
> at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:486)
> at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:275)
> at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:270)
> at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:257)
> at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:142)
> at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:205)
> at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.access$300(ModelControllerClientOperationHandler.java:110)
> at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$2.run(ModelControllerClientOperationHandler.java:157)
> at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$2.run(ModelControllerClientOperationHandler.java:153)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:153)
> at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:296)
> at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:518)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> at org.jboss.threads.JBossThread.run(JBossThread.java:122)
> I see nothing relevant in the other thread dumps.
> The master HC is blocking waiting for a response from the slave HC.
> Just before the client makes the request that triggers this, it has reloaded the slave HC. The test is not robust in terms of how it checks that the slave is ready before continuing on with the test; it merely connects directly to the slave checks that the slave's root resource can be read. That can succeed relatively early in boot even before the slave has registered with the master.
> Fixing that may prevent the hangs, but it would likely paper over whatever problem is causing the hangs. A client invoking multi-host operations while a slave is booting and registering is a use case that can't hang.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the jboss-jira
mailing list