[JBoss JIRA] (JGRP-2250) Stuck Threads - BLOCKED infinitely on - JGroups - org.jgroups.protocols.FlowControl$Credit.decrementIfEnoughCredits(FlowControl.java:567)
by Rohit Singh (JIRA)
[ https://issues.jboss.org/browse/JGRP-2250?page=com.atlassian.jira.plugin.... ]
Rohit Singh updated JGRP-2250:
------------------------------
Labels: Infinispan JGroups (was: )
> Stuck Threads - BLOCKED infinitely on - JGroups - org.jgroups.protocols.FlowControl$Credit.decrementIfEnoughCredits(FlowControl.java:567)
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: JGRP-2250
> URL: https://issues.jboss.org/browse/JGRP-2250
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.4.1
> Environment: Oracle Linux - Exalogic/Exadata stack
> AS: Weblogic 12.1.3
> DB: Oracle 12.1.0.2
> Reporter: Rohit Singh
> Assignee: Bela Ban
> Priority: Blocker
> Labels: Infinispan, JGroups
>
> Ours is a J2EE application (clustered across 4 nodes) internally using *{color:red}Infinispan Cache (Ver 6.0.2) with JGroups (Ver 3.4.1){color}* as transport layer of our cache.
> Application had been running perfectly fine, but suddenly on one node, one cache-put got stuck with below stack trace. *(Cause being blocked on org.jgroups.protocols.FlowControl$Credit.decrementIfEnoughCredits(FlowControl.java:567)).*
> Now all further puts are getting stuck - blocked - on the same line on the same node. Other nodes are working fine in this respect.
> Currently, while raising the issue, the threads are still in stuck state. And the first stuck thread is stuck from last 27 hours.
> Stack Trace:
> "[STUCK] ExecuteThread: '37' for queue: 'weblogic.kernel.Default (self-tuning)'" #107862 daemon prio=1 os_prio=0 tid=0x00007f5964373000 nid=0x75a9 in Object.wait() [0x00007f57d596b000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at *{color:red}org.jgroups.protocols.FlowControl$Credit.decrementIfEnoughCredits(FlowControl.java:567){color}*
> - locked <0x00007f5e64893168> (a org.jgroups.protocols.FlowControl$Credit)
> at org.jgroups.protocols.UFC.handleDownMessage(UFC.java:121)
> at org.jgroups.protocols.FlowControl.down(FlowControl.java:330)
> at org.jgroups.protocols.FlowControl.down(FlowControl.java:340)
> at org.jgroups.protocols.FRAG2.down(FRAG2.java:136)
> at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:1024)
> at org.jgroups.JChannel.down(JChannel.java:760)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:683)
> at org.jgroups.blocks.RequestCorrelator.sendUnicastRequest(RequestCorrelator.java:202)
> at org.jgroups.blocks.UnicastRequest.sendRequest(UnicastRequest.java:43)
> at org.jgroups.blocks.Request.execute(Request.java:83)
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:399)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:353)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:167)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:521)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:281)
> at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:233)
> at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:72)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:321)
> at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:402)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:164)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:68)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:219)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:141)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:148)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:134)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1306)
> at org.infinispan.CacheImpl.putInternal(CacheImpl.java:878)
> at org.infinispan.CacheImpl.put(CacheImpl.java:870)
> at org.infinispan.DecoratedCache.put(DecoratedCache.java:401)
> at com.nucleus.finnone.pro.cache.common.NeutrinoCache.put(NeutrinoCache.java:18)
> at com.nucleus.config.persisted.service.ConfigurationServiceImpl.getConfigurationGroupFor(ConfigurationServiceImpl.java:478)
> *Note: Seems related to below-*
> https://issues.jboss.org/browse/JGRP-1675
> https://issues.jboss.org/browse/ISPN-3645
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months
[JBoss JIRA] (JGRP-2250) Stuck Threads - BLOCKED infinitely on - JGroups - org.jgroups.protocols.FlowControl$Credit.decrementIfEnoughCredits(FlowControl.java:567)
by Rohit Singh (JIRA)
Rohit Singh created JGRP-2250:
---------------------------------
Summary: Stuck Threads - BLOCKED infinitely on - JGroups - org.jgroups.protocols.FlowControl$Credit.decrementIfEnoughCredits(FlowControl.java:567)
Key: JGRP-2250
URL: https://issues.jboss.org/browse/JGRP-2250
Project: JGroups
Issue Type: Bug
Affects Versions: 3.4.1
Environment: Oracle Linux - Exalogic/Exadata stack
AS: Weblogic 12.1.3
DB: Oracle 12.1.0.2
Reporter: Rohit Singh
Assignee: Bela Ban
Priority: Blocker
Ours is a J2EE application (clustered across 4 nodes) internally using *{color:red}Infinispan Cache (Ver 6.0.2) with JGroups (Ver 3.4.1){color}* as transport layer of our cache.
Application had been running perfectly fine, but suddenly on one node, one cache-put got stuck with below stack trace. *(Cause being blocked on org.jgroups.protocols.FlowControl$Credit.decrementIfEnoughCredits(FlowControl.java:567)).*
Now all further puts are getting stuck - blocked - on the same line on the same node. Other nodes are working fine in this respect.
Currently, while raising the issue, the threads are still in stuck state. And the first stuck thread is stuck from last 27 hours.
Stack Trace:
"[STUCK] ExecuteThread: '37' for queue: 'weblogic.kernel.Default (self-tuning)'" #107862 daemon prio=1 os_prio=0 tid=0x00007f5964373000 nid=0x75a9 in Object.wait() [0x00007f57d596b000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at *{color:red}org.jgroups.protocols.FlowControl$Credit.decrementIfEnoughCredits(FlowControl.java:567){color}*
- locked <0x00007f5e64893168> (a org.jgroups.protocols.FlowControl$Credit)
at org.jgroups.protocols.UFC.handleDownMessage(UFC.java:121)
at org.jgroups.protocols.FlowControl.down(FlowControl.java:330)
at org.jgroups.protocols.FlowControl.down(FlowControl.java:340)
at org.jgroups.protocols.FRAG2.down(FRAG2.java:136)
at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:1024)
at org.jgroups.JChannel.down(JChannel.java:760)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:683)
at org.jgroups.blocks.RequestCorrelator.sendUnicastRequest(RequestCorrelator.java:202)
at org.jgroups.blocks.UnicastRequest.sendRequest(UnicastRequest.java:43)
at org.jgroups.blocks.Request.execute(Request.java:83)
at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:399)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:353)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:167)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:521)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:281)
at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:233)
at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:72)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:321)
at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:402)
at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:164)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:68)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:219)
at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:141)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:148)
at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:134)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1306)
at org.infinispan.CacheImpl.putInternal(CacheImpl.java:878)
at org.infinispan.CacheImpl.put(CacheImpl.java:870)
at org.infinispan.DecoratedCache.put(DecoratedCache.java:401)
at com.nucleus.finnone.pro.cache.common.NeutrinoCache.put(NeutrinoCache.java:18)
at com.nucleus.config.persisted.service.ConfigurationServiceImpl.getConfigurationGroupFor(ConfigurationServiceImpl.java:478)
*Note: Seems related to below-*
https://issues.jboss.org/browse/JGRP-1675
https://issues.jboss.org/browse/ISPN-3645
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months
[JBoss JIRA] (WFCORE-3590) Hang in ServerStartFailureTestCase
by Richard Opalka (JIRA)
[ https://issues.jboss.org/browse/WFCORE-3590?page=com.atlassian.jira.plugi... ]
Richard Opalka commented on WFCORE-3590:
----------------------------------------
MSC executor shouldn't be rejecting tasks until all services are removed [~brian.stansberry]. I'm assigning this issue to me [~dmlloyd].
> Hang in ServerStartFailureTestCase
> ----------------------------------
>
> Key: WFCORE-3590
> URL: https://issues.jboss.org/browse/WFCORE-3590
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management, Server
> Affects Versions: 4.0.0.Alpha9
> Reporter: Brian Stansberry
> Assignee: David Lloyd
> Priority: Critical
> Attachments: WFCORE-3590-threads.txt
>
>
> Hang observed in https://ci.wildfly.org/viewLog.html?buildId=88611&buildTypeId=WildFlyCore...
> I'll attach the thread dump.
> [~dmlloyd] I assigned this to you mostly as a form of ping, as I want to talk to you about it and you are away today.
> Interesting parts of the thread dump:
> {code}
> "Thread-2" #11 prio=5 os_prio=0 tid=0xe13f0400 nid=0x4c49 waiting on condition [0xde4ed000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xe5ea9de8> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at org.jboss.as.server.BootstrapImpl$ShutdownHook.shutdown(BootstrapImpl.java:276)
> at org.jboss.as.server.BootstrapImpl$ShutdownHook.run(BootstrapImpl.java:240)
> "Controller Boot Thread" #25 prio=5 os_prio=0 tid=0xe0ca4c00 nid=0x4c35 waiting for monitor entry [0xdf3fe000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.Shutdown.exit(Shutdown.java:212)
> - waiting to lock <0xe31d5e18> (a java.lang.Class for java.lang.Shutdown)
> at java.lang.Runtime.exit(Runtime.java:109)
> at java.lang.System.exit(System.java:971)
> at org.jboss.as.server.SystemExiter$DefaultExiter.exit(SystemExiter.java:117)
> at org.jboss.as.server.SystemExiter.logAndExit(SystemExiter.java:98)
> at org.jboss.as.server.ServerService.boot(ServerService.java:405)
> at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:370)
> at java.lang.Thread.run(Thread.java:748)
> "MSC service thread 1-8" #20 prio=5 os_prio=0 tid=0x087b8c00 nid=0x4c2f in Object.wait() [0xe03ba000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe32a3f28> (a org.jboss.msc.service.ServiceRegistrationImpl)
> at java.lang.Object.wait(Object.java:502)
> at org.jboss.msc.service.Lockable.acquireWrite(Lockable.java:97)
> at org.jboss.msc.service.ServiceControllerImpl$RemoveTask.execute(ServiceControllerImpl.java:1865)
> - locked <0xe32a3f28> (a org.jboss.msc.service.ServiceRegistrationImpl)
> at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1527)
> at org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControllerImpl.java:788)
> at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1537)
> at org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControllerImpl.java:788)
> at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1537)
> at org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControllerImpl.java:788)
> at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1537)
> at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1979)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1481)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1374)
> at java.lang.Thread.run(Thread.java:748)
> "main" #1 prio=5 os_prio=0 tid=0xf6509000 nid=0x4c02 in Object.wait() [0xf6685000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe32b58e8> (a org.jboss.as.server.BootstrapImpl$ShutdownHook)
> at java.lang.Thread.join(Thread.java:1252)
> - locked <0xe32b58e8> (a org.jboss.as.server.BootstrapImpl$ShutdownHook)
> at java.lang.Thread.join(Thread.java:1326)
> at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106)
> at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
> at java.lang.Shutdown.runHooks(Shutdown.java:123)
> at java.lang.Shutdown.sequence(Shutdown.java:167)
> at java.lang.Shutdown.exit(Shutdown.java:212)
> - locked <0xe31d5e18> (a java.lang.Class for java.lang.Shutdown)
> at java.lang.Runtime.exit(Runtime.java:109)
> at java.lang.System.exit(System.java:971)
> at org.jboss.as.server.SystemExiter$DefaultExiter.exit(SystemExiter.java:117)
> at org.jboss.as.server.SystemExiter.logAndExit(SystemExiter.java:98)
> at org.jboss.as.server.DomainServerMain.main(DomainServerMain.java:183)
> at java.lang.invoke.LambdaForm$DMH/7468253.invokeStatic_L_V(LambdaForm$DMH)
> at java.lang.invoke.LambdaForm$MH/7742980.invokeExact_MT(LambdaForm$MH)
> at org.jboss.modules.Module.runMainMethod(Module.java:348)
> at org.jboss.modules.Module.run(Module.java:328)
> at org.jboss.modules.Main.main(Main.java:557)
> {code}
> This is a domain server. The "main" thread has recognized that its ProcessController has closed its stdin, so it is shutting down via System.exit.
> "Thread-2" is running BootstrapImpl.ShutdownHook, waiting on a latch for the MSC ServiceContainer to complete termination. So the SC not completing termination is the basic issue.
> "Controller Boot Thread" is there because this termination occurred during boot. That caused some problem during boot (not surprising) so it is responding to that problem by trying to terminate the process, via System.exit. It's blocking waiting for "main" which has done the same. This thread should not be preventing MSC terminating though; it's not, for example called as part of a StartContext.asynchronous thing. IOW I don't think this thread is relevant to the problem.
> "MSC service thread 1-8" is the most interesting one to me. An MSC thread is blocked but it's not clear to me why. An interesting frame in the stack is org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControllerImpl.java:788). That shows that ServiceControllerImpl$RemoveTask was passed to the executor but a RejectedExecutionException was thrown, so the task is being run from the thread that attempted to pass it to the executor. Should the MSC executor be rejecting tasks before all service controllers are removed?
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months
[JBoss JIRA] (WFCORE-3590) Hang in ServerStartFailureTestCase
by Richard Opalka (JIRA)
[ https://issues.jboss.org/browse/WFCORE-3590?page=com.atlassian.jira.plugi... ]
Richard Opalka reassigned WFCORE-3590:
--------------------------------------
Assignee: Richard Opalka (was: David Lloyd)
> Hang in ServerStartFailureTestCase
> ----------------------------------
>
> Key: WFCORE-3590
> URL: https://issues.jboss.org/browse/WFCORE-3590
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management, Server
> Affects Versions: 4.0.0.Alpha9
> Reporter: Brian Stansberry
> Assignee: Richard Opalka
> Priority: Critical
> Attachments: WFCORE-3590-threads.txt
>
>
> Hang observed in https://ci.wildfly.org/viewLog.html?buildId=88611&buildTypeId=WildFlyCore...
> I'll attach the thread dump.
> [~dmlloyd] I assigned this to you mostly as a form of ping, as I want to talk to you about it and you are away today.
> Interesting parts of the thread dump:
> {code}
> "Thread-2" #11 prio=5 os_prio=0 tid=0xe13f0400 nid=0x4c49 waiting on condition [0xde4ed000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xe5ea9de8> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at org.jboss.as.server.BootstrapImpl$ShutdownHook.shutdown(BootstrapImpl.java:276)
> at org.jboss.as.server.BootstrapImpl$ShutdownHook.run(BootstrapImpl.java:240)
> "Controller Boot Thread" #25 prio=5 os_prio=0 tid=0xe0ca4c00 nid=0x4c35 waiting for monitor entry [0xdf3fe000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.Shutdown.exit(Shutdown.java:212)
> - waiting to lock <0xe31d5e18> (a java.lang.Class for java.lang.Shutdown)
> at java.lang.Runtime.exit(Runtime.java:109)
> at java.lang.System.exit(System.java:971)
> at org.jboss.as.server.SystemExiter$DefaultExiter.exit(SystemExiter.java:117)
> at org.jboss.as.server.SystemExiter.logAndExit(SystemExiter.java:98)
> at org.jboss.as.server.ServerService.boot(ServerService.java:405)
> at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:370)
> at java.lang.Thread.run(Thread.java:748)
> "MSC service thread 1-8" #20 prio=5 os_prio=0 tid=0x087b8c00 nid=0x4c2f in Object.wait() [0xe03ba000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe32a3f28> (a org.jboss.msc.service.ServiceRegistrationImpl)
> at java.lang.Object.wait(Object.java:502)
> at org.jboss.msc.service.Lockable.acquireWrite(Lockable.java:97)
> at org.jboss.msc.service.ServiceControllerImpl$RemoveTask.execute(ServiceControllerImpl.java:1865)
> - locked <0xe32a3f28> (a org.jboss.msc.service.ServiceRegistrationImpl)
> at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1527)
> at org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControllerImpl.java:788)
> at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1537)
> at org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControllerImpl.java:788)
> at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1537)
> at org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControllerImpl.java:788)
> at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1537)
> at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1979)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1481)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1374)
> at java.lang.Thread.run(Thread.java:748)
> "main" #1 prio=5 os_prio=0 tid=0xf6509000 nid=0x4c02 in Object.wait() [0xf6685000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe32b58e8> (a org.jboss.as.server.BootstrapImpl$ShutdownHook)
> at java.lang.Thread.join(Thread.java:1252)
> - locked <0xe32b58e8> (a org.jboss.as.server.BootstrapImpl$ShutdownHook)
> at java.lang.Thread.join(Thread.java:1326)
> at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106)
> at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
> at java.lang.Shutdown.runHooks(Shutdown.java:123)
> at java.lang.Shutdown.sequence(Shutdown.java:167)
> at java.lang.Shutdown.exit(Shutdown.java:212)
> - locked <0xe31d5e18> (a java.lang.Class for java.lang.Shutdown)
> at java.lang.Runtime.exit(Runtime.java:109)
> at java.lang.System.exit(System.java:971)
> at org.jboss.as.server.SystemExiter$DefaultExiter.exit(SystemExiter.java:117)
> at org.jboss.as.server.SystemExiter.logAndExit(SystemExiter.java:98)
> at org.jboss.as.server.DomainServerMain.main(DomainServerMain.java:183)
> at java.lang.invoke.LambdaForm$DMH/7468253.invokeStatic_L_V(LambdaForm$DMH)
> at java.lang.invoke.LambdaForm$MH/7742980.invokeExact_MT(LambdaForm$MH)
> at org.jboss.modules.Module.runMainMethod(Module.java:348)
> at org.jboss.modules.Module.run(Module.java:328)
> at org.jboss.modules.Main.main(Main.java:557)
> {code}
> This is a domain server. The "main" thread has recognized that its ProcessController has closed its stdin, so it is shutting down via System.exit.
> "Thread-2" is running BootstrapImpl.ShutdownHook, waiting on a latch for the MSC ServiceContainer to complete termination. So the SC not completing termination is the basic issue.
> "Controller Boot Thread" is there because this termination occurred during boot. That caused some problem during boot (not surprising) so it is responding to that problem by trying to terminate the process, via System.exit. It's blocking waiting for "main" which has done the same. This thread should not be preventing MSC terminating though; it's not, for example called as part of a StartContext.asynchronous thing. IOW I don't think this thread is relevant to the problem.
> "MSC service thread 1-8" is the most interesting one to me. An MSC thread is blocked but it's not clear to me why. An interesting frame in the stack is org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControllerImpl.java:788). That shows that ServiceControllerImpl$RemoveTask was passed to the executor but a RejectedExecutionException was thrown, so the task is being run from the thread that attempted to pass it to the executor. Should the MSC executor be rejecting tasks before all service controllers are removed?
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months
[JBoss JIRA] (WFLY-9788) EJB over HTTP fails with Arrays in Request
by jaikiran pai (JIRA)
[ https://issues.jboss.org/browse/WFLY-9788?page=com.atlassian.jira.plugin.... ]
jaikiran pai commented on WFLY-9788:
------------------------------------
A PR containing a potential fix has been submitted https://github.com/wildfly/wildfly-http-client/pull/16
> EJB over HTTP fails with Arrays in Request
> ------------------------------------------
>
> Key: WFLY-9788
> URL: https://issues.jboss.org/browse/WFLY-9788
> Project: WildFly
> Issue Type: Bug
> Components: EJB
> Affects Versions: 11.0.0.Final
> Reporter: Heiko Lettmann
> Attachments: Test.zip
>
>
> I stumbled over the issue WFLY-9573. Then I updated to wildfly-http-client-1.0.9.Final which made a few invocations work. There I discovered another issue. I attached a modified Quickstart version to demonstrate it!
> Exception is on the client side:
> Exception in thread "main" javax.ejb.EJBException: java.io.IOException: java.lang.ClassCastException: java.lang.Class cannot be cast to java.lang.String
> at org.jboss.as.ejb3.remote.AssociationImpl.receiveInvocationRequest(AssociationImpl.java:128)
> at org.wildfly.httpclient.ejb.HttpInvocationHandler.lambda$handleInternal$0(HttpInvocationHandler.java:130)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: java.lang.ClassCastException: java.lang.Class cannot be cast to java.lang.String
> at org.wildfly.httpclient.ejb.HttpInvocationHandler$1.getRequestContent(HttpInvocationHandler.java:204)
> at org.jboss.as.ejb3.remote.AssociationImpl.receiveInvocationRequest(AssociationImpl.java:126)
> ... 4 more
> Caused by: java.lang.ClassCastException: java.lang.Class cannot be cast to java.lang.String
> at org.wildfly.httpclient.ejb.HttpInvocationHandler$1.getRequestContent(HttpInvocationHandler.java:178)
> ... 5 more
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months