[jboss-jira] [JBoss JIRA] (JGRP-2250) Stuck Threads - BLOCKED infinitely on - JGroups - org.jgroups.protocols.FlowControl$Credit.decrementIfEnoughCredits(FlowControl.java:567)

[Rohit Singh (JIRA)]

Rohit Singh created JGRP-2250:
---------------------------------

             Summary: Stuck Threads - BLOCKED infinitely on - JGroups - org.jgroups.protocols.FlowControl$Credit.decrementIfEnoughCredits(FlowControl.java:567)
                 Key: JGRP-2250
                 URL: https://issues.jboss.org/browse/JGRP-2250
             Project: JGroups
          Issue Type: Bug
    Affects Versions: 3.4.1
         Environment: Oracle Linux - Exalogic/Exadata stack
AS: Weblogic 12.1.3
DB: Oracle 12.1.0.2
            Reporter: Rohit Singh
            Assignee: Bela Ban
            Priority: Blocker


Ours is a  J2EE application (clustered across 4 nodes) internally using *{color:red}Infinispan Cache (Ver 6.0.2) with JGroups (Ver 3.4.1){color}* as transport layer of our cache.

Application had been running perfectly fine, but suddenly on one node, one cache-put got stuck with below stack trace. *(Cause being blocked on org.jgroups.protocols.FlowControl$Credit.decrementIfEnoughCredits(FlowControl.java:567)).*

Now all further puts are getting stuck - blocked - on the same line on the same node. Other nodes are working fine in this respect.

Currently, while raising the issue, the threads are still in stuck state. And the first stuck thread is stuck from last 27 hours.

Stack Trace:
"[STUCK] ExecuteThread: '37' for queue: 'weblogic.kernel.Default (self-tuning)'" #107862 daemon prio=1 os_prio=0 tid=0x00007f5964373000 nid=0x75a9 in Object.wait() [0x00007f57d596b000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at *{color:red}org.jgroups.protocols.FlowControl$Credit.decrementIfEnoughCredits(FlowControl.java:567){color}*
	- locked <0x00007f5e64893168> (a org.jgroups.protocols.FlowControl$Credit)
	at org.jgroups.protocols.UFC.handleDownMessage(UFC.java:121)
	at org.jgroups.protocols.FlowControl.down(FlowControl.java:330)
	at org.jgroups.protocols.FlowControl.down(FlowControl.java:340)
	at org.jgroups.protocols.FRAG2.down(FRAG2.java:136)
	at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:1024)
	at org.jgroups.JChannel.down(JChannel.java:760)
	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:683)
	at org.jgroups.blocks.RequestCorrelator.sendUnicastRequest(RequestCorrelator.java:202)
	at org.jgroups.blocks.UnicastRequest.sendRequest(UnicastRequest.java:43)
	at org.jgroups.blocks.Request.execute(Request.java:83)
	at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:399)
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:353)
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:167)
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:521)
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:281)
	at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:233)
	at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:72)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
	at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:321)
	at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:402)
	at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:164)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
	at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:68)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
	at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:219)
	at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:141)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
	at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:148)
	at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:134)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)
	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70)
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
	at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1306)
	at org.infinispan.CacheImpl.putInternal(CacheImpl.java:878)
	at org.infinispan.CacheImpl.put(CacheImpl.java:870)
	at org.infinispan.DecoratedCache.put(DecoratedCache.java:401)
	at com.nucleus.finnone.pro.cache.common.NeutrinoCache.put(NeutrinoCache.java:18)
	at com.nucleus.config.persisted.service.ConfigurationServiceImpl.getConfigurationGroupFor(ConfigurationServiceImpl.java:478)


*Note: Seems related to below-*
https://issues.jboss.org/browse/JGRP-1675
https://issues.jboss.org/browse/ISPN-3645



--
This message was sent by Atlassian JIRA
(v7.5.0#75005)