[jboss-jira] [JBoss JIRA] (JGRP-2250) Stuck Threads - BLOCKED infinitely on - JGroups - org.jgroups.protocols.FlowControl$Credit.decrementIfEnoughCredits(FlowControl.java:567)

Stuck Threads - BLOCKED infinitely on - JGroups - org.jgroups.protocols.FlowControl$Credit.decrementIfEnoughCredits(FlowControl.java:567) ----------------------------------------------------------------------------------------------------------------------------------------- Key: JGRP-2250 URL: https://issues.jboss.org/browse/JGRP-2250 Project: JGroups Issue Type: Bug Affects Versions: 3.4.1 Environment: Oracle Linux - Exalogic/Exadata stack AS: Weblogic 12.1.3 DB: Oracle 12.1.0.2 Reporter: Rohit Singh Assignee: Bela Ban Priority: Blocker Labels: Infinispan, JGroups Ours is a J2EE application (clustered across 4 nodes) internally using *{color:red}Infinispan Cache (Ver 6.0.2) with JGroups (Ver 3.4.1){color}* as transport layer of our cache. Application had been running perfectly fine, but suddenly on one node, one cache-put got stuck with below stack trace. *(Cause being blocked on org.jgroups.protocols.FlowControl$Credit.decrementIfEnoughCredits(FlowControl.java:567)).* Now all further puts are getting stuck - blocked - on the same line on the same node. Other nodes are working fine in this respect. Currently, while raising the issue, the threads are still in stuck state. And the first stuck thread is stuck from last 27 hours. Stack Trace: "[STUCK] ExecuteThread: '37' for queue: 'weblogic.kernel.Default (self-tuning)'" #107862 daemon prio=1 os_prio=0 tid=0x00007f5964373000 nid=0x75a9 in Object.wait() [0x00007f57d596b000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at *{color:red}org.jgroups.protocols.FlowControl$Credit.decrementIfEnoughCredits(FlowControl.java:567){color}* - locked <0x00007f5e64893168> (a org.jgroups.protocols.FlowControl$Credit) at org.jgroups.protocols.UFC.handleDownMessage(UFC.java:121) at org.jgroups.protocols.FlowControl.down(FlowControl.java:330) at org.jgroups.protocols.FlowControl.down(FlowControl.java:340) at org.jgroups.protocols.FRAG2.down(FRAG2.java:136) at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:1024) at org.jgroups.JChannel.down(JChannel.java:760) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:683) at org.jgroups.blocks.RequestCorrelator.sendUnicastRequest(RequestCorrelator.java:202) at org.jgroups.blocks.UnicastRequest.sendRequest(UnicastRequest.java:43) at org.jgroups.blocks.Request.execute(Request.java:83) at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:399) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:353) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:167) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:521) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:281) at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:233) at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:72) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:321) at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:402) at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:164) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:68) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:219) at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:141) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:148) at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:134) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73) at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333) at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1306) at org.infinispan.CacheImpl.putInternal(CacheImpl.java:878) at org.infinispan.CacheImpl.put(CacheImpl.java:870) at org.infinispan.DecoratedCache.put(DecoratedCache.java:401) at com.nucleus.finnone.pro.cache.common.NeutrinoCache.put(NeutrinoCache.java:18) at com.nucleus.config.persisted.service.ConfigurationServiceImpl.getConfigurationGroupFor(ConfigurationServiceImpl.java:478) *Note: Seems related to below-* https://issues.jboss.org/browse/JGRP-1675 https://issues.jboss.org/browse/ISPN-3645