[infinispan-issues] [JBoss JIRA] (ISPN-2710) Huge amount of OOB threads during performance test

Tue Jun 4 12:18:54 EDT 2013

    [ https://issues.jboss.org/browse/ISPN-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779260#comment-12779260 ] 

Robert Stupp commented on ISPN-2710:
------------------------------------

With 5.3.0.CR1 there is no very huge amount of OOB threads - although I did not succeed with a load/performance test.
The amount of OOB threads is still high - not sure whether this is critical.

A huge amount of "org.infinispan.util.concurrent.TimeoutException"s occured:
{noformat}
org.infinispan.util.concurrent.TimeoutException: Node SOME-JGROUPS-ADDRESS timed out
        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:195) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:158) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:91) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.interceptors.distribution.L1NonTxInterceptor.handleDataWriteCommand(L1NonTxInterceptor.java:152) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.interceptors.distribution.L1NonTxInterceptor.visitPutKeyValueCommand(L1NonTxInterceptor.java:108) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:290) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:68) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:133) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1337) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.CacheImpl.putInternal(CacheImpl.java:898) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.CacheImpl.put(CacheImpl.java:890) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.CacheImpl.put(CacheImpl.java:883) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
        at org.infinispan.CacheImpl.put(CacheImpl.java:234) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
{noformat}

> Huge amount of OOB threads during performance test
> --------------------------------------------------
>
>                 Key: ISPN-2710
>                 URL: https://issues.jboss.org/browse/ISPN-2710
>             Project: Infinispan
>          Issue Type: Feature Request
>    Affects Versions: 5.2.0.CR1
>            Reporter: Robert Stupp
>            Assignee: Mircea Markus
>
> While running our performance test (as described in ISPN-2240), two of the four servers are running at 80 to 100% CPU - while the others just run at 10%.
> Before that phenomenom a huge amount (several 100s) of threads has been created (all called {{OOB-xxxx}}.
> The performance test just reads cached data - there was no cache put operation at that time.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira