[JBoss JIRA] (ISPN-3169) Define a Query API that will be common for library mode + remote
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3169?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-3169:
--------------------------------
Description:
Some notes on the query API:
- fluent, supporting a predefined set of filters (equality, range, etc). No pluggable are supported as this API will be available both embedded and *remotely* by the Java HR client
- the filters accept the schema(protobuf) primitive types as arguments (int, string, byte)
- should support paging: start page + pageSize
- projections are not supported
For an overview on the remote query design refer to: https://community.jboss.org/wiki/QueryingDesignInInfinispan
was:
Some notes on the query API:
- fluent, supporting a predefined set of filters (equality, range, etc). No pluggable are supported as this API will be available both embedded and *remotely* by the Java HR client
- should support paging: start page + pageSize
- projections are not supported
For an overview on the remote query design refer to: https://community.jboss.org/wiki/QueryingDesignInInfinispan
> Define a Query API that will be common for library mode + remote
> ----------------------------------------------------------------
>
> Key: ISPN-3169
> URL: https://issues.jboss.org/browse/ISPN-3169
> Project: Infinispan
> Issue Type: Feature Request
> Reporter: Mircea Markus
> Assignee: Mircea Markus
>
> Some notes on the query API:
> - fluent, supporting a predefined set of filters (equality, range, etc). No pluggable are supported as this API will be available both embedded and *remotely* by the Java HR client
> - the filters accept the schema(protobuf) primitive types as arguments (int, string, byte)
> - should support paging: start page + pageSize
> - projections are not supported
> For an overview on the remote query design refer to: https://community.jboss.org/wiki/QueryingDesignInInfinispan
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-2240) Per-key lock container leads to superfluous TimeoutExceptions on concurrent access to same key
by Robert Stupp (JIRA)
[ https://issues.jboss.org/browse/ISPN-2240?page=com.atlassian.jira.plugin.... ]
Robert Stupp commented on ISPN-2240:
------------------------------------
This issue is still reproducable with 5.3.0.CR1
{{org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [10 seconds] on key [.........] for requestor [Thread[remote-thread-27,5,main]]! Lock held by [Thread[remote-thread-23,5,main]]}}
Current stack traces are:
{noformat}
2013-06-04 18:09:46,682 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (mpx-request-Invocation-8100-87) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Node ................. timed out
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:195) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:158) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:91) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.distribution.L1NonTxInterceptor.handleDataWriteCommand(L1NonTxInterceptor.java:152) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.distribution.L1NonTxInterceptor.visitPutKeyValueCommand(L1NonTxInterceptor.java:108) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.distribution.L1NonTxInterceptor.handleDataWriteCommand(L1NonTxInterceptor.java:152) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.distribution.L1NonTxInterceptor.visitPutKeyValueCommand(L1NonTxInterceptor.java:108) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:290) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:68) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:133) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1337) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.CacheImpl.putInternal(CacheImpl.java:898) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.CacheImpl.put(CacheImpl.java:890) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.CacheImpl.put(CacheImpl.java:883) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.CacheImpl.put(CacheImpl.java:234) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
{noformat}
{noformat}
2013-06-04 18:08:56,815 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (mpx-request-Invocation-8100-52) ISPN000136: Execution error: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from .................., see cause for remote stack trace
at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:70) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:381) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:188) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:158) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:91) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.distribution.L1NonTxInterceptor.handleDataWriteCommand(L1NonTxInterceptor.java:152) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.distribution.L1NonTxInterceptor.visitPutKeyValueCommand(L1NonTxInterceptor.java:108) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:290) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:68) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:133) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1337) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.CacheImpl.putInternal(CacheImpl.java:898) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.CacheImpl.put(CacheImpl.java:890) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.CacheImpl.put(CacheImpl.java:883) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.CacheImpl.put(CacheImpl.java:234) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
...
Caused by: org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [10 seconds] on key [.........] for requestor [Thread[remote-thread-27,5,main]]! Lock held by [Thread[remote-thread-23,5,main]]
at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:214) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLockNoCheck(LockManagerImpl.java:197) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockKey(AbstractLockingInterceptor.java:149) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockKey(AbstractLockingInterceptor.java:145) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:69) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:133) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:61) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:70) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:68) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:194) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]
... 1 more
{noformat}
> Per-key lock container leads to superfluous TimeoutExceptions on concurrent access to same key
> ----------------------------------------------------------------------------------------------
>
> Key: ISPN-2240
> URL: https://issues.jboss.org/browse/ISPN-2240
> Project: Infinispan
> Issue Type: Bug
> Components: Locking and Concurrency
> Affects Versions: 5.1.6.FINAL, 5.1.x
> Reporter: Robert Stupp
> Assignee: Mircea Markus
> Fix For: 5.3.0.Final
>
> Attachments: ISPN-2240_fix_TimeoutExceptions.patch, somehow.zip
>
>
> Hi,
> I've encountered a lot of TimeoutExceptions just running a load test against an infinispan cluster.
> I tracked down the reason and found out, that the code in org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer#releaseLock() causes these superfluous TimeoutExceptions.
> A small test case (which just prints out timeouts, too late timeouts and "paints" a lot of dots to the console - more dots/second on the console means better throughput ;-)
> In a short test I extended the class ReentrantPerEntryLockContainer and changed the implementation of releaseLock() as follows:
> {noformat}
> public void releaseLock(Object lockOwner, Object key) {
> ReentrantLock l = locks.get(key);
> if (l != null) {
> if (!l.isHeldByCurrentThread())
> throw new IllegalStateException("Lock for [" + key + "] not held by current thread " + Thread.currentThread());
> while (l.isHeldByCurrentThread())
> unlock(l, lockOwner);
> if (!l.hasQueuedThreads())
> locks.remove(key);
> }
> else
> throw new IllegalStateException("No lock for [" + key + ']');
> }
> {noformat}
> The main improvement is that locks are not removed from the concurrent map as long as other threads are waiting on that lock.
> If the lock is removed from the map while other threads are waiting for it, they may run into timeouts and force TimeoutExceptions to the client.
> The above methods "paints more dots per second" - means: it gives a better throughput for concurrent accesses to the same key.
> The re-implemented method should also fix some replication timeout exceptions.
> Please, please add this to 5.1.7, if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-2710) Huge amount of OOB threads during performance test
by Robert Stupp (JIRA)
[ https://issues.jboss.org/browse/ISPN-2710?page=com.atlassian.jira.plugin.... ]
Robert Stupp commented on ISPN-2710:
------------------------------------
With 5.3.0.CR1 there is no very huge amount of OOB threads - although I did not succeed with a load/performance test.
The amount of OOB threads is still high - not sure whether this is critical.
A huge amount of "org.infinispan.util.concurrent.TimeoutException"s occured:
{noformat}
org.infinispan.util.concurrent.TimeoutException: Node SOME-JGROUPS-ADDRESS timed out
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:195) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:158) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:91) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.distribution.L1NonTxInterceptor.handleDataWriteCommand(L1NonTxInterceptor.java:152) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.distribution.L1NonTxInterceptor.visitPutKeyValueCommand(L1NonTxInterceptor.java:108) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:290) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:68) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:133) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1337) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.CacheImpl.putInternal(CacheImpl.java:898) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.CacheImpl.put(CacheImpl.java:890) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.CacheImpl.put(CacheImpl.java:883) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
at org.infinispan.CacheImpl.put(CacheImpl.java:234) [infinispan-core-5.3.0.CR1.jar:5.3.0.CR1]
{noformat}
> Huge amount of OOB threads during performance test
> --------------------------------------------------
>
> Key: ISPN-2710
> URL: https://issues.jboss.org/browse/ISPN-2710
> Project: Infinispan
> Issue Type: Feature Request
> Affects Versions: 5.2.0.CR1
> Reporter: Robert Stupp
> Assignee: Mircea Markus
>
> While running our performance test (as described in ISPN-2240), two of the four servers are running at 80 to 100% CPU - while the others just run at 10%.
> Before that phenomenom a huge amount (several 100s) of threads has been created (all called {{OOB-xxxx}}.
> The performance test just reads cached data - there was no cache put operation at that time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-3173) Define a new query operation over hotrod
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3173?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-3173:
--------------------------------
Summary: Define a new query operation over hotrod (was: Add a query operation over java hotrod client )
> Define a new query operation over hotrod
> ----------------------------------------
>
> Key: ISPN-3173
> URL: https://issues.jboss.org/browse/ISPN-3173
> Project: Infinispan
> Issue Type: Feature Request
> Reporter: Mircea Markus
> Assignee: Mircea Markus
>
> HR request: [op_id] [payload].
> The request payload is query specific (not defined in the HR protocol) and at this stage has the following format: [Q_TYPE] [QUERY_ST] [FIRST_INDEX] [PAGE_SIZE]. This format accommodates the remote query requirements as defined in ISPN-3169.
> - Q_TYPE (protobuf's byte) and query identifier, 1 for JPAQL (this is the query type we'll support). In future we'll add different query types as well.
> - QUERY_STRING (protobuf's string)- JPAQL string generated by the fluent API (ISPN-3169). Parameters are encoded in this string (vs being sent separately)
> - FIRST_INDEX + PAGE_SIZE (protobuf's int)- used for paginating/iterating over the result set.
> HR response: [HR_SUCESS_FLAG] [payload]
> PAYLOAD = [NUM_EL] [PROJ_SIZE] [ELEMENTS]
> - even though at this stage we don't support projections (see ISPN-3169) PROJ_SIZE is added for future compatibility when projection will be supported.
> Note that the payload for both request and response should be marshalled with protobuf as this information is read/written over multiple clients.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-2836) org.jgroups.TimeoutException after invoking MapCombineCommand in Map/Reduce task with 2 nodes
by Alan Field (JIRA)
[ https://issues.jboss.org/browse/ISPN-2836?page=com.atlassian.jira.plugin.... ]
Alan Field commented on ISPN-2836:
----------------------------------
Thanks [~pruivo], I'll try running with the updated code and a larger replication timeout. I'm not sure what the suggested value for the replication timeout is for a cluster, whether it uses Map Reduce or not.
> org.jgroups.TimeoutException after invoking MapCombineCommand in Map/Reduce task with 2 nodes
> ---------------------------------------------------------------------------------------------
>
> Key: ISPN-2836
> URL: https://issues.jboss.org/browse/ISPN-2836
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce
> Affects Versions: 5.2.1.Final
> Reporter: Alan Field
> Assignee: Pedro Ruivo
> Priority: Blocker
> Labels: onboard
> Fix For: 5.3.0.Final
>
> Attachments: afield-tcp-521-final.txt, benchmark-mapreduce-multifilesize.xml, dist-udp-no-tx.xml, jgroups-udp.xml, udp-edg-perf01.txt, udp-edg-perf02.txt
>
>
> Using RadarGun and two nodes to execute the example WordCount Map/Reduce job against a cache with ~550 keys with a value size of 1MB is producing a thread deadlock. The cache is distributed with transactions disabled.
> TCP transport deadlocks without throwing an exception. Disabling the send queue and setting UNICAST2.conn_expiry_timeout=0 prevents the deadlock, but the job does not complete. The nodes send "are-you-alive" messages back and forth, and I have seen the following exception:
> {noformat}
> 11:44:29,970 ERROR [org.jgroups.protocols.TCP] (OOB-98,default,edg-perf01-1907) failed sending message to edg-perf02-32536 (76 bytes): java.net.SocketException: Socket closed, cause: null
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:352)
> at org.radargun.cachewrappers.InfinispanMapReduceWrapper.executeMapReduceTask(InfinispanMapReduceWrapper.java:98)
> at org.radargun.stages.MapReduceStage.executeOnSlave(MapReduceStage.java:74)
> at org.radargun.Slave$2.run(Slave.java:103)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.util.concurrent.ExecutionException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to edg-perf02-32536
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.infinispan.distexec.mapreduce.MapReduceTask$TaskPart.get(MapReduceTask.java:832)
> at org.infinispan.distexec.mapreduce.MapReduceTask.executeMapPhaseWithLocalReduction(MapReduceTask.java:477)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:350)
> ... 9 more
> Caused by: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to edg-perf02-32536
> at org.infinispan.util.Util.rewrapAsCacheException(Util.java:541)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515)
> 11:44:29,978 ERROR [org.jgroups.protocols.TCP] (Timer-3,default,edg-perf01-1907) failed sending message to edg-perf02-32536 (60 bytes): java.net.SocketException: Socket closed, cause: null
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:175)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:197)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:254)
> at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:80)
> at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:288)
> ... 5 more
> Caused by: org.jgroups.TimeoutException: timeout sending message to edg-perf02-32536
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:390)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:301)
> 11:44:29,979 ERROR [org.jgroups.protocols.TCP] (Timer-4,default,edg-perf01-1907) failed sending message to edg-perf02-32536 (63 bytes): java.net.SocketException: Socket closed, cause: null
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179)
> ... 11 more
> {noformat}
> With UDP transport, both threads are deadlocked. I will attach thread dumps from runs using TCP and UDP transport.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-2836) org.jgroups.TimeoutException after invoking MapCombineCommand in Map/Reduce task with 2 nodes
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-2836?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo commented on ISPN-2836:
-----------------------------------
#1: this is the code I'm using: https://github.com/pruivo/radargun/tree/ISPN-2836
#2: yes. in my laptop, it was executing the map phase locally (duration 1min 30sec +-) then it waits for the reply from the remote (the waiting time is given by Sync.replTimeout if I'm not mistaken).
> org.jgroups.TimeoutException after invoking MapCombineCommand in Map/Reduce task with 2 nodes
> ---------------------------------------------------------------------------------------------
>
> Key: ISPN-2836
> URL: https://issues.jboss.org/browse/ISPN-2836
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce
> Affects Versions: 5.2.1.Final
> Reporter: Alan Field
> Assignee: Pedro Ruivo
> Priority: Blocker
> Labels: onboard
> Fix For: 5.3.0.Final
>
> Attachments: afield-tcp-521-final.txt, benchmark-mapreduce-multifilesize.xml, dist-udp-no-tx.xml, jgroups-udp.xml, udp-edg-perf01.txt, udp-edg-perf02.txt
>
>
> Using RadarGun and two nodes to execute the example WordCount Map/Reduce job against a cache with ~550 keys with a value size of 1MB is producing a thread deadlock. The cache is distributed with transactions disabled.
> TCP transport deadlocks without throwing an exception. Disabling the send queue and setting UNICAST2.conn_expiry_timeout=0 prevents the deadlock, but the job does not complete. The nodes send "are-you-alive" messages back and forth, and I have seen the following exception:
> {noformat}
> 11:44:29,970 ERROR [org.jgroups.protocols.TCP] (OOB-98,default,edg-perf01-1907) failed sending message to edg-perf02-32536 (76 bytes): java.net.SocketException: Socket closed, cause: null
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:352)
> at org.radargun.cachewrappers.InfinispanMapReduceWrapper.executeMapReduceTask(InfinispanMapReduceWrapper.java:98)
> at org.radargun.stages.MapReduceStage.executeOnSlave(MapReduceStage.java:74)
> at org.radargun.Slave$2.run(Slave.java:103)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.util.concurrent.ExecutionException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to edg-perf02-32536
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.infinispan.distexec.mapreduce.MapReduceTask$TaskPart.get(MapReduceTask.java:832)
> at org.infinispan.distexec.mapreduce.MapReduceTask.executeMapPhaseWithLocalReduction(MapReduceTask.java:477)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:350)
> ... 9 more
> Caused by: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to edg-perf02-32536
> at org.infinispan.util.Util.rewrapAsCacheException(Util.java:541)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515)
> 11:44:29,978 ERROR [org.jgroups.protocols.TCP] (Timer-3,default,edg-perf01-1907) failed sending message to edg-perf02-32536 (60 bytes): java.net.SocketException: Socket closed, cause: null
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:175)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:197)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:254)
> at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:80)
> at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:288)
> ... 5 more
> Caused by: org.jgroups.TimeoutException: timeout sending message to edg-perf02-32536
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:390)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:301)
> 11:44:29,979 ERROR [org.jgroups.protocols.TCP] (Timer-4,default,edg-perf01-1907) failed sending message to edg-perf02-32536 (63 bytes): java.net.SocketException: Socket closed, cause: null
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179)
> ... 11 more
> {noformat}
> With UDP transport, both threads are deadlocked. I will attach thread dumps from runs using TCP and UDP transport.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-2836) org.jgroups.TimeoutException after invoking MapCombineCommand in Map/Reduce task with 2 nodes
by Alan Field (JIRA)
[ https://issues.jboss.org/browse/ISPN-2836?page=com.atlassian.jira.plugin.... ]
Alan Field edited comment on ISPN-2836 at 6/4/13 11:38 AM:
-----------------------------------------------------------
[~pruivo] OK, so I have a couple of questions:
1) Can you send me the updated executeMapReduceTask() code?
2) Does the cache configuration need to be changed to avoid the TimeoutException with larger amounts of data in the cache?
The problem with the characters is that the bytes from the file are not being decoded properly. The latest version of LoadFileStage doesn't decode the bytes at all, but to put String values in the cache, this is necessary. I'm trying to determine the best approach here for data loading stages now.
was (Author: afield):
[~pruivo] OK, so I have a couple of questions:
1) Can you send me the updated executeMapReduceTask() code?
2) Does the cache configuration need to be changed to avoid the TimeoutException with larger amounts of data in the cache?
> org.jgroups.TimeoutException after invoking MapCombineCommand in Map/Reduce task with 2 nodes
> ---------------------------------------------------------------------------------------------
>
> Key: ISPN-2836
> URL: https://issues.jboss.org/browse/ISPN-2836
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce
> Affects Versions: 5.2.1.Final
> Reporter: Alan Field
> Assignee: Pedro Ruivo
> Priority: Blocker
> Labels: onboard
> Fix For: 5.3.0.Final
>
> Attachments: afield-tcp-521-final.txt, benchmark-mapreduce-multifilesize.xml, dist-udp-no-tx.xml, jgroups-udp.xml, udp-edg-perf01.txt, udp-edg-perf02.txt
>
>
> Using RadarGun and two nodes to execute the example WordCount Map/Reduce job against a cache with ~550 keys with a value size of 1MB is producing a thread deadlock. The cache is distributed with transactions disabled.
> TCP transport deadlocks without throwing an exception. Disabling the send queue and setting UNICAST2.conn_expiry_timeout=0 prevents the deadlock, but the job does not complete. The nodes send "are-you-alive" messages back and forth, and I have seen the following exception:
> {noformat}
> 11:44:29,970 ERROR [org.jgroups.protocols.TCP] (OOB-98,default,edg-perf01-1907) failed sending message to edg-perf02-32536 (76 bytes): java.net.SocketException: Socket closed, cause: null
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:352)
> at org.radargun.cachewrappers.InfinispanMapReduceWrapper.executeMapReduceTask(InfinispanMapReduceWrapper.java:98)
> at org.radargun.stages.MapReduceStage.executeOnSlave(MapReduceStage.java:74)
> at org.radargun.Slave$2.run(Slave.java:103)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.util.concurrent.ExecutionException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to edg-perf02-32536
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.infinispan.distexec.mapreduce.MapReduceTask$TaskPart.get(MapReduceTask.java:832)
> at org.infinispan.distexec.mapreduce.MapReduceTask.executeMapPhaseWithLocalReduction(MapReduceTask.java:477)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:350)
> ... 9 more
> Caused by: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to edg-perf02-32536
> at org.infinispan.util.Util.rewrapAsCacheException(Util.java:541)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515)
> 11:44:29,978 ERROR [org.jgroups.protocols.TCP] (Timer-3,default,edg-perf01-1907) failed sending message to edg-perf02-32536 (60 bytes): java.net.SocketException: Socket closed, cause: null
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:175)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:197)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:254)
> at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:80)
> at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:288)
> ... 5 more
> Caused by: org.jgroups.TimeoutException: timeout sending message to edg-perf02-32536
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:390)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:301)
> 11:44:29,979 ERROR [org.jgroups.protocols.TCP] (Timer-4,default,edg-perf01-1907) failed sending message to edg-perf02-32536 (63 bytes): java.net.SocketException: Socket closed, cause: null
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179)
> ... 11 more
> {noformat}
> With UDP transport, both threads are deadlocked. I will attach thread dumps from runs using TCP and UDP transport.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months