[JBoss JIRA] (ISPN-3443) WriteCommand may be ignored during state transfer
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3443?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-3443:
--------------------------------
Labels: 620 nbst (was: 620)
> WriteCommand may be ignored during state transfer
> -------------------------------------------------
>
> Key: ISPN-3443
> URL: https://issues.jboss.org/browse/ISPN-3443
> Project: Infinispan
> Issue Type: Bug
> Components: Locking and Concurrency, State transfer
> Affects Versions: 6.0.0.Alpha3
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Blocker
> Labels: 620, nbst
> Fix For: 6.0.0.CR1
>
>
> Distributed sync non-tx cache.
> Situation:
> 1) A node is joining the cluster, requesting some segment
> 2) RemoveCommand is sent to backup owner with ignorePreviousValue=true
> 3) It looks up the entry and finds null
> 4) State transfer invokes the PutKeyValueCommand and sets the value for removed entry (updateKeys has not the key yet)
> 5) RemoveCommand adds its key to updateKeys set, but it does not remove the value as it is already null (in its context)
> Result: the value is removed on primary but on backup this is still present
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-3051) Allow configuring the number of segments per node
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3051?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-3051:
--------------------------------
Labels: nbst (was: )
> Allow configuring the number of segments per node
> -------------------------------------------------
>
> Key: ISPN-3051
> URL: https://issues.jboss.org/browse/ISPN-3051
> Project: Infinispan
> Issue Type: Feature Request
> Components: Distributed Cache
> Reporter: Guillermo GARCIA OCHOA
> Assignee: Dan Berindei
> Priority: Critical
> Labels: nbst
> Fix For: 6.0.0.Beta1, 6.0.0.Final
>
>
> This should allow for the following use cases:
> - a node to take more load
> - a node to take no load
> A simple way for specifying this would be to configure a load factor per node, e.g. more powerful machine would be 2*x and that would mean that it would take twice the load of an "ordinary" machine in the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-3449) ReplaceCommand with ignorePrevValue=true does not work on null entries
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3449?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-3449:
--------------------------------
Labels: 620 nbst (was: 620)
> ReplaceCommand with ignorePrevValue=true does not work on null entries
> ----------------------------------------------------------------------
>
> Key: ISPN-3449
> URL: https://issues.jboss.org/browse/ISPN-3449
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 6.0.0.Alpha3
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
> Labels: 620, nbst
> Fix For: 6.0.0.Beta2
>
>
> If ReplaceCommand with ignorePrevValue=true is executed on a backup owner node which has not the entry in the container (for example because the state transfer was not completed yet), the EntryWrappingInterceptor/EntryFactoryImpl won't put it the command's context. Then, in the ReplaceCommand.perform() the entry is not replaced and the command fails.
> The command on primary owner succeeds as it does not check whether the responses are successful.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-3527) Transaction boundary commands can be completed before state transfer applies an active transaction
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3527?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-3527:
--------------------------------
Labels: nbst (was: )
> Transaction boundary commands can be completed before state transfer applies an active transaction
> --------------------------------------------------------------------------------------------------
>
> Key: ISPN-3527
> URL: https://issues.jboss.org/browse/ISPN-3527
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.7.Final
> Reporter: Erik Salter
> Assignee: Dan Berindei
> Labels: nbst
> Fix For: 6.0.0.Beta2, 6.0.0.Final
>
>
> There is a condition where a transaction boundary command, like a RollbackCommand, can be processed by the new owner of a tx-encapsulated key before it applies the active transaction.
> On a view change:
> The forwarding node waits until its transactions from the new topology ID is set before issuing the command.
> 2013-09-17 14:28:33,725 TRACE [org.infinispan.interceptors.InvocationContextInterceptor] (OOB-350,session-resource-cluster,east-dht5-60816(CMC-Denver-CO)) Invoked with command RollbackCommand {gtx=GlobalTransaction:<east-dht5-60816(CMC-Denver-CO)>:48002:local, cacheName='eigAllocation', topologyId=-1} and InvocationContext [org.infinispan.context.impl.LocalTxInvocationContext@1f225e51]
> 2013-09-17 14:28:33,727 TRACE [org.infinispan.statetransfer.StateTransferInterceptor] (OOB-350,session-resource-cluster,east-dht5-60816(CMC-Denver-CO)) handleTopologyAffectedCommand for command RollbackCommand {gtx=GlobalTransaction:<east-dht5-60816(CMC-Denver-CO)>:48002:local, cacheName='eigAllocation', topologyId=-1}
> 2013-09-17 14:28:33,727 TRACE [org.infinispan.statetransfer.StateTransferLockImpl] (OOB-350,session-resource-cluster,east-dht5-60816(CMC-Denver-CO)) Waiting for transaction data for topology 26, current topology is 25
> ...
> 2013-09-17 14:28:34,101 TRACE [org.infinispan.remoting.rpc.RpcManagerImpl] (OOB-350,session-resource-cluster,east-dht5-60816(CMC-Denver-CO)) east-dht5-60816(CMC-Denver-CO) broadcasting call RollbackCommand {gtx=GlobalTransaction:<east-dht5-60816(CMC-Denver-CO)>:48002:local, cacheName='eigAllocation', topologyId=26} to recipient list [west-dht4-48045(CH2-Chicago-IL), east-dht5-60816(CMC-Denver-CO), east-dht2-60243(CMC-Denver-CO)]
> However, the receiving node has not finished applying the transactions from the forwarding node yet for this topology ID. So the tx cannot be found, resulting in a stale tx and an unusable key.
> 2013-09-17 14:28:34,116 TRACE [org.infinispan.remoting.InboundInvocationHandlerImpl] (OOB-153,session-resource-cluster,west-dht4-48045(CH2-Chicago-IL)) Calling perform() on RollbackCommand {gtx=GlobalTransaction:<east-dht5-60816(CMC-Denver-CO)>:48002:remote, cacheName='eigAllocation', topologyId=26}
> 2013-09-17 14:28:34,116 TRACE [org.infinispan.commands.tx.AbstractTransactionBoundaryCommand] (OOB-153,session-resource-cluster,west-dht4-48045(CH2-Chicago-IL)) Did not find a RemoteTransaction for GlobalTransaction:<east-dht5-60816(CMC-Denver-CO)>:48002:remote
> 2013-09-17 14:28:34,116 TRACE [org.infinispan.remoting.InboundInvocationHandlerImpl] (OOB-153,session-resource-cluster,west-dht4-48045(CH2-Chicago-IL)) About to send back response SuccessfulResponse{responseValue=null} for command RollbackCommand {gtx=GlobalTransaction:<east-dht5-60816(CMC-Denver-CO)>:48002:remote, cacheName='eigAllocation', topologyId=26}
> ...
> 2013-09-17 14:28:34,161 TRACE [org.infinispan.transaction.TransactionTable] (OOB-408,session-resource-cluster,west-dht4-48045(CH2-Chicago-IL)) Created and registered remote transaction RemoteTransaction{modifications=[PutKeyValueCommand{key=EdgeResourceCacheKey[edgeDeviceId=4109,resourceId=16825], value=EdgeInputResource [edgeInputId=16825, currentBandwidth=0, maxBandwidth=1048576000], flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true, ignorePreviousValue=false}], lookedUpEntries={}, lockedKeys=null, backupKeyLocks=null, missingLookedUpEntries=false, isMarkedForRollback=false, tx=GlobalTransaction:<east-dht5-60816(CMC-Denver-CO)>:48002:remote}
> 2013-09-17 14:28:34,733 TRACE [org.infinispan.statetransfer.StateTransferLockImpl] (OOB-408,session-resource-cluster,west-dht4-48045(CH2-Chicago-IL)) Signalling transaction data received for topology 26
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-3289) PutForExternalRead should only write the value on the originator once
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3289?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-3289:
--------------------------------
Labels: nbst (was: )
> PutForExternalRead should only write the value on the originator once
> ---------------------------------------------------------------------
>
> Key: ISPN-3289
> URL: https://issues.jboss.org/browse/ISPN-3289
> Project: Infinispan
> Issue Type: Bug
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Labels: nbst
> Fix For: 6.0.0.CR1
>
>
> putForExternalRead behaves as an asynchronous, non-transactional write operation even in tx caches, and as such it uses lock delegation.
> Let's say we have a cluster with two nodes: [A, B], and a key k with owners(k) = [A, B].
> If B is the originator of a PFER(k, v) operation, the command is first executed locally on B, then it is invoked asynchronously on the primary owner A, and A invokes it asynchronously on B (because of the FORCE_ASNCHRONOUS flag).
> For regular non-tx write operations, the entry k=v is only written to the data container on B during the second invocation. For PFER, however, the entry is written twice: once during each execution.
> This causes random failures in {{PutForExternalRead*Test.testNoOpWhenKeyPresent}}, because the test assumes that the PFER finished once the entry was written once on each owner.
> Arguably the fact that the primary owner forwards the PFER command asynchronously is also a problem, because the put command will write the value on B without holding a lock on A, the primary owner.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-3281) Deadlock in non-transactional caches during rebalance
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3281?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-3281:
--------------------------------
Labels: nbst (was: )
> Deadlock in non-transactional caches during rebalance
> -----------------------------------------------------
>
> Key: ISPN-3281
> URL: https://issues.jboss.org/browse/ISPN-3281
> Project: Infinispan
> Issue Type: Bug
> Components: Locking and Concurrency, State transfer
> Affects Versions: 5.3.0.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: nbst
> Fix For: 6.0.0.Alpha1, 6.0.0.CR1
>
>
> Say we have a cache with node A and node B joins. The cache topology id is 1, primary_owner(k) = A in the current CH and primary_owner(k) = B in the pending CH.
> 1. Node A starts a put(k, v) command during the rebalance. It thinks it's the primary owner, so it acquires the lock locally and it forwards the command to B.
> 2. B installs topology 2, primary_owner(k) = B in the current CH, and there is no pending CH.
> 3. B receives the put(k, v) command from A. It thinks it's the primary owner, so it acquires the lock locally and it forwards the command to A.
> 4. A receives the put(k, v) command from B. Again it thinks it's the primary owner and tries to acquire the lock locally, but it times out because the lock is held by another thread (from step 1).
> I think it may be enough to update the topology id in the put(k, v) command on node B, before forwarding it back to A. That way, the command will block on node A until topology 2 is installed, and it won't try to lock the key again.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-3357) Insufficient owners with putIfAbsent during rebalance
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3357?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-3357:
--------------------------------
Labels: nbst (was: )
> Insufficient owners with putIfAbsent during rebalance
> -----------------------------------------------------
>
> Key: ISPN-3357
> URL: https://issues.jboss.org/browse/ISPN-3357
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.4.Final, 6.0.0.Alpha1
> Reporter: Takayoshi Kimura
> Assignee: Dan Berindei
> Priority: Critical
> Labels: nbst
> Fix For: 5.2.8.Final, 6.0.0.Alpha3
>
> Attachments: 7c29bccb.log, ISPN-3357-full-logs-leave.zip
>
>
> Here is test scenario:
> * DIST numOwners=2, start with 3 nodes cluster then join 1 node during load
> * HotRod putIfAbsent accesses from 40 threads (1 process, 1 remote cache instance), 40000 entries total
> After the test run, the numberOfEntries on each node are:
> * node1: 20074
> * node2: 19888
> * node3: 20114
> * node4: 18885
> Total is 78961, 1039 entries are missing. No error on HotRod client side so 80000 entries should be there.
> Let's take a look at example missing entry, hash(thread01key151) = 7c29bccb.
> Current CH: owners(7c29bccb) are [node1, node2]
> Pending CH: owners(7c29bccb) are [node1, node2, node4]
> Balanced CH: owners(7c29bccb) are [node1, node4]
> The events sequence is:
> * hotrod -> node1
> * node1 -> node2, node4
> * node2 committed entry
> * node4 performed clustered get before write, got a value from node2 and will not commit the entry because this node thinks it's not changed/created
> * node1 committed entry
> * node2 invalidates the entry because it's no longer an owner
> Result owners(7c29bccb) are only node1 and node4 is missing. This entry may be completely lost by further rebalances when node4 is donor for this segment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-3217) Rebalance doesn't store data into cache store
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3217?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-3217:
--------------------------------
Labels: nbst (was: )
> Rebalance doesn't store data into cache store
> ---------------------------------------------
>
> Key: ISPN-3217
> URL: https://issues.jboss.org/browse/ISPN-3217
> Project: Infinispan
> Issue Type: Bug
> Components: Core API
> Affects Versions: 5.2.4.Final, 5.2.5.Final, 5.2.6.Final, 5.3.0.CR1
> Reporter: Takayoshi Kimura
> Assignee: Dan Berindei
> Priority: Blocker
> Labels: nbst
> Fix For: 5.2.7.Final, 5.3.0.Final
>
> Attachments: ISPN-3217-logs.zip
>
>
> In DistCacheStoreInterceptor.skip():
> {noformat}
> private boolean skip(InvocationContext ctx, Object key, FlagAffectedCommand command) {
> return skip(ctx, command) || skipKey(key) || (isUsingLockDelegation && !cdl.localNodeIsPrimaryOwner(key) && (!cdl.localNodeIsOwner(key) || ctx.isOriginLocal()));
> }
> {noformat}
> The 3rd condition returns true on rebalance, so the data is not stored in the cache store.
> - The caller is org.infinispan.statetransfer.StateConsumerImpl.doApplyState
> - The iic is org.infinispan.context.SingleKeyNonTxInvocationContext
> - The example command is:
> {noformat}
> PutKeyValueCommand{key=ByteArrayKey{data=ByteArray{size=9, hashCode=cb62ce78, array=0x033e06666f6f3839..}}, value=CacheValue{data=ByteArray{size=6, array=0x033e03626172..}, version=4294968192}, flags=[CACHE_MODE_LOCAL, SKIP_REMOTE_LOOKUP, PUT_FOR_STATE_TRANSFER, SKIP_SHARED_CACHE_STORE, SKIP_OWNERSHIP_CHECK, IGNORE_RETURN_VALUES, SKIP_XSITE_BACKUP], putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true}
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-3140) JMX operation to suppress state transfer
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3140?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-3140:
--------------------------------
Labels: nbst (was: )
> JMX operation to suppress state transfer
> ----------------------------------------
>
> Key: ISPN-3140
> URL: https://issues.jboss.org/browse/ISPN-3140
> Project: Infinispan
> Issue Type: Feature Request
> Components: Distributed Cache, State transfer
> Affects Versions: 5.2.6.Final
> Reporter: Manik Surtani
> Assignee: Dan Berindei
> Labels: nbst
> Fix For: 5.2.7.Final, 5.3.0.CR2, 5.3.0.Final
>
>
> This feature request is to expose a JMX operation on each node, to suppress state transfer for a period of time. This flag would be {{false}} by default.
> The use case of this flag would be to ease bringing down (and up) a cluster for maintenance work. A typical workflow would be:
> 1) Shut down application requests to the data grid
> 2) Suppress state transfer on all nodes via JMX
> 3) Bring down all nodes
> 4) Perform maintenance work
> 5) Bring up nodes, one at a time. As each node comes up, disable state transfer for the node via JMX.
> 6) Once all nodes are up, enable state transfer for each node again via JMX
> 7) Allow application requests to reach the grid again.
> The purpose of this is to allow smooth and fast shutdown and startup, remove the risk of OOM errors (when bringing a grid down).
> This is a small but useful subset of full manual state transfer as defined in ISPN-1394.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-2966) NBST: Concurrent leavers can lead to deadlock
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2966?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2966:
--------------------------------
Labels: nbst state_transfer (was: state_transfer)
> NBST: Concurrent leavers can lead to deadlock
> ---------------------------------------------
>
> Key: ISPN-2966
> URL: https://issues.jboss.org/browse/ISPN-2966
> Project: Infinispan
> Issue Type: Bug
> Reporter: Pedro Ruivo
> Assignee: Dan Berindei
> Labels: nbst, state_transfer
> Fix For: 5.3.0.Final
>
> Attachments: thread-dump.txt, trace.log
>
>
> This sequence of events, leads to a thread deadlock in the coordinator
> {code}
> 1) NodeF sends LEAVE message. new topologyId=8
> 2) NodeE delivers REBALANCE_START(8)
> 3) NodeF and NodeG delivers REBALANCE_START(8)
> 4) NodeH delivers GET_TRANSACTION(8) from NodeE ==> Transactions were requested by node ConcurrentNonOverlappingLeaveTest-NodeE-28744 with topology 8, greater than the local topology (7). Waiting for topology 8 to be installed locally.
> 5) NodeH sends LEAVE message. new topologyId=9
> 6) NodeH delivers REBALANCE_START(8) ==> Ignoring rebalance 8 for cache dist that doesn't exist locally
> 7) NodeH delivers GET_TRANSACTION(8) from NodeG ==> Transactions were requested by node ConcurrentNonOverlappingLeaveTest-NodeG-31669 with topology 8, greater than the local topology (7). Waiting for topology 8 to be installed locally.
> {code}
> Possible solutions are:
> - send the REBALANCE_START/CH_UPDATE async
> - throw an exception when a GET_TRANSACTION is received and the node is shutting down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years