[JBoss JIRA] (ISPN-4137) Transaction executed multiple times due to forwarded CommitCommand
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-4137?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-4137:
------------------------------------
Sorry, your algorithm description got a little long and I couldn't follow it. I'm not sure how it would be better than just removing the timeout from the CommitCommand RPC and waiting for the response indefinitely, though (since you can't report success to the user unless everyone has committed).
Note that the new primary owner *must* have been an owner in the previous cache topology, so it must have received the transaction either directly from the originator or via state transfer. The only exception is when all the owners of the key leave, but we don't offer any guarantees in that case.
> Transaction executed multiple times due to forwarded CommitCommand
> ------------------------------------------------------------------
>
> Key: ISPN-4137
> URL: https://issues.jboss.org/browse/ISPN-4137
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer, Transactions
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
>
> When the {{StateTransferInterceptor}} forwards a CommitCommand for the new topology, multiple CommitCommands may be broadcast across the cluster. If the command (forwarded already from originator) times out, the transaction may be correctly finished by the first one and the application considers TX as succeeded (useSynchronizations=true), although one more Rollback is sent as well.
> Then, again in STI, when the CommitCommand arrives with higher topologyId than the one used for the first TX execution, another artificial Prepare (followed by the commit) is executed - see {{STI.visitCommitCommand}}.
> However, this execution may be delayed a lot and originator may have already executed another TX on the same entries. Then, this forwarded Commit will overwrite the already updated entries, causing inconsistency of data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-4162) Can DIST use multicast when num_owners >= cluster size?
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/ISPN-4162?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on ISPN-4162:
--------------------------------
Note that multicasts and unicasts are *independent wrt ordering*, e.g.
* The cluster is \{A,B,C,D,E\}
* A sends message M1 as multiple unicasts to B and C
* A sends message M2 as a multicast (to all)
* Unless both M1 and M2 are *synchronous*, the ordering of M1 and M2 on delivery is undefined. B might receive M1 and M2, whereas C might receive M2 and then M1.
> Can DIST use multicast when num_owners >= cluster size?
> -------------------------------------------------------
>
> Key: ISPN-4162
> URL: https://issues.jboss.org/browse/ISPN-4162
> Project: Infinispan
> Issue Type: Feature Request
> Components: Core
> Affects Versions: 7.0.0.Alpha1
> Reporter: Paul Ferraro
> Assignee: Dan Berindei
>
> It would be nice if DIST could conditionally use multicast instead of multiple unicasts when the number of owners to which to send a given command >= the size of the cluster. In fact, we could probably come up with some configurable ratio of the size of the cluster (perhaps as high as 50%?), above which it is more optimal to use multicast vs multiple unicasts. This would optimize the use of DIST in smaller, dynamic clusters.
> With such an optimization in place, REPL would be completely obsolete.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-2916) Group-based expiration
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/ISPN-2916?page=com.atlassian.jira.plugin.... ]
Radim Vansa commented on ISPN-2916:
-----------------------------------
As expiration is driven based on wall-clock time, you'd have to store all the keys on one node (this is doable through key affinity service). Regrettably, current implementation of DefaultTimeService uses System.currentTimeMillis() which is not monotonic -> when one entry is reported as expired, you can't be sure that another entry with the same expiration will be reported as expired as well.
The option using expiration listeners (ISPN-694) on primary key has another problem - even if you executed all the removes within transaction, transaction writes are *not* atomic with respect to reads (if {{k1=v1}} and {{k2=v2}} and you do {{T1: remove(k1), remove(k2)}} and {{T2: read(k1), read(k2)}} you can get {{read(k1) = A, read(k2) = null}}). Atomicity of reads could be achieved only with pessimistic transactions and using the FORCE_WRITE_LOCK flag for reads. Probably better solution would be to always check for the presence of primary key and serve the session according to that. The check would have to be executed only after all entries are read.
> Group-based expiration
> ----------------------
>
> Key: ISPN-2916
> URL: https://issues.jboss.org/browse/ISPN-2916
> Project: Infinispan
> Issue Type: Feature Request
> Components: Core
> Affects Versions: 5.2.5.Final
> Reporter: Paul Ferraro
> Assignee: Mircea Markus
>
> Now that WildFly represents a web session as a group of cache entries (instead of a single entry containing an atomic map), there are a few hurdles preventing us from leveraging infinispan-managed expiration.
> One of which is the fact that expiration of cache entries within a group *must* be atomic. It would be very bad if a request arrives for a session whose entries are in the process of being expired. The expiration thread should obtain a lock on *all* the keys for a group (or at least some pre-determined "primary" key), so that late session access doesn't result in session attributes mysteriously disappearing (because they were independently expired). This can also cause integrity constraint violations if cache entries reference other entries in the group. e.g. currently the "primary" cache entry for a session (keyed by session id), contains a set of attribute names. These names correspond to other cache entries that contain the value of a given attribute for that session.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-4162) Can DIST use multicast when num_owners >= cluster size?
by Paul Ferraro (JIRA)
Paul Ferraro created ISPN-4162:
----------------------------------
Summary: Can DIST use multicast when num_owners >= cluster size?
Key: ISPN-4162
URL: https://issues.jboss.org/browse/ISPN-4162
Project: Infinispan
Issue Type: Feature Request
Components: Core
Affects Versions: 7.0.0.Alpha1
Reporter: Paul Ferraro
Assignee: Dan Berindei
It would be nice if DIST could conditionally use multicast instead of multiple unicasts when the number of owners to which to send a given command >= the size of the cluster. In fact, we could probably come up with some configurable ratio of the size of the cluster (perhaps as high as 50%?), above which it is more optimal to use multicast vs multiple unicasts. This would optimize the use of DIST in smaller, dynamic clusters.
With such an optimization in place, REPL would be completely obsolete.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-4137) Transaction executed multiple times due to forwarded CommitCommand
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/ISPN-4137?page=com.atlassian.jira.plugin.... ]
Radim Vansa edited comment on ISPN-4137 at 3/26/14 11:12 AM:
-------------------------------------------------------------
{quote}
That's an interesting approach, but I don't see any way of implementing that right now - when we get a timeout from an RPC, we can't spawn another thread to wait for the "real" response. So we have to either wait forever, or set a timeout and somehow release the locks when a timeout occurs without causing more inconsistencies than necessary.
{quote}
My idea is that originator *must* send only the prepare command. Originator *can* send rollback if and only if it got negative ack from some of the nodes. It *can* sent commit if it got positive ack from all nodes.
An automatic reconciliation is executed periodically (on any node participating in the transaction), to find out what's the state of each ongoing transaction (after some random timeout in order to not create too much load in case that everything goes smoothly). If any of the nodes report that prepare failed, rollback is sent to all nodes. In other case, commit is sent to all non-committed nodes.
When a node is committed, it sends a confirmation to the originator. After calling prepare, the originator waits for all confirmations - if it gets them, TX is successful, otherwise an exception is thrown.
With this algorithm, the transaction result is determined by the success in prepare, and commit/rollback are a matter of the whole cluster, not originator - this is only notified.
Also, rollback is faster, as it is async.
If primary owner crashes during transaction, new primary owner should trigger the reconciliation. It has to find out whether there's any committed node in this transaction - if it is, it has to mark itself prepared (as such node could not be committed unless the previous owner got itself prepared as well). If there was no committed node, it gets complicated - should we get prepared or should we declare that prepare has failed for this node?
There is a race condition - some node could receive ack from the old primary owner and become committed just after it responded to the new primary that it's prepared. That way, we could get new node failed and old one committed. Therefore, any query with new topology has to invalidate all responses from lower topologies received after that - so it can't make a decision after being queried before sending and receiving synchronous query from all other nodes in the new topology (including this node, which has not responded yet to anyone).
The commit from originator having lower topology has to be ignored as well.
Then, as we know nobody would become committed before we answer them, we may mark ourselves failed. (The reason why we have to fail is that there may be another transaction on the same key, which has some nodes committed)
Regarding multiple new primaries, these will always query only the old nodes - all new primaries should fail unless there is some committed old node.
Another problem I see is when the prepare is delayed. Then, the reconciliation has to mark the tx prepare as failed on the node, therefore, rolling back the TX, and make sure that when the prepare arrives, it won't execute it. When the transaction is completed on one node, we can't forget it immediately. We have to keep the information until the transaction is committed in all nodes. This is where the TxCompletionNotification should be involved. The way to detect how long we have to keep the information about tx is a matter of ISPN-4131. Forwarding makes this a bit complicated.
Note that the reconciliation may be explicitly triggered by originator after timeout for ack (negative or positive) for the prepare, but it does not have to keep the thread working on the transaction waiting. (In fact regular commit is a blocking reconciliation as well).
Are there any other shortcomings?
{quote}
There's also a problem with reporting success before the transaction committed on all the owners. A subsequent get(k) on the same thread may return the value from the node that didn't commit put(k, v) yet, so the user would see an inconsistency.
{quote}
Definitely, you can't report success unless you got positive acks for everything.
{quote}
But I can't agree with you on that rule, the constant XAException.XA_HEURMIX is exactly for this kind of situation.
{quote}
The guys who designed transaction API are definitely smarter than me, but let's find out whether we could design it in a way to evade it (maybe it's not possible).
was (Author: rvansa):
{quote}
That's an interesting approach, but I don't see any way of implementing that right now - when we get a timeout from an RPC, we can't spawn another thread to wait for the "real" response. So we have to either wait forever, or set a timeout and somehow release the locks when a timeout occurs without causing more inconsistencies than necessary.
{quote}
My idea is that originator *must* send only the prepare command. Originator *can* send rollback if and only if it got negative ack from some of the nodes. It *can* sent commit if it got positive ack from all nodes.
An automatic reconciliation is executed periodically (on any node participating in the transaction), to find out what's the state of each ongoing transaction (after some random timeout in order to not create too much load in case that everything goes smoothly). If any of the nodes report that prepare failed, rollback is sent to all nodes. In other case, commit is sent to all non-committed nodes.
When a node is committed, it sends a confirmation to the originator. After calling prepare, the originator waits for all confirmations - if it gets them, TX is successful, otherwise an exception is thrown.
With this algorithm, the transaction result is determined by the success in prepare, and commit/rollback are a matter of the whole cluster, not originator - this is only notified.
Also, rollback is faster, as it is async.
If primary owner crashes during transaction, new primary owner should trigger the reconciliation. It has to find out whether there's any committed node in this transaction - if it is, it has to mark itself prepared (as such node could not be committed unless the previous owner got itself prepared as well). If there was no committed node, it gets complicated - should we get prepared or should we declare that prepare has failed for this node?
There is a race condition - some node could receive ack from the old primary owner and become committed just after it responded to the new primary that it's prepared. That way, we could get new node failed and old one committed. Therefore, any query with new topology has to invalidate all responses from lower topologies received after that - so it can't make a decision after being queried before sending and receiving synchronous query from all other nodes in the new topology (including this node, which has not responded yet to anyone). Then, as we know nobody would become committed before we answer them, we may mark ourselves failed. (The reason why we have to fail is that there may be another transaction on the same key, which has some nodes committed)
Regarding multiple new primaries, these will always query only the old nodes - all new primaries should fail unless there is some committed old node.
Another problem I see is when the prepare is delayed. Then, the reconciliation has to mark the tx prepare as failed on the node, therefore, rolling back the TX, and make sure that when the prepare arrives, it won't execute it. When the transaction is completed on one node, we can't forget it immediately. We have to keep the information until the transaction is committed in all nodes. This is where the TxCompletionNotification should be involved. The way to detect how long we have to keep the information about tx is a matter of ISPN-4131. Forwarding makes this a bit complicated.
Note that the reconciliation may be explicitly triggered by originator after timeout for ack (negative or positive) for the prepare, but it does not have to keep the thread working on the transaction waiting. (In fact regular commit is a blocking reconciliation as well).
Are there any other shortcomings?
{quote}
There's also a problem with reporting success before the transaction committed on all the owners. A subsequent get(k) on the same thread may return the value from the node that didn't commit put(k, v) yet, so the user would see an inconsistency.
{quote}
Definitely, you can't report success unless you got positive acks for everything.
{quote}
But I can't agree with you on that rule, the constant XAException.XA_HEURMIX is exactly for this kind of situation.
{quote}
The guys who designed transaction API are definitely smarter than me, but let's find out whether we could design it in a way to evade it (maybe it's not possible).
> Transaction executed multiple times due to forwarded CommitCommand
> ------------------------------------------------------------------
>
> Key: ISPN-4137
> URL: https://issues.jboss.org/browse/ISPN-4137
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer, Transactions
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
>
> When the {{StateTransferInterceptor}} forwards a CommitCommand for the new topology, multiple CommitCommands may be broadcast across the cluster. If the command (forwarded already from originator) times out, the transaction may be correctly finished by the first one and the application considers TX as succeeded (useSynchronizations=true), although one more Rollback is sent as well.
> Then, again in STI, when the CommitCommand arrives with higher topologyId than the one used for the first TX execution, another artificial Prepare (followed by the commit) is executed - see {{STI.visitCommitCommand}}.
> However, this execution may be delayed a lot and originator may have already executed another TX on the same entries. Then, this forwarded Commit will overwrite the already updated entries, causing inconsistency of data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-4137) Transaction executed multiple times due to forwarded CommitCommand
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/ISPN-4137?page=com.atlassian.jira.plugin.... ]
Radim Vansa commented on ISPN-4137:
-----------------------------------
{quote}
That's an interesting approach, but I don't see any way of implementing that right now - when we get a timeout from an RPC, we can't spawn another thread to wait for the "real" response. So we have to either wait forever, or set a timeout and somehow release the locks when a timeout occurs without causing more inconsistencies than necessary.
{quote}
My idea is that originator *must* send only the prepare command. Originator *can* send rollback if and only if it got negative ack from some of the nodes. It *can* sent commit if it got positive ack from all nodes.
An automatic reconciliation is executed periodically (on any node participating in the transaction), to find out what's the state of each ongoing transaction (after some random timeout in order to not create too much load in case that everything goes smoothly). If any of the nodes report that prepare failed, rollback is sent to all nodes. In other case, commit is sent to all non-committed nodes.
When a node is committed, it sends a confirmation to the originator. After calling prepare, the originator waits for all confirmations - if it gets them, TX is successful, otherwise an exception is thrown.
With this algorithm, the transaction result is determined by the success in prepare, and commit/rollback are a matter of the whole cluster, not originator - this is only notified.
Also, rollback is faster, as it is async.
If primary owner crashes during transaction, new primary owner should trigger the reconciliation. It has to find out whether there's any committed node in this transaction - if it is, it has to mark itself prepared (as such node could not be committed unless the previous owner got itself prepared as well). If there was no committed node, it gets complicated - should we get prepared or should we declare that prepare has failed for this node?
There is a race condition - some node could receive ack from the old primary owner and become committed just after it responded to the new primary that it's prepared. That way, we could get new node failed and old one committed. Therefore, any query with new topology has to invalidate all responses from lower topologies received after that - so it can't make a decision after being queried before sending and receiving synchronous query from all other nodes in the new topology (including this node, which has not responded yet to anyone). Then, as we know nobody would become committed before we answer them, we may mark ourselves failed. (The reason why we have to fail is that there may be another transaction on the same key, which has some nodes committed)
Regarding multiple new primaries, these will always query only the old nodes - all new primaries should fail unless there is some committed old node.
Another problem I see is when the prepare is delayed. Then, the reconciliation has to mark the tx prepare as failed on the node, therefore, rolling back the TX, and make sure that when the prepare arrives, it won't execute it. When the transaction is completed on one node, we can't forget it immediately. We have to keep the information until the transaction is committed in all nodes. This is where the TxCompletionNotification should be involved. The way to detect how long we have to keep the information about tx is a matter of ISPN-4131. Forwarding makes this a bit complicated.
Note that the reconciliation may be explicitly triggered by originator after timeout for ack (negative or positive) for the prepare, but it does not have to keep the thread working on the transaction waiting. (In fact regular commit is a blocking reconciliation as well).
Are there any other shortcomings?
{quote}
There's also a problem with reporting success before the transaction committed on all the owners. A subsequent get(k) on the same thread may return the value from the node that didn't commit put(k, v) yet, so the user would see an inconsistency.
{quote}
Definitely, you can't report success unless you got positive acks for everything.
{quote}
But I can't agree with you on that rule, the constant XAException.XA_HEURMIX is exactly for this kind of situation.
{quote}
The guys who designed transaction API are definitely smarter than me, but let's find out whether we could design it in a way to evade it (maybe it's not possible).
> Transaction executed multiple times due to forwarded CommitCommand
> ------------------------------------------------------------------
>
> Key: ISPN-4137
> URL: https://issues.jboss.org/browse/ISPN-4137
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer, Transactions
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
>
> When the {{StateTransferInterceptor}} forwards a CommitCommand for the new topology, multiple CommitCommands may be broadcast across the cluster. If the command (forwarded already from originator) times out, the transaction may be correctly finished by the first one and the application considers TX as succeeded (useSynchronizations=true), although one more Rollback is sent as well.
> Then, again in STI, when the CommitCommand arrives with higher topologyId than the one used for the first TX execution, another artificial Prepare (followed by the commit) is executed - see {{STI.visitCommitCommand}}.
> However, this execution may be delayed a lot and originator may have already executed another TX on the same entries. Then, this forwarded Commit will overwrite the already updated entries, causing inconsistency of data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-4161) Remove cache command for caches not created on all nodes fails with CacheException and an NPE root cause
by Vladimir Blagojevic (JIRA)
Vladimir Blagojevic created ISPN-4161:
-----------------------------------------
Summary: Remove cache command for caches not created on all nodes fails with CacheException and an NPE root cause
Key: ISPN-4161
URL: https://issues.jboss.org/browse/ISPN-4161
Project: Infinispan
Issue Type: Bug
Reporter: Vladimir Blagojevic
Assignee: Dan Berindei
While testing creation of intermediate caches for M/R tasks I noticed that remove command fails with CacheException and NPE if the cache has not been created on all nodes.
The culprit is in RemoteCommandsFactory where the following line of code:
registry.getNamedComponentRegistry(cacheName).getComponent(PersistenceManager.class)
assumes that component with cacheName has been registered.
org.infinispan.commons.CacheException: Error removing cache
at org.infinispan.manager.DefaultCacheManager.removeCache(DefaultCacheManager.java:477)
at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:421)
at org.infinispan.distexec.mapreduce.DistributedIntermediateCacheFourNodesMapReduceTest.testIntermediateCacheNotCreatedOnAllNodes(DistributedIntermediateCacheFourNodesMapReduceTest.java:66)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
at org.testng.TestRunner.privateRun(TestRunner.java:767)
at org.testng.TestRunner.run(TestRunner.java:617)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from DistributedIntermediateCacheFourNodesMapReduceTest-NodeD-24545, see cause for remote stack trace
at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:41)
at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:66)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:548)
at org.infinispan.manager.DefaultCacheManager.removeCache(DefaultCacheManager.java:469)
... 22 more
Caused by: org.infinispan.commons.CacheException: Problems invoking command.
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:219)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:457)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:374)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:668)
at org.jgroups.JChannel.up(JChannel.java:711)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
at org.jgroups.protocols.RSVP.up(RSVP.java:187)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:370)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:381)
at org.jgroups.protocols.tom.TOA.up(TOA.java:121)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1040)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:404)
at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:774)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:570)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:296)
at org.jgroups.protocols.Discovery.up(Discovery.java:379)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1502)
at org.jgroups.protocols.TP$MyHandler.run(TP.java:1719)
... 3 more
Caused by: java.lang.NullPointerException
at org.infinispan.commands.RemoteCommandsFactory.fromStream(RemoteCommandsFactory.java:198)
at org.infinispan.marshall.exts.ReplicableCommandExternalizer.fromStream(ReplicableCommandExternalizer.java:106)
at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:151)
at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:62)
at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:404)
at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:212)
at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:152)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-4160) Do not allow the state transfer chunk size to be <= 0
by Dan Berindei (JIRA)
Dan Berindei created ISPN-4160:
----------------------------------
Summary: Do not allow the state transfer chunk size to be <= 0
Key: ISPN-4160
URL: https://issues.jboss.org/browse/ISPN-4160
Project: Infinispan
Issue Type: Task
Components: Configuration, State Transfer
Affects Versions: 7.0.0.Alpha1
Reporter: Dan Berindei
Assignee: Vladimir Blagojevic
Fix For: 7.0.0.Alpha3
The state transfer chunk size is used in other places as well, and the case of {{chunkSize <= 0}} has to be handled everywhere independently. StateTransferConfigurationBuilder should validate that {{chunkSize > 0}}, so that users of the chunk size setting don't need a default value.
The users will still be able to force state transfer to transfer everything at once with {{chunkSize == Integer.MAX_VALUE}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-4137) Transaction executed multiple times due to forwarded CommitCommand
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-4137?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-4137:
------------------------------------
{quote}
I agree that the Rollback should not be sent after timeout in commit. But I don't understand what would we gain if the lock release was in TxCompletion - the Rollback might then release the few resources. For further writes, the lock is held all the time - no difference - and reads without force write lock can return current or future value anyway (even in correct situations the transaction commit is not atomic with respect to reads). It could be even worse - if you allowed to rollback after commit, you could read uncommitted value for a while.
Maybe an example sequence commands would help me understanding.
{quote}
What we gain is that we can release the locks with the TxCompleteNotificationCommand, knowing that the backup won't try to write anything after the primary released the lock. The rollback command won't release any locks, but it will mark the transaction as completed and it will prevent the commit command from writing anything.
{quote}
I am not suggesting that - if Prepare was successful, once we sent the Commit, we have to kick all owners to commit the transaction, eventually. Only in the case that all nodes are found prepared and originator is dead, we may rollback the commit.
If primary keeps alive, it should synchronize further transactions by holding the lock (and releasing with commit) - backups don't matter, they will either commit the new value as well (the lock is held on primary), or the one change won't be spotted. If the primary dies, new primary should acquire the lock and not release it until it receives the commit command, synchronizing the writes again.
{quote}
The primary has to receive a TxCompletedNotificationCommand in order to release the locks. Who do you suggest should send the TxCompletionNotificationCommand, if not the originator, after it received the response from all the owners?
Off-topic, if the primary dies, the new primary can't acquire any real locks - there may be more than one prepared tx writing to the same key.
{quote}
There should be no reason for a prepared transaction to fail. Network failures should be fixed by JGroups, eventually. Timeouts should release application threads, potentially some worker threads to do another work, but these should not change the logical output of the operation.
{quote}
That's an interesting approach, but I don't see any way of implementing that right now - when we get a timeout from an RPC, we can't spawn another thread to wait for the "real" response. So we have to either wait forever, or set a timeout and somehow release the locks when a timeout occurs without causing more inconsistencies than necessary.
There's also a problem with reporting success before the transaction committed on all the owners. A subsequent get(k) on the same thread may return the value from the node that didn't commit put(k, v) yet, so the user would see an inconsistency.
{quote}
Bugs in code may happen, but user should find out that there's something wrong (locking repeatedly fails due to stale lock) - that's where the human intervention through recovery could be useful. For handling bugs in production.
{quote}
The user is already notified if the commit fails on one of the nodes - he gets an heuristic exception from {{commit()}} (unless using synchronization, but that's a separate issue).
Keeping the locks may be better for some (or most) of the users. But considering how long it may take an administrator to notice the in-doubt transaction, I don't see it as clearly better than the option of releasing the locks and re-acquiring them when the administrator force-commits the tx.
{quote}
So this is not correct either - some nodes may have already committed it. We can't rollback unless we're sure that nobody committed, that should be a firm rule when designing the system.
{quote}
Yeah, it's not correct, that's why we have a bug for it - ISPN-3421.
But I can't agree with you on that rule, the constant {{XAException.XA_HEURMIX}} is exactly for this kind of situation.
> Transaction executed multiple times due to forwarded CommitCommand
> ------------------------------------------------------------------
>
> Key: ISPN-4137
> URL: https://issues.jboss.org/browse/ISPN-4137
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer, Transactions
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
>
> When the {{StateTransferInterceptor}} forwards a CommitCommand for the new topology, multiple CommitCommands may be broadcast across the cluster. If the command (forwarded already from originator) times out, the transaction may be correctly finished by the first one and the application considers TX as succeeded (useSynchronizations=true), although one more Rollback is sent as well.
> Then, again in STI, when the CommitCommand arrives with higher topologyId than the one used for the first TX execution, another artificial Prepare (followed by the commit) is executed - see {{STI.visitCommitCommand}}.
> However, this execution may be delayed a lot and originator may have already executed another TX on the same entries. Then, this forwarded Commit will overwrite the already updated entries, causing inconsistency of data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-4159) DefaultTwoWayKey2StringMapper encodes objects to strings in a manner that is incompatible with string handling of some databases
by Adrian Nistor (JIRA)
Adrian Nistor created ISPN-4159:
-----------------------------------
Summary: DefaultTwoWayKey2StringMapper encodes objects to strings in a manner that is incompatible with string handling of some databases
Key: ISPN-4159
URL: https://issues.jboss.org/browse/ISPN-4159
Project: Infinispan
Issue Type: Bug
Components: Loaders and Stores
Affects Versions: 6.0.0.Final
Reporter: Adrian Nistor
Assignee: Adrian Nistor
Fix For: 7.0.0.Alpha3, 7.0.0.Final
DefaultTwoWayKey2StringMapper uses two neat tricks.
1. it does not encode all supported types, it only encodes non-Strings. Strings are kept unmodified.
2. it uses a special prefix (unicode char 0xfeff) to mark which strings were encoded and which are plain.
Unfortunately some databases, notably MySql, interpret the endianness mark (0xfeff, 0xfffe), convert to native byte order and then drop it.
This leaves us with no clue the string is not an actual String but an encoded representation of another type. This misinterpretation leads later to ClassCastExceptions in various places in core and user code.
Proposed fix: get rid of #1 and #2 optimisations. Encode all objects, including Strings and always use the ?n prefix (where n stands for the original type). Drop the 0xFEFF marker prefix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years