[JBoss JIRA] (ISPN-4137) Transaction executed multiple times due to forwarded CommitCommand
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/ISPN-4137?page=com.atlassian.jira.plugin.... ]
Radim Vansa commented on ISPN-4137:
-----------------------------------
{quote}
Without recovery, we will still run the rollback to make sure the locks are released. I don't have any suggestions on how to improve that, except maybe retrying the commit indefinitely.
{quote}
I don't see the point in committing indefinitely. There's now way how the CommitCommand can be lost - JGroups should deliver it reliably, and we are not dropping delivered commands anywhere in Infinispan (or do we?). It can just take a while before it is delivered and responded. The only consideration for any resending is sending to new nodes.
What's the contract, anyway? When the commit() throws exception, are there any guarantees that none of the operations were written? *Is this described anywhere?*
If there are no such guarantees, trying to finish the TX with commit even if exception was reported on originator is IMO better than send a rollback (and hope things will settle) or keeping the locks stale. If there are any such guarantees, we can't do anything, and we should rather keep the lock stale (blocking further txs) than break the contract. Thinking about it again, there can't be any guarantees because the commit can be already executed - the contract would be broken.
{quote}
If the primary owner crashes, the txs on the backup owners still have "backup locks". Each prepare on the new primary owner will check in the entire tx table for backup locks, and will block until those locks are released. The real problem is when the originator dies...
{quote}
When the originator dies after prepare, the transaction keeps hanging anyway. Is it then reported in-doubt in recovery?
> Transaction executed multiple times due to forwarded CommitCommand
> ------------------------------------------------------------------
>
> Key: ISPN-4137
> URL: https://issues.jboss.org/browse/ISPN-4137
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer, Transactions
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
>
> When the {{StateTransferInterceptor}} forwards a CommitCommand for the new topology, multiple CommitCommands may be broadcast across the cluster. If the command (forwarded already from originator) times out, the transaction may be correctly finished by the first one and the application considers TX as succeeded (useSynchronizations=true), although one more Rollback is sent as well.
> Then, again in STI, when the CommitCommand arrives with higher topologyId than the one used for the first TX execution, another artificial Prepare (followed by the commit) is executed - see {{STI.visitCommitCommand}}.
> However, this execution may be delayed a lot and originator may have already executed another TX on the same entries. Then, this forwarded Commit will overwrite the already updated entries, causing inconsistency of data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-4155) hotrod client osgi bundle - add repository name
by Ion Savin (JIRA)
Ion Savin created ISPN-4155:
-------------------------------
Summary: hotrod client osgi bundle - add repository name
Key: ISPN-4155
URL: https://issues.jboss.org/browse/ISPN-4155
Project: Infinispan
Issue Type: Feature Request
Reporter: Ion Savin
Assignee: Ion Savin
The repository name is empty.
{code}
karaf@root()> repo-list
Repository | URL
-------------------------------------------------------------------------------------------------
camel-2.10.0 | mvn:org.apache.camel.karaf/apache-camel/2.10.0/xml/features
standard-3.0.0 | mvn:org.apache.karaf.features/standard/3.0.0/xml/features
jclouds-1.4.0 | mvn:org.jclouds.karaf/jclouds-karaf/1.4.0/xml/features
cxf-2.6.1 | mvn:org.apache.cxf.karaf/apache-cxf/2.6.1/xml/features
org.ops4j.pax.web-3.0.5 | mvn:org.ops4j.pax.web/pax-web-features/3.0.5/xml/features
| mvn:org.infinispan/infinispan-client-hotrod/7.0.0-SNAPSHOT/xml/features
enterprise-3.0.0 | mvn:org.apache.karaf.features/enterprise/3.0.0/xml/features
spring-3.0.0 | mvn:org.apache.karaf.features/spring/3.0.0/xml/features
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-4137) Transaction executed multiple times due to forwarded CommitCommand
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/ISPN-4137?page=com.atlassian.jira.plugin.... ]
Radim Vansa edited comment on ISPN-4137 at 3/25/14 10:05 AM:
-------------------------------------------------------------
Mea culpa, the ST data are ignore because the entry was invalidated by ST. ISPN-3633 is not related. I've filed ISPN-4154 for the issue.
was (Author: rvansa):
Mea culpa, the ST data are ignore because the entry was invalidated by ST. ISPN-3633 is not related.
> Transaction executed multiple times due to forwarded CommitCommand
> ------------------------------------------------------------------
>
> Key: ISPN-4137
> URL: https://issues.jboss.org/browse/ISPN-4137
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer, Transactions
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
>
> When the {{StateTransferInterceptor}} forwards a CommitCommand for the new topology, multiple CommitCommands may be broadcast across the cluster. If the command (forwarded already from originator) times out, the transaction may be correctly finished by the first one and the application considers TX as succeeded (useSynchronizations=true), although one more Rollback is sent as well.
> Then, again in STI, when the CommitCommand arrives with higher topologyId than the one used for the first TX execution, another artificial Prepare (followed by the commit) is executed - see {{STI.visitCommitCommand}}.
> However, this execution may be delayed a lot and originator may have already executed another TX on the same entries. Then, this forwarded Commit will overwrite the already updated entries, causing inconsistency of data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-4137) Transaction executed multiple times due to forwarded CommitCommand
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-4137?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-4137:
------------------------------------
Indeed [~rvansa], waiting for a human intervention doesn't sound very good. Before starting the work on this I was convinced that's how recovery is supposed to work, now I'm not so sure...
Without recovery, we will still run the rollback to make sure the locks are released. I don't have any suggestions on how to improve that, except maybe retrying the commit indefinitely.
If the primary owner crashes, the txs on the backup owners still have "backup locks". Each prepare on the new primary owner will check in the entire tx table for backup locks, and will block until those locks are released. The real problem is when the originator dies...
> Transaction executed multiple times due to forwarded CommitCommand
> ------------------------------------------------------------------
>
> Key: ISPN-4137
> URL: https://issues.jboss.org/browse/ISPN-4137
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer, Transactions
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
>
> When the {{StateTransferInterceptor}} forwards a CommitCommand for the new topology, multiple CommitCommands may be broadcast across the cluster. If the command (forwarded already from originator) times out, the transaction may be correctly finished by the first one and the application considers TX as succeeded (useSynchronizations=true), although one more Rollback is sent as well.
> Then, again in STI, when the CommitCommand arrives with higher topologyId than the one used for the first TX execution, another artificial Prepare (followed by the commit) is executed - see {{STI.visitCommitCommand}}.
> However, this execution may be delayed a lot and originator may have already executed another TX on the same entries. Then, this forwarded Commit will overwrite the already updated entries, causing inconsistency of data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-4154) Cancelled segment transfer causes future entry transfer to be ignored
by Radim Vansa (JIRA)
Radim Vansa created ISPN-4154:
---------------------------------
Summary: Cancelled segment transfer causes future entry transfer to be ignored
Key: ISPN-4154
URL: https://issues.jboss.org/browse/ISPN-4154
Project: Infinispan
Issue Type: Bug
Components: State Transfer
Affects Versions: 7.0.0.Alpha1
Reporter: Radim Vansa
Assignee: Dan Berindei
Priority: Critical
Distributed transactional cache.
1) Coordinator is gracefully leaving the cluster, sends a REBALANCE_START with topologyId 14, ST begins.
2) Node receives chunk from segment X, writes entry K=V to the container.
3) New coordinator jumps in with CH_UPDATE topology 16
4) Node receives CANCEL_STATE_TRANSFER and cancels transfer of segment X, invalidating K. In CommitManager, this operation is tracked and DiscardPolicy is set to DISCARD_STATE_TRANSFER for key K.
5) New coordinator starts rebalance with topology 17
6) Node starts new ST for segment X
7) Node receives the X: K=V, but in CommitManager it finds out that the policy is set to DISCARD_STATE_TRANSFER and ignores this update.
Result: entry value is lost on some node.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-4154) Cancelled segment transfer causes future entry transfer to be ignored
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/ISPN-4154?page=com.atlassian.jira.plugin.... ]
Radim Vansa commented on ISPN-4154:
-----------------------------------
>From IRC with [~dan.berindei]:
(02:17:26 PM) dberindei: rvansa: I think there may be a problem with StateConsumerImpl.onTopologyUpdate(), it sets waitingForState to false too soon to avoid sending an unnecessary REBALANCE_CONFIRM to the coordinator, but it also prevents the call to stopApplyingState()
(02:18:08 PM) dberindei: rvansa: normally that's not a problem because the state transfer tasks have already finished, and the last task to finish called stopApplyingState()
(02:18:20 PM) dberindei: rvansa: but when the coordinator leaves it's a bit different
> Cancelled segment transfer causes future entry transfer to be ignored
> ---------------------------------------------------------------------
>
> Key: ISPN-4154
> URL: https://issues.jboss.org/browse/ISPN-4154
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 7.0.0.Alpha1
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
>
> Distributed transactional cache.
> 1) Coordinator is gracefully leaving the cluster, sends a REBALANCE_START with topologyId 14, ST begins.
> 2) Node receives chunk from segment X, writes entry K=V to the container.
> 3) New coordinator jumps in with CH_UPDATE topology 16
> 4) Node receives CANCEL_STATE_TRANSFER and cancels transfer of segment X, invalidating K. In CommitManager, this operation is tracked and DiscardPolicy is set to DISCARD_STATE_TRANSFER for key K.
> 5) New coordinator starts rebalance with topology 17
> 6) Node starts new ST for segment X
> 7) Node receives the X: K=V, but in CommitManager it finds out that the policy is set to DISCARD_STATE_TRANSFER and ignores this update.
> Result: entry value is lost on some node.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-4137) Transaction executed multiple times due to forwarded CommitCommand
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/ISPN-4137?page=com.atlassian.jira.plugin.... ]
Radim Vansa commented on ISPN-4137:
-----------------------------------
Mea culpa, the ST data are ignore because the entry was invalidated by ST. ISPN-3633 is not related.
> Transaction executed multiple times due to forwarded CommitCommand
> ------------------------------------------------------------------
>
> Key: ISPN-4137
> URL: https://issues.jboss.org/browse/ISPN-4137
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer, Transactions
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
>
> When the {{StateTransferInterceptor}} forwards a CommitCommand for the new topology, multiple CommitCommands may be broadcast across the cluster. If the command (forwarded already from originator) times out, the transaction may be correctly finished by the first one and the application considers TX as succeeded (useSynchronizations=true), although one more Rollback is sent as well.
> Then, again in STI, when the CommitCommand arrives with higher topologyId than the one used for the first TX execution, another artificial Prepare (followed by the commit) is executed - see {{STI.visitCommitCommand}}.
> However, this execution may be delayed a lot and originator may have already executed another TX on the same entries. Then, this forwarded Commit will overwrite the already updated entries, causing inconsistency of data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-4137) Transaction executed multiple times due to forwarded CommitCommand
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/ISPN-4137?page=com.atlassian.jira.plugin.... ]
Radim Vansa commented on ISPN-4137:
-----------------------------------
[~dan.berindei] Running your branch, I've spotted something that looks like ISPN-3633 - this may be related to CommitManager introduced in ISPN-4024. However, there should be a test for ISPN-3633, I am not sure whether it is working correctly now. [~william.burns], was the test for ISPN-3633 reliable (failing every time before applying the fix)?
> Transaction executed multiple times due to forwarded CommitCommand
> ------------------------------------------------------------------
>
> Key: ISPN-4137
> URL: https://issues.jboss.org/browse/ISPN-4137
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer, Transactions
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
>
> When the {{StateTransferInterceptor}} forwards a CommitCommand for the new topology, multiple CommitCommands may be broadcast across the cluster. If the command (forwarded already from originator) times out, the transaction may be correctly finished by the first one and the application considers TX as succeeded (useSynchronizations=true), although one more Rollback is sent as well.
> Then, again in STI, when the CommitCommand arrives with higher topologyId than the one used for the first TX execution, another artificial Prepare (followed by the commit) is executed - see {{STI.visitCommitCommand}}.
> However, this execution may be delayed a lot and originator may have already executed another TX on the same entries. Then, this forwarded Commit will overwrite the already updated entries, causing inconsistency of data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-4153) Server can't start a cache with custom cache store
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-4153?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration updated ISPN-4153:
------------------------------------------
Bugzilla Update: Perform
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=1080419
> Server can't start a cache with custom cache store
> --------------------------------------------------
>
> Key: ISPN-4153
> URL: https://issues.jboss.org/browse/ISPN-4153
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Reporter: Jakub Markos
> Assignee: William Burns
>
> This test https://github.com/infinispan/infinispan/blob/master/server/integration/t... fails with current snapshot on server startup:
> {code}
> 12:57:25,277 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-3) MSC000001: Failed to start service jboss.infinispan.local.default.config: org.jboss.msc.service.StartException in service jboss.infinispan.local.default.config: Failed to start service
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1767) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1156) [rt.jar:1.7.0]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:626) [rt.jar:1.7.0]
> at java.lang.Thread.run(Thread.java:780) [vm.jar:1.7.0]
> Caused by: org.infinispan.commons.CacheConfigurationException: Missing BuiltBy annotation for configuration bean org.infinispan.configuration.cache.AbstractStoreConfiguration
> at org.infinispan.commons.configuration.ConfigurationUtils.builderFor(ConfigurationUtils.java:20)
> at org.infinispan.configuration.cache.PersistenceConfigurationBuilder.read(PersistenceConfigurationBuilder.java:124)
> at org.infinispan.configuration.cache.ConfigurationBuilder.read(ConfigurationBuilder.java:218)
> at org.infinispan.manager.DefaultCacheManager.defineConfiguration(DefaultCacheManager.java:356) at org.infinispan.manager.DefaultCacheManager.defineConfiguration(DefaultCacheManager.java:322) at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.defineConfiguration(DefaultEmbeddedCacheManager.java:62)
> at org.jboss.as.clustering.infinispan.subsystem.AbstractCacheConfigurationService.start(AbstractCacheConfigurationService.java:81)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
> ... 3 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years
[JBoss JIRA] (ISPN-4153) Server can't start a cache with custom cache store
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-4153?page=com.atlassian.jira.plugin.... ]
William Burns reassigned ISPN-4153:
-----------------------------------
Assignee: William Burns (was: Mircea Markus)
> Server can't start a cache with custom cache store
> --------------------------------------------------
>
> Key: ISPN-4153
> URL: https://issues.jboss.org/browse/ISPN-4153
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Reporter: Jakub Markos
> Assignee: William Burns
>
> This test https://github.com/infinispan/infinispan/blob/master/server/integration/t... fails with current snapshot on server startup:
> {code}
> 12:57:25,277 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-3) MSC000001: Failed to start service jboss.infinispan.local.default.config: org.jboss.msc.service.StartException in service jboss.infinispan.local.default.config: Failed to start service
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1767) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1156) [rt.jar:1.7.0]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:626) [rt.jar:1.7.0]
> at java.lang.Thread.run(Thread.java:780) [vm.jar:1.7.0]
> Caused by: org.infinispan.commons.CacheConfigurationException: Missing BuiltBy annotation for configuration bean org.infinispan.configuration.cache.AbstractStoreConfiguration
> at org.infinispan.commons.configuration.ConfigurationUtils.builderFor(ConfigurationUtils.java:20)
> at org.infinispan.configuration.cache.PersistenceConfigurationBuilder.read(PersistenceConfigurationBuilder.java:124)
> at org.infinispan.configuration.cache.ConfigurationBuilder.read(ConfigurationBuilder.java:218)
> at org.infinispan.manager.DefaultCacheManager.defineConfiguration(DefaultCacheManager.java:356) at org.infinispan.manager.DefaultCacheManager.defineConfiguration(DefaultCacheManager.java:322) at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.defineConfiguration(DefaultEmbeddedCacheManager.java:62)
> at org.jboss.as.clustering.infinispan.subsystem.AbstractCacheConfigurationService.start(AbstractCacheConfigurationService.java:81)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
> ... 3 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years