[JBoss JIRA] (ISPN-8232) Transaction inconsistency during network partitions
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-8232?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo commented on ISPN-8232:
-----------------------------------
the stable topology is updated in the majority partition. That is why the TransactionTable rollbacks the transaction.
> Transaction inconsistency during network partitions
> ---------------------------------------------------
>
> Key: ISPN-8232
> URL: https://issues.jboss.org/browse/ISPN-8232
> Project: Infinispan
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 9.1.0.Final
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
> Priority: Critical
>
> In scenario where the originator stays in minor partition (in our test suite, the originator isolated tests), it is possible to a transaction to be committed and rolled back in the majority partition.
> In {{Pessimitic Locking}}, the transaction is committed in one-phase using the {{PrepareCommand}}. If the partition happens when the originator sends the {{PrepareCommand}}, the nodes in the majority partition may or may not receive it. We can have the case where some nodes receive the {{PrepareCommand}} and applied and other don't receive it.
> When the topology is updated in the majority partition, the {{TransactionTable}} rollbacks all transaction in which the originator isn't present. So, in the nodes where the {{PrepareCommand}} isn't received, the transaction is rolled back.
> The originator in the minory partition detects the partition and marks the transaction partially completed. When the merge occurs, it tries to commit the transaction again. In the nodes where the transaction is rolled back, the transaction is marked as completed and when the {{PrepareCommand}} is received, it throws an {{IllegalStateException}} ({{TransactionTable:386, getOrCreateRemoteTransaction()}}). In this case, the transaction isn't removed from the {{PartitionHandlingManager}} and our test suite fails with {{"there are pending tx".}}
> Other theoretically scenario is the {{PrepareCommand}} to be executed when no locks are acquired.
> The same issue can happen with {{Optimistic Locking}} for the {{CommitCommand}}.
> The problem is the transaction table can't identify is the node left gracefully or not. A solution would be to have an {{"expected members"}} list, ideally separated from the {{CacheTopology}} to avoid sending it every time. Also, it would need some sysadmin tools for the case where the node crashes and it won't be back online for a while (or for some reason, it doesn't need to be back online).
> A sysadmin could remove the node from this list ({{CacheTopology}} is updated and there is no need to increase it) and decide what to do with the pending transactions (or an automatic mechanism to auto-commit/rollback the transaction).
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
6 years, 7 months
[JBoss JIRA] (ISPN-8232) Transaction inconsistency during network partitions
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-8232?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-8232:
------------------------------------
This is exactly what the stable topology is meant for. Unless you mean that the "expected members" list can only be updated manually by the admin, but that's not an option because the cluster would become unavailable every time a node crashes.
Actually I don't think this has anything to do with partitioning: the originator crashing during 1PC-Prepare/2PC-Commit would have exactly the same effect.
> Transaction inconsistency during network partitions
> ---------------------------------------------------
>
> Key: ISPN-8232
> URL: https://issues.jboss.org/browse/ISPN-8232
> Project: Infinispan
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 9.1.0.Final
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
> Priority: Critical
>
> In scenario where the originator stays in minor partition (in our test suite, the originator isolated tests), it is possible to a transaction to be committed and rolled back in the majority partition.
> In {{Pessimitic Locking}}, the transaction is committed in one-phase using the {{PrepareCommand}}. If the partition happens when the originator sends the {{PrepareCommand}}, the nodes in the majority partition may or may not receive it. We can have the case where some nodes receive the {{PrepareCommand}} and applied and other don't receive it.
> When the topology is updated in the majority partition, the {{TransactionTable}} rollbacks all transaction in which the originator isn't present. So, in the nodes where the {{PrepareCommand}} isn't received, the transaction is rolled back.
> The originator in the minory partition detects the partition and marks the transaction partially completed. When the merge occurs, it tries to commit the transaction again. In the nodes where the transaction is rolled back, the transaction is marked as completed and when the {{PrepareCommand}} is received, it throws an {{IllegalStateException}} ({{TransactionTable:386, getOrCreateRemoteTransaction()}}). In this case, the transaction isn't removed from the {{PartitionHandlingManager}} and our test suite fails with {{"there are pending tx".}}
> Other theoretically scenario is the {{PrepareCommand}} to be executed when no locks are acquired.
> The same issue can happen with {{Optimistic Locking}} for the {{CommitCommand}}.
> The problem is the transaction table can't identify is the node left gracefully or not. A solution would be to have an {{"expected members"}} list, ideally separated from the {{CacheTopology}} to avoid sending it every time. Also, it would need some sysadmin tools for the case where the node crashes and it won't be back online for a while (or for some reason, it doesn't need to be back online).
> A sysadmin could remove the node from this list ({{CacheTopology}} is updated and there is no need to increase it) and decide what to do with the pending transactions (or an automatic mechanism to auto-commit/rollback the transaction).
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
6 years, 7 months
[JBoss JIRA] (ISPN-8232) Transaction inconsistency during network partitions
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-8232?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-8232:
-------------------------------
Description:
In scenario where the originator stays in minor partition (in our test suite, the originator isolated tests), it is possible to a transaction to be committed and rolled back in the majority partition.
In {{Pessimitic Locking}}, the transaction is committed in one-phase using the {{PrepareCommand}}. If the partition happens when the originator sends the {{PrepareCommand}}, the nodes in the majority partition may or may not receive it. We can have the case where some nodes receive the {{PrepareCommand}} and applied and other don't receive it.
When the topology is updated in the majority partition, the {{TransactionTable}} rollbacks all transaction in which the originator isn't present. So, in the nodes where the {{PrepareCommand}} isn't received, the transaction is rolled back.
The originator in the minory partition detects the partition and marks the transaction partially completed. When the merge occurs, it tries to commit the transaction again. In the nodes where the transaction is rolled back, the transaction is marked as completed and when the {{PrepareCommand}} is received, it throws an {{IllegalStateException}} ({{TransactionTable:386, getOrCreateRemoteTransaction()}}). In this case, the transaction isn't removed from the {{PartitionHandlingManager}} and our test suite fails with {{"there are pending tx".}}
Other theoretically scenario is the {{PrepareCommand}} to be executed when no locks are acquired.
The same issue can happen with {{Optimistic Locking}} for the {{CommitCommand}}.
The problem is the transaction table can't identify is the node left gracefully or not. A solution would be to have an {{"expected members"}} list, ideally separated from the {{CacheTopology}} to avoid sending it every time. Also, it would need some sysadmin tools for the case where the node crashes and it won't be back online for a while (or for some reason, it doesn't need to be back online).
A sysadmin could remove the node from this list ({{CacheTopology}} is updated and there is no need to increase it) and decide what to do with the pending transactions (or an automatic mechanism to auto-commit/rollback the transaction).
was:
In scenario where the originator stays in minor partition (in our test suite, the originator isolated tests), it is possible to a transaction to be committed and rolled back in the majority partition.
In {{Pessimitic Locking}}, the transaction is committed in one-phase using the {{PrepareCommand}}. If the partition happens when the originator sends the {{PrepareCommand}}, the nodes in the majority partition may or may not receive it. We can have the case where some nodes receive the {{PrepareCommand}} and applied and other don't receive it.
When the topology is updated in the majority partition, the {{TransactionTable}} rollbacks all transaction in which the originator isn't present. So, in the nodes where the {{PrepareCommand}} isn't received, the transaction is rolled back.
The originator in the minory partition detects the partition and marks the transaction partially completed. When the merge occurs, it tries to commit the transaction again. In the nodes where the transaction is rolled back, the transaction is marked as completed and when the {{PrepareCommand}} is received, it throws an {{IllegalStateException}} ({{TransactionTable:386, getOrCreateRemoteTransaction()}}). In this case, the transaction isn't removed from the {{PartitionHandlingManager}} and our test suite fails with {{"there are pending tx".}}
Other theoretically scenario is the {{PrepareCommand}} to be executed when no locks are acquired.
The same issue can happen with {{Optimistic Locking}} for the {{CommitCommand}}.
The problem is the transaction table can't identify is the node left gracefully or not. A solution would be to have an {{"expected members" }}list, ideally separated from the {{CacheTopology}} to avoid sending it every time. Also, it would need some sysadmin tools for the case where the node crashes and it won't be back online for a while (or for some reason, it doesn't need to be back online).
A sysadmin could remove the node from this list ({{CacheTopology}} is updated and there is no need to increase it) and decide what to do with the pending transactions (or an automatic mechanism to auto-commit/rollback the transaction).
> Transaction inconsistency during network partitions
> ---------------------------------------------------
>
> Key: ISPN-8232
> URL: https://issues.jboss.org/browse/ISPN-8232
> Project: Infinispan
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 9.1.0.Final
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
> Priority: Critical
>
> In scenario where the originator stays in minor partition (in our test suite, the originator isolated tests), it is possible to a transaction to be committed and rolled back in the majority partition.
> In {{Pessimitic Locking}}, the transaction is committed in one-phase using the {{PrepareCommand}}. If the partition happens when the originator sends the {{PrepareCommand}}, the nodes in the majority partition may or may not receive it. We can have the case where some nodes receive the {{PrepareCommand}} and applied and other don't receive it.
> When the topology is updated in the majority partition, the {{TransactionTable}} rollbacks all transaction in which the originator isn't present. So, in the nodes where the {{PrepareCommand}} isn't received, the transaction is rolled back.
> The originator in the minory partition detects the partition and marks the transaction partially completed. When the merge occurs, it tries to commit the transaction again. In the nodes where the transaction is rolled back, the transaction is marked as completed and when the {{PrepareCommand}} is received, it throws an {{IllegalStateException}} ({{TransactionTable:386, getOrCreateRemoteTransaction()}}). In this case, the transaction isn't removed from the {{PartitionHandlingManager}} and our test suite fails with {{"there are pending tx".}}
> Other theoretically scenario is the {{PrepareCommand}} to be executed when no locks are acquired.
> The same issue can happen with {{Optimistic Locking}} for the {{CommitCommand}}.
> The problem is the transaction table can't identify is the node left gracefully or not. A solution would be to have an {{"expected members"}} list, ideally separated from the {{CacheTopology}} to avoid sending it every time. Also, it would need some sysadmin tools for the case where the node crashes and it won't be back online for a while (or for some reason, it doesn't need to be back online).
> A sysadmin could remove the node from this list ({{CacheTopology}} is updated and there is no need to increase it) and decide what to do with the pending transactions (or an automatic mechanism to auto-commit/rollback the transaction).
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
6 years, 7 months
[JBoss JIRA] (ISPN-8232) Transaction inconsistency during network partitions
by Pedro Ruivo (JIRA)
Pedro Ruivo created ISPN-8232:
---------------------------------
Summary: Transaction inconsistency during network partitions
Key: ISPN-8232
URL: https://issues.jboss.org/browse/ISPN-8232
Project: Infinispan
Issue Type: Bug
Components: Transactions
Affects Versions: 9.1.0.Final
Reporter: Pedro Ruivo
Assignee: Pedro Ruivo
Priority: Critical
In scenario where the originator stays in minor partition (in our test suite, the originator isolated tests), it is possible to a transaction to be committed and rolled back in the majority partition.
In {{Pessimitic Locking}}, the transaction is committed in one-phase using the {{PrepareCommand}}. If the partition happens when the originator sends the {{PrepareCommand}}, the nodes in the majority partition may or may not receive it. We can have the case where some nodes receive the {{PrepareCommand}} and applied and other don't receive it.
When the topology is updated in the majority partition, the {{TransactionTable}} rollbacks all transaction in which the originator isn't present. So, in the nodes where the {{PrepareCommand}} isn't received, the transaction is rolled back.
The originator in the minory partition detects the partition and marks the transaction partially completed. When the merge occurs, it tries to commit the transaction again. In the nodes where the transaction is rolled back, the transaction is marked as completed and when the {{PrepareCommand}} is received, it throws an {{IllegalStateException}} ({{TransactionTable:386, getOrCreateRemoteTransaction()}}). In this case, the transaction isn't removed from the {{PartitionHandlingManager}} and our test suite fails with {{"there are pending tx".}}
Other theoretically scenario is the {{PrepareCommand}} to be executed when no locks are acquired.
The same issue can happen with {{Optimistic Locking}} for the {{CommitCommand}}.
The problem is the transaction table can't identify is the node left gracefully or not. A solution would be to have an {{"expected members" }}list, ideally separated from the {{CacheTopology}} to avoid sending it every time. Also, it would need some sysadmin tools for the case where the node crashes and it won't be back online for a while (or for some reason, it doesn't need to be back online).
A sysadmin could remove the node from this list ({{CacheTopology}} is updated and there is no need to increase it) and decide what to do with the pending transactions (or an automatic mechanism to auto-commit/rollback the transaction).
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
6 years, 7 months
[JBoss JIRA] (ISPN-8231) InfinispanModulesStoreJdbcIT.testXmlConfig and InfinispanStoreJdbcIT.testXmlConfig random failure
by Katia Aresti (JIRA)
[ https://issues.jboss.org/browse/ISPN-8231?page=com.atlassian.jira.plugin.... ]
Katia Aresti updated ISPN-8231:
-------------------------------
Status: Pull Request Sent (was: Coding In Progress)
Git Pull Request: https://github.com/infinispan/infinispan/pull/5396
> InfinispanModulesStoreJdbcIT.testXmlConfig and InfinispanStoreJdbcIT.testXmlConfig random failure
> -------------------------------------------------------------------------------------------------
>
> Key: ISPN-8231
> URL: https://issues.jboss.org/browse/ISPN-8231
> Project: Infinispan
> Issue Type: Bug
> Reporter: Katia Aresti
> Assignee: Katia Aresti
> Labels: testsuite_stability
> Fix For: 9.1.1.Final
>
>
> org.infinispan.test.integration.as.InfinispanModulesStoreJdbcIT.testXmlConfig and
> org.infinispan.test.integration.as.InfinispanStoreJdbcIT.testXmlConfig
> conflict on JMX
> testXmlConfig
> org.infinispan.jmx.JmxDomainConflictException: ISPN000034: There's already a JMX MBean instance type=CacheManager,name="DefaultCacheManager" already registered under 'org.infinispan' JMX domain. If you want to allow multiple instances configured with same JMX domain enable 'allowDuplicateDomains' attribute in 'globalJmxStatistics' config element at org.infinispan.test.integration.as.InfinispanModulesStoreJdbcIT.testXmlConfig(InfinispanModulesStoreJdbcIT.java:57)
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
6 years, 7 months
[JBoss JIRA] (ISPN-8231) InfinispanModulesStoreJdbcIT.testXmlConfig and InfinispanStoreJdbcIT.testXmlConfig random failure
by Katia Aresti (JIRA)
Katia Aresti created ISPN-8231:
----------------------------------
Summary: InfinispanModulesStoreJdbcIT.testXmlConfig and InfinispanStoreJdbcIT.testXmlConfig random failure
Key: ISPN-8231
URL: https://issues.jboss.org/browse/ISPN-8231
Project: Infinispan
Issue Type: Bug
Reporter: Katia Aresti
Fix For: 9.1.1.Final
org.infinispan.test.integration.as.InfinispanModulesStoreJdbcIT.testXmlConfig and
org.infinispan.test.integration.as.InfinispanStoreJdbcIT.testXmlConfig
conflict on JMX
testXmlConfig
org.infinispan.jmx.JmxDomainConflictException: ISPN000034: There's already a JMX MBean instance type=CacheManager,name="DefaultCacheManager" already registered under 'org.infinispan' JMX domain. If you want to allow multiple instances configured with same JMX domain enable 'allowDuplicateDomains' attribute in 'globalJmxStatistics' config element at org.infinispan.test.integration.as.InfinispanModulesStoreJdbcIT.testXmlConfig(InfinispanModulesStoreJdbcIT.java:57)
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
6 years, 7 months
[JBoss JIRA] (ISPN-8231) InfinispanModulesStoreJdbcIT.testXmlConfig and InfinispanStoreJdbcIT.testXmlConfig random failure
by Katia Aresti (JIRA)
[ https://issues.jboss.org/browse/ISPN-8231?page=com.atlassian.jira.plugin.... ]
Katia Aresti reassigned ISPN-8231:
----------------------------------
Assignee: Katia Aresti
> InfinispanModulesStoreJdbcIT.testXmlConfig and InfinispanStoreJdbcIT.testXmlConfig random failure
> -------------------------------------------------------------------------------------------------
>
> Key: ISPN-8231
> URL: https://issues.jboss.org/browse/ISPN-8231
> Project: Infinispan
> Issue Type: Bug
> Reporter: Katia Aresti
> Assignee: Katia Aresti
> Labels: testsuite_stability
> Fix For: 9.1.1.Final
>
>
> org.infinispan.test.integration.as.InfinispanModulesStoreJdbcIT.testXmlConfig and
> org.infinispan.test.integration.as.InfinispanStoreJdbcIT.testXmlConfig
> conflict on JMX
> testXmlConfig
> org.infinispan.jmx.JmxDomainConflictException: ISPN000034: There's already a JMX MBean instance type=CacheManager,name="DefaultCacheManager" already registered under 'org.infinispan' JMX domain. If you want to allow multiple instances configured with same JMX domain enable 'allowDuplicateDomains' attribute in 'globalJmxStatistics' config element at org.infinispan.test.integration.as.InfinispanModulesStoreJdbcIT.testXmlConfig(InfinispanModulesStoreJdbcIT.java:57)
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
6 years, 7 months
[JBoss JIRA] (ISPN-8231) InfinispanModulesStoreJdbcIT.testXmlConfig and InfinispanStoreJdbcIT.testXmlConfig random failure
by Katia Aresti (JIRA)
[ https://issues.jboss.org/browse/ISPN-8231?page=com.atlassian.jira.plugin.... ]
Katia Aresti updated ISPN-8231:
-------------------------------
Status: Open (was: New)
> InfinispanModulesStoreJdbcIT.testXmlConfig and InfinispanStoreJdbcIT.testXmlConfig random failure
> -------------------------------------------------------------------------------------------------
>
> Key: ISPN-8231
> URL: https://issues.jboss.org/browse/ISPN-8231
> Project: Infinispan
> Issue Type: Bug
> Reporter: Katia Aresti
> Assignee: Katia Aresti
> Labels: testsuite_stability
> Fix For: 9.1.1.Final
>
>
> org.infinispan.test.integration.as.InfinispanModulesStoreJdbcIT.testXmlConfig and
> org.infinispan.test.integration.as.InfinispanStoreJdbcIT.testXmlConfig
> conflict on JMX
> testXmlConfig
> org.infinispan.jmx.JmxDomainConflictException: ISPN000034: There's already a JMX MBean instance type=CacheManager,name="DefaultCacheManager" already registered under 'org.infinispan' JMX domain. If you want to allow multiple instances configured with same JMX domain enable 'allowDuplicateDomains' attribute in 'globalJmxStatistics' config element at org.infinispan.test.integration.as.InfinispanModulesStoreJdbcIT.testXmlConfig(InfinispanModulesStoreJdbcIT.java:57)
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
6 years, 7 months
[JBoss JIRA] (ISPN-8231) InfinispanModulesStoreJdbcIT.testXmlConfig and InfinispanStoreJdbcIT.testXmlConfig random failure
by Katia Aresti (JIRA)
[ https://issues.jboss.org/browse/ISPN-8231?page=com.atlassian.jira.plugin.... ]
Work on ISPN-8231 started by Katia Aresti.
------------------------------------------
> InfinispanModulesStoreJdbcIT.testXmlConfig and InfinispanStoreJdbcIT.testXmlConfig random failure
> -------------------------------------------------------------------------------------------------
>
> Key: ISPN-8231
> URL: https://issues.jboss.org/browse/ISPN-8231
> Project: Infinispan
> Issue Type: Bug
> Reporter: Katia Aresti
> Assignee: Katia Aresti
> Labels: testsuite_stability
> Fix For: 9.1.1.Final
>
>
> org.infinispan.test.integration.as.InfinispanModulesStoreJdbcIT.testXmlConfig and
> org.infinispan.test.integration.as.InfinispanStoreJdbcIT.testXmlConfig
> conflict on JMX
> testXmlConfig
> org.infinispan.jmx.JmxDomainConflictException: ISPN000034: There's already a JMX MBean instance type=CacheManager,name="DefaultCacheManager" already registered under 'org.infinispan' JMX domain. If you want to allow multiple instances configured with same JMX domain enable 'allowDuplicateDomains' attribute in 'globalJmxStatistics' config element at org.infinispan.test.integration.as.InfinispanModulesStoreJdbcIT.testXmlConfig(InfinispanModulesStoreJdbcIT.java:57)
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
6 years, 7 months