[JBoss JIRA] (ISPN-11101) Purge on JDBC shared stores can cause deadlocks
by Tristan Tarrant (Jira)
[ https://issues.redhat.com/browse/ISPN-11101?page=com.atlassian.jira.plugi... ]
Tristan Tarrant updated ISPN-11101:
-----------------------------------
Status: Open (was: New)
> Purge on JDBC shared stores can cause deadlocks
> -----------------------------------------------
>
> Key: ISPN-11101
> URL: https://issues.redhat.com/browse/ISPN-11101
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 9.4.17.Final, 10.1.0.CR1
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Priority: Major
> Fix For: 10.1.0.Final, 9.4.18.Final
>
>
> ISPN-10337 ensured that the JdbcStringBasedStore correctly acquired the locks of expired rows during the purging of store entries, however it has exposed an issue with shared stores. When the jdbc store is shared, the coordinator locks the rows of the affected entries and releases them once they have all been removed as part of the purge transaction. However, the call to {{ExpirationManager::handleInStoreExpiration}} also sends a {{RemoveExpiredCommand}} to ensure that entries are removed from memory. This results in a {{java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction}} being thrown by the store's delete method when the purge process is still in progress, because the expired rows are still locked and the purge process cannot complete until the {{RemoveExpiredCommand}} has been executed.
> In summary:
> # purge locks all the rows
> # purge sends a RemoveExpiredCommand and waits for it to complete
> # RemoveExpiredCommand tries to remove the row but it can't as it is locked
> Solution, send the {{RemoveExpiredCommand}} with the {{SKIP_CACHE_STORE}} flag when a store is shared,so that it only removes entries from memory.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 4 months
[JBoss JIRA] (ISPN-11101) Purge on JDBC shared stores can cause deadlocks
by Tristan Tarrant (Jira)
[ https://issues.redhat.com/browse/ISPN-11101?page=com.atlassian.jira.plugi... ]
Tristan Tarrant updated ISPN-11101:
-----------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/7692
> Purge on JDBC shared stores can cause deadlocks
> -----------------------------------------------
>
> Key: ISPN-11101
> URL: https://issues.redhat.com/browse/ISPN-11101
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 9.4.17.Final, 10.1.0.CR1
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Priority: Major
> Fix For: 10.1.0.Final, 9.4.18.Final
>
>
> ISPN-10337 ensured that the JdbcStringBasedStore correctly acquired the locks of expired rows during the purging of store entries, however it has exposed an issue with shared stores. When the jdbc store is shared, the coordinator locks the rows of the affected entries and releases them once they have all been removed as part of the purge transaction. However, the call to {{ExpirationManager::handleInStoreExpiration}} also sends a {{RemoveExpiredCommand}} to ensure that entries are removed from memory. This results in a {{java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction}} being thrown by the store's delete method when the purge process is still in progress, because the expired rows are still locked and the purge process cannot complete until the {{RemoveExpiredCommand}} has been executed.
> In summary:
> # purge locks all the rows
> # purge sends a RemoveExpiredCommand and waits for it to complete
> # RemoveExpiredCommand tries to remove the row but it can't as it is locked
> Solution, send the {{RemoveExpiredCommand}} with the {{SKIP_CACHE_STORE}} flag when a store is shared,so that it only removes entries from memory.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 4 months
[JBoss JIRA] (ISPN-11108) Move eviction components to impl package
by Dan Berindei (Jira)
Dan Berindei created ISPN-11108:
-----------------------------------
Summary: Move eviction components to impl package
Key: ISPN-11108
URL: https://issues.redhat.com/browse/ISPN-11108
Project: Infinispan
Issue Type: Task
Components: Core
Affects Versions: 10.1.0.CR1
Reporter: Dan Berindei
Assignee: Dan Berindei
Fix For: 10.1.0.Final
{{ActivationManager}} and {{PassivationManager}} are in a public package, probably because they were supposed to be used by custom {{DataContainer}} implementations. We no longer support custom {{DataContainer}} implementations, so we should move them to the impl package.
{{EvictionManager}} is currently accessible through {{AdvancedCache.getEvictionManager()}}, so we can only deprecate it now and remove it in 11.0.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 4 months
[JBoss JIRA] (ISPN-11108) Move eviction components to impl package
by Dan Berindei (Jira)
[ https://issues.redhat.com/browse/ISPN-11108?page=com.atlassian.jira.plugi... ]
Dan Berindei updated ISPN-11108:
--------------------------------
Status: Open (was: New)
> Move eviction components to impl package
> ----------------------------------------
>
> Key: ISPN-11108
> URL: https://issues.redhat.com/browse/ISPN-11108
> Project: Infinispan
> Issue Type: Task
> Components: Core
> Affects Versions: 10.1.0.CR1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Fix For: 10.1.0.Final
>
>
> {{ActivationManager}} and {{PassivationManager}} are in a public package, probably because they were supposed to be used by custom {{DataContainer}} implementations. We no longer support custom {{DataContainer}} implementations, so we should move them to the impl package.
> {{EvictionManager}} is currently accessible through {{AdvancedCache.getEvictionManager()}}, so we can only deprecate it now and remove it in 11.0.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 4 months
[JBoss JIRA] (ISPN-10882) Remove the warn message ISPN000026: Caught exception purging data container
by Dan Berindei (Jira)
[ https://issues.redhat.com/browse/ISPN-10882?page=com.atlassian.jira.plugi... ]
Dan Berindei updated ISPN-10882:
--------------------------------
Status: Open (was: New)
> Remove the warn message ISPN000026: Caught exception purging data container
> ---------------------------------------------------------------------------
>
> Key: ISPN-10882
> URL: https://issues.redhat.com/browse/ISPN-10882
> Project: Infinispan
> Issue Type: Bug
> Reporter: Diego Lovison
> Assignee: Dan Berindei
> Priority: Major
>
> When the node1 is leaving the cluster we could have the following warn message:
> {noformat}
> [0m[33m18:43:28,387 WARN [org.infinispan.expiration.impl.ClusterExpirationManager] (expiration-thread--p7-t1) ISPN000026: Caught exception purging data container!: java.lang.IllegalArgumentException: Node edg-perf01-2556 is not a member
> at org.infinispan.distribution.ch.impl.DefaultConsistentHash.getPrimarySegmentsForOwner(DefaultConsistentHash.java:128)
> at org.infinispan.distribution.group.impl.PartitionerConsistentHash.getPrimarySegmentsForOwner(PartitionerConsistentHash.java:76)
> at org.infinispan.expiration.impl.ClusterExpirationManager.purgeInMemoryContents(ClusterExpirationManager.java:123)
> at org.infinispan.expiration.impl.ClusterExpirationManager.processExpiration(ClusterExpirationManager.java:98)
> at org.infinispan.expiration.impl.ExpirationManagerImpl$ScheduledTask.run(ExpirationManagerImpl.java:282)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {noformat}
> Dan added the following comments:
> {noformat}
> Dan: but I think it can actually happen during start and/or after a merge
> Dan: when joining, the first cache topology usually doesn't have the joiner as a member
> Dan: after a merge as well, nodes that are not in the majority partition will receive a cache topology in which they are not members
> Dan: luckily StateConsumerImpl clears the data container and private stores after receiving that cache topology anyway, so there's nothing to expire
> Dan: and after the node becomes a full member again expiration will work
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 4 months