[JBoss JIRA] (ISPN-5241) Cache topology updates should use the NO_FC flag
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-5241?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo updated ISPN-5241:
------------------------------
Fix Version/s: 8.0.0.Final
(was: 7.2.0.Final)
> Cache topology updates should use the NO_FC flag
> ------------------------------------------------
>
> Key: ISPN-5241
> URL: https://issues.jboss.org/browse/ISPN-5241
> Project: Infinispan
> Issue Type: Bug
> Components: Core, State Transfer
> Affects Versions: 7.1.0.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Fix For: 8.0.0.Final
>
>
> Topology updates are sent while holding the ClusterCacheStatus lock, so they should never block. However, when MFC is present, the topology update can block waiting for enough credits. As most CacheTopologyControlCommands need to acquire the ClusterCacheStatus lock, this can easily lead to a full remote-executor pool (and OOB pool) and the appearance of a deadlock.
> What's more, if one node is not responsive, it can block all the other nodes from receiving further topology updates. Topology updates should be as prompt as possible, so we should use the NO_FC flag to ensure that each node receives topology updates as soon as possible.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years
[JBoss JIRA] (ISPN-5379) Persistence with stringKeyedJdbcStore throwing ConcurrentModificationException while adding data to cache
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-5379?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo updated ISPN-5379:
------------------------------
Fix Version/s: 8.0.0.Final
(was: 7.2.0.Final)
> Persistence with stringKeyedJdbcStore throwing ConcurrentModificationException while adding data to cache
> ----------------------------------------------------------------------------------------------------------
>
> Key: ISPN-5379
> URL: https://issues.jboss.org/browse/ISPN-5379
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Loaders and Stores
> Affects Versions: 6.0.2.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 8.0.0.Final
>
>
> From Dennis Reed:
> A ConcurrentModificationException in the CacheWriter interceptor during a commit.
> java.util.ConcurrentModificationException
> at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:953)
> at java.util.LinkedList$ListItr.next(LinkedList.java:886)
> at org.infinispan.interceptors.CacheWriterInterceptor.store(CacheWriterInterceptor.java:210)
> at org.infinispan.interceptors.CacheWriterInterceptor.commitCommand(CacheWriterInterceptor.java:115)
> The code is looping through the modifications associated with the transaction:
> List<WriteCommand> modifications = ctx.getCacheTransaction().getAllModifications();
> ...
> 210: for (WriteCommand cacheCommand : modifications) {
> The transaction's modification list is stored as a synchronized list with a comment "we need to synchronize this collection to be able to get a valid snapshot from another thread during state transfer".
> But this thread is not doing state transfer, and I'd assume "get a valid snapshot" wouldn't include modification?
> Is it valid for this list to be modified from another thread (in which case all iterations should be synchronized), or is something modifying it incorrectly?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years
[JBoss JIRA] (ISPN-5420) Thread pools are depleted by ClusterTopologyManagerImpl.waitForView() and causing deadlock
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-5420?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo updated ISPN-5420:
------------------------------
Fix Version/s: 8.0.0.Final
(was: 7.2.0.Final)
> Thread pools are depleted by ClusterTopologyManagerImpl.waitForView() and causing deadlock
> ------------------------------------------------------------------------------------------
>
> Key: ISPN-5420
> URL: https://issues.jboss.org/browse/ISPN-5420
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 6.0.2.Final, 7.1.1.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Fix For: 8.0.0.Final
>
>
> The join process was designed in the idea that a node would start its caches in sequential order, so {{ClusterTopologyManager.waitForView()}} would block at most once for each joining node. However, WildFly actually starts {{2 * Runtime.availableProcessors()}} caches in parallel, and this can be a problem when the machine has a lot of cores and multiple nodes.
> {{ClustertopologyManager.handleClusterView()}} only updates the {{viewId}} after it updated the cache topologies of each cache AND after it confirmed the availability of all the nodes with a {{POLICY_GET_STATUS}} RPC. This RPC can block, and it's very easy for the remote-executor thread pool on the coordinator to become overloades with threads like this:
> {noformat}
> "remote-thread-172" daemon prio=10 tid=0x00007f0cc48c0000 nid=0x28ca4 in Object.wait() [0x00007f0c5f25b000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at org.infinispan.topology.ClusterTopologyManagerImpl.waitForView(ClusterTopologyManagerImpl.java:357)
> - locked <0x00000000ff3bd900> (a java.lang.Object)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleJoin(ClusterTopologyManagerImpl.java:123)
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:162)
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:144)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:276)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years