[infinispan-issues] [JBoss JIRA] (ISPN-4964) Map/Reduce intermittently hangs when the cache is updated during execution

[Alexandre Nikolov (JIRA)]

Alexandre Nikolov created ISPN-4964:
---------------------------------------

             Summary: Map/Reduce intermittently hangs when the cache is updated during execution
                 Key: ISPN-4964
                 URL: https://issues.jboss.org/browse/ISPN-4964
             Project: Infinispan
          Issue Type: Bug
          Components: Core
    Affects Versions: 7.0.0.Final, 6.0.2.Final, 6.0.1.Final
         Environment: JDK 7 (1.2.0_51)  Glassfish 3.2.1.1, WildFly 8.1.0.Final
            Reporter: Alexandre Nikolov


Map/Reduce tasks on a cache with intensive write operations intermittently hangs, resulting in high CPU and Heap Memory usage. If left like that eventually it crashes the VM after multiple out of memory exceptions.

Here is a stack dump of a hanging thread:
Name: transport-thread--p3-t17
State: RUNNABLE
Total blocked: 709  Total waited: 15 204

Stack trace: 
java.util.HashMap.hash(HashMap.java:366)
java.util.HashMap.put(HashMap.java:496)
java.util.HashSet.add(HashSet.java:217)
org.infinispan.persistence.async.AdvancedAsyncCacheLoader.process(AdvancedAsyncCacheLoader.java:81)
org.infinispan.persistence.manager.PersistenceManagerImpl.processOnAllStores(PersistenceManagerImpl.java:418)
org.infinispan.persistence.manager.PersistenceManagerImpl.processOnAllStores(PersistenceManagerImpl.java:403)
org.infinispan.persistence.manager.PersistenceManagerImpl.processOnAllStores(PersistenceManagerImpl.java:398)
org.infinispan.distexec.mapreduce.MapReduceManagerImpl.map(MapReduceManagerImpl.java:213)
org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForLocalReduction(MapReduceManagerImpl.java:94)
org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart.invokeMapCombineLocallyForLocalReduction(MapReduceTask.java:1162)
org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart.access$400(MapReduceTask.java:1101)
org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart$2.call(MapReduceTask.java:1133)
org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart$2.call(MapReduceTask.java:1129)
java.util.concurrent.FutureTask.run(FutureTask.java:262)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
java.util.concurrent.FutureTask.run(FutureTask.java:262)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)


The hanging thread keep cycling between these 2 stack entries:
java.util.HashSet.add(HashSet.java:217)
org.infinispan.persistence.async.AdvancedAsyncCacheLoader.process(AdvancedAsyncCacheLoader.java:81)

Here is the configuration for 7.x:
<?xml version="1.0" encoding="UTF-8"?>
<infinispan
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="urn:infinispan:config:7.0 http://www.infinispan.org/schemas/infinispan-config-7.0.xsd"
      xmlns="urn:infinispan:config:7.0">
    <cache-container default-cache="default">
        <jmx duplicate-domains="true"/>
        <local-cache name="default">
            <eviction max-entries="5000" strategy="LIRS"/> 
            <persistence passivation="true" >
                <!--soft-index-file-store xmlns="urn:infinispan:config:soft-index:7.0"
                        purge="false" preload="true">
                    <index path="/cache/index" />
                    <data path="/cache/data" />
                    <write-behind/>
                </soft-index-file-store-->                
                <file-store fetch-state="true" preload="true" path="/cache">
                    <write-behind/>
                </file-store>
            </persistence>
            <locking isolation="REPEATABLE_READ"/>
            <transaction mode="BATCH" auto-commit="true"/>
        </local-cache>
        
        <local-cache name="intraday">
            <expiration lifespan="86400000"/>
        </local-cache>
    </cache-container>
</infinispan>

The issue also happens on versions 6.x.
It happens with different combination of locking and transaction settings - that is it happens with transactional and non-transactional caches with different locking configurations. I tried all possible combinations and the issue happens intermittently on all of them.




--
This message was sent by Atlassian JIRA
(v6.3.8#6338)