[JBoss JIRA] (ISPN-4471) MapReduceTask: memory leak with useIntermediateSharedCache = true
by Rich DiCroce (JIRA)
Rich DiCroce created ISPN-4471:
----------------------------------
Summary: MapReduceTask: memory leak with useIntermediateSharedCache = true
Key: ISPN-4471
URL: https://issues.jboss.org/browse/ISPN-4471
Project: Infinispan
Issue Type: Bug
Security Level: Public (Everyone can see)
Affects Versions: 6.0.2.Final
Reporter: Rich DiCroce
Assignee: Mircea Markus
When using an intermediate shared cache for the reduce phase, MapReduceTask puts the entries into the cache with no expiration and apparently never removes them. This eventually results in OutOfMemoryErrors.
One workaround is to disable use of the intermediate shared cache, which causes a new cache to be created and destroyed for every task, which "fixes" the problem of not removing intermediate values. However, it causes a ton of log spam:
{noformat}
2014-07-02 11:55:10,014 INFO [org.infinispan.jmx.CacheJmxRegistration] (transport-thread-21) ISPN000031: MBeans were successfully registered to the platform MBean server.
2014-07-02 11:55:10,016 INFO [org.jboss.as.clustering.infinispan] (transport-thread-21) JBAS010281: Started e71dddc0-60ce-4cb9-ac8c-615d60866393 cache from GamingPortal container
2014-07-02 11:55:10,023 INFO [org.infinispan.jmx.CacheJmxRegistration] (transport-thread-5) ISPN000031: MBeans were successfully registered to the platform MBean server.
2014-07-02 11:55:10,024 INFO [org.infinispan.jmx.CacheJmxRegistration] (transport-thread-4) ISPN000031: MBeans were successfully registered to the platform MBean server.
2014-07-02 11:55:10,025 INFO [org.jboss.as.clustering.infinispan] (transport-thread-5) JBAS010281: Started 22d387d6-69c6-48b2-9701-ea64c08d66ad cache from GamingPortal container
2014-07-02 11:55:10,026 INFO [org.jboss.as.clustering.infinispan] (transport-thread-4) JBAS010281: Started bfaf92a0-a030-4624-93a7-0fee097415d7 cache from NMS container
2014-07-02 11:55:10,037 INFO [org.jboss.as.clustering.infinispan] (EJB default - 2) JBAS010282: Stopped 22d387d6-69c6-48b2-9701-ea64c08d66ad cache from GamingPortal container
2014-07-02 11:55:10,040 INFO [org.jboss.as.clustering.infinispan] (EJB default - 1) JBAS010282: Stopped bfaf92a0-a030-4624-93a7-0fee097415d7 cache from NMS container
2014-07-02 11:55:10,047 INFO [org.jboss.as.clustering.infinispan] (EJB default - 6) JBAS010282: Stopped e71dddc0-60ce-4cb9-ac8c-615d60866393 cache from GamingPortal container
2014-07-02 11:55:10,047 INFO [org.infinispan.jmx.CacheJmxRegistration] (transport-thread-0) ISPN000031: MBeans were successfully registered to the platform MBean server.
2014-07-02 11:55:10,048 INFO [org.jboss.as.clustering.infinispan] (transport-thread-0) JBAS010281: Started bed74bd3-a227-43e0-b262-62c19dd444a7 cache from GamingPortal container
2014-07-02 11:55:10,052 INFO [org.jboss.as.clustering.infinispan] (EJB default - 2) JBAS010282: Stopped bed74bd3-a227-43e0-b262-62c19dd444a7 cache from GamingPortal container
2014-07-02 11:55:10,063 INFO [org.infinispan.jmx.CacheJmxRegistration] (transport-thread-7) ISPN000031: MBeans were successfully registered to the platform MBean server.
2014-07-02 11:55:10,064 INFO [org.jboss.as.clustering.infinispan] (transport-thread-7) JBAS010281: Started 63cce570-0169-40c2-bc9f-e045c2864702 cache from GamingPortal container
2014-07-02 11:55:10,068 INFO [org.jboss.as.clustering.infinispan] (EJB default - 2) JBAS010282: Stopped 63cce570-0169-40c2-bc9f-e045c2864702 cache from GamingPortal container
2014-07-02 11:55:10,072 INFO [org.infinispan.jmx.CacheJmxRegistration] (transport-thread-19) ISPN000031: MBeans were successfully registered to the platform MBean server.
2014-07-02 11:55:10,073 INFO [org.jboss.as.clustering.infinispan] (transport-thread-19) JBAS010281: Started 83f7b355-d4c6-4a0a-aade-ce2509293d77 cache from GamingPortal container
2014-07-02 11:55:10,077 INFO [org.jboss.as.clustering.infinispan] (EJB default - 2) JBAS010282: Stopped 83f7b355-d4c6-4a0a-aade-ce2509293d77 cache from GamingPortal container
{noformat}
I also observed one NullPointerException with distributeReducePhase = true and useIntermediateSharedCache = false. This could be related to ISPN-4460, but I'm not sure.
{noformat}
Caused by: org.infinispan.commons.CacheException: java.util.concurrent.ExecutionException: org.infinispan.commons.CacheException: java.lang.NullPointerException
at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:348) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:634) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
at org.infinispan.distexec.mapreduce.MapReduceTask$3.call(MapReduceTask.java:652) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
at org.infinispan.distexec.mapreduce.MapReduceTask$MapReduceTaskFuture.get(MapReduceTask.java:760) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
... 63 more
Caused by: java.util.concurrent.ExecutionException: org.infinispan.commons.CacheException: java.lang.NullPointerException
at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_45]
at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_45]
at org.infinispan.distexec.mapreduce.MapReduceTask$TaskPart.get(MapReduceTask.java:845) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
at org.infinispan.distexec.mapreduce.MapReduceTask.executeMapPhase(MapReduceTask.java:439) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:342) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
... 66 more
Caused by: org.infinispan.commons.CacheException: java.lang.NullPointerException
at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:100) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
at org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart.invokeMapCombineLocally(MapReduceTask.java:967) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
at org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart.access$200(MapReduceTask.java:894) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
at org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart$1.call(MapReduceTask.java:916) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
at org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart$1.call(MapReduceTask.java:912) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_45]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]
Caused by: java.lang.NullPointerException
at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapKeysToNodes(MapReduceManagerImpl.java:355) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.migrateIntermediateKeys(MapReduceManagerImpl.java:264) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:258) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:98) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
... 10 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
9 years, 10 months
[JBoss JIRA] (ISPN-4467) keySet operation via HotRod in compatibility mode throws ClassCastException
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-4467?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño updated ISPN-4467:
-----------------------------------
Affects Version/s: 7.0.0.Alpha1
> keySet operation via HotRod in compatibility mode throws ClassCastException
> ---------------------------------------------------------------------------
>
> Key: ISPN-4467
> URL: https://issues.jboss.org/browse/ISPN-4467
> Project: Infinispan
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Affects Versions: 7.0.0.Alpha1
> Reporter: Martin Gencur
> Assignee: Galder Zamarreño
> Fix For: 7.0.0.Alpha5
>
>
> The HotRod client's keySet() operation throws ClassCastException due to the following reason:
> When Encoder2x.scala in its writeResponse() method decodes the operation as BulkGetKeysResponse, it runs a Map/Reduce job that returns a set of keys in the whole cluster.
> The operation returns a set of "unmarshalled" (cos we're in compatibility mode) entries. However, Scala infers the type of individual entries as "Bytes" which is an alias for Array[Byte].
> As a result, when this iterator from this key set is retrieved, it is not possible to iterate through the entries because Scala automatically tries to convert each (unmarshalled) entry into a byte array, which results in the exception.
> This line results in throwing CCE:
> https://github.com/infinispan/infinispan/blob/master/server/hotrod/src/ma...
>
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
9 years, 10 months
[JBoss JIRA] (ISPN-4467) keySet operation via HotRod in compatibility mode throws ClassCastException
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-4467?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño commented on ISPN-4467:
----------------------------------------
The reason this is failing now is due to ISPN-2284, which changed the way the cached entries are iterated. Before, cache.getAdvancedCache().withFlags(Flag.SKIP_CACHE_LOAD).keySet() was used which guaranteed that the correct transformations would happen as a result of keySet() going through the interceptor chain. Starting with ISPN-2284, the data container is hit directly passing in a task to iterate over the data container contents directly, avoiding the interceptor chain altogether.
> keySet operation via HotRod in compatibility mode throws ClassCastException
> ---------------------------------------------------------------------------
>
> Key: ISPN-4467
> URL: https://issues.jboss.org/browse/ISPN-4467
> Project: Infinispan
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Reporter: Martin Gencur
> Assignee: Galder Zamarreño
> Fix For: 7.0.0.Alpha5
>
>
> The HotRod client's keySet() operation throws ClassCastException due to the following reason:
> When Encoder2x.scala in its writeResponse() method decodes the operation as BulkGetKeysResponse, it runs a Map/Reduce job that returns a set of keys in the whole cluster.
> The operation returns a set of "unmarshalled" (cos we're in compatibility mode) entries. However, Scala infers the type of individual entries as "Bytes" which is an alias for Array[Byte].
> As a result, when this iterator from this key set is retrieved, it is not possible to iterate through the entries because Scala automatically tries to convert each (unmarshalled) entry into a byte array, which results in the exception.
> This line results in throwing CCE:
> https://github.com/infinispan/infinispan/blob/master/server/hotrod/src/ma...
>
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
9 years, 10 months
[JBoss JIRA] (ISPN-4467) keySet operation via HotRod in compatibility mode throws ClassCastException
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-4467?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño updated ISPN-4467:
-----------------------------------
Fix Version/s: 7.0.0.Alpha5
> keySet operation via HotRod in compatibility mode throws ClassCastException
> ---------------------------------------------------------------------------
>
> Key: ISPN-4467
> URL: https://issues.jboss.org/browse/ISPN-4467
> Project: Infinispan
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Reporter: Martin Gencur
> Assignee: Galder Zamarreño
> Fix For: 7.0.0.Alpha5
>
>
> The HotRod client's keySet() operation throws ClassCastException due to the following reason:
> When Encoder2x.scala in its writeResponse() method decodes the operation as BulkGetKeysResponse, it runs a Map/Reduce job that returns a set of keys in the whole cluster.
> The operation returns a set of "unmarshalled" (cos we're in compatibility mode) entries. However, Scala infers the type of individual entries as "Bytes" which is an alias for Array[Byte].
> As a result, when this iterator from this key set is retrieved, it is not possible to iterate through the entries because Scala automatically tries to convert each (unmarshalled) entry into a byte array, which results in the exception.
> This line results in throwing CCE:
> https://github.com/infinispan/infinispan/blob/master/server/hotrod/src/ma...
>
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
9 years, 10 months
[JBoss JIRA] (ISPN-4467) keySet operation via HotRod in compatibility mode throws ClassCastException
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-4467?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño reassigned ISPN-4467:
--------------------------------------
Assignee: Galder Zamarreño (was: Martin Gencur)
> keySet operation via HotRod in compatibility mode throws ClassCastException
> ---------------------------------------------------------------------------
>
> Key: ISPN-4467
> URL: https://issues.jboss.org/browse/ISPN-4467
> Project: Infinispan
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Reporter: Martin Gencur
> Assignee: Galder Zamarreño
>
> The HotRod client's keySet() operation throws ClassCastException due to the following reason:
> When Encoder2x.scala in its writeResponse() method decodes the operation as BulkGetKeysResponse, it runs a Map/Reduce job that returns a set of keys in the whole cluster.
> The operation returns a set of "unmarshalled" (cos we're in compatibility mode) entries. However, Scala infers the type of individual entries as "Bytes" which is an alias for Array[Byte].
> As a result, when this iterator from this key set is retrieved, it is not possible to iterate through the entries because Scala automatically tries to convert each (unmarshalled) entry into a byte array, which results in the exception.
> This line results in throwing CCE:
> https://github.com/infinispan/infinispan/blob/master/server/hotrod/src/ma...
>
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
9 years, 10 months
[JBoss JIRA] (ISPN-4467) keySet operation via HotRod in compatibility mode throws ClassCastException
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-4467?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño commented on ISPN-4467:
----------------------------------------
Bulk get keys relies on map/reduce functions to retrieve all keys in the cache cluster-wide. Map/reduce however skips the interceptor chain altogether, skiping the interceptor doing transformations, and instead hits directly the data container to do its job. AdvancedCache.filterEntries() cannot be used alternatively due to similar problems, so the cleanest way to solve this is to use a different collator for when compatibility is enabled.
> keySet operation via HotRod in compatibility mode throws ClassCastException
> ---------------------------------------------------------------------------
>
> Key: ISPN-4467
> URL: https://issues.jboss.org/browse/ISPN-4467
> Project: Infinispan
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Reporter: Martin Gencur
> Assignee: Martin Gencur
>
> The HotRod client's keySet() operation throws ClassCastException due to the following reason:
> When Encoder2x.scala in its writeResponse() method decodes the operation as BulkGetKeysResponse, it runs a Map/Reduce job that returns a set of keys in the whole cluster.
> The operation returns a set of "unmarshalled" (cos we're in compatibility mode) entries. However, Scala infers the type of individual entries as "Bytes" which is an alias for Array[Byte].
> As a result, when this iterator from this key set is retrieved, it is not possible to iterate through the entries because Scala automatically tries to convert each (unmarshalled) entry into a byte array, which results in the exception.
> This line results in throwing CCE:
> https://github.com/infinispan/infinispan/blob/master/server/hotrod/src/ma...
>
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
9 years, 10 months
[JBoss JIRA] (ISPN-4470) cache.keySet().size() returns different value than cache.size() for HotRod client
by Martin Gencur (JIRA)
Martin Gencur created ISPN-4470:
-----------------------------------
Summary: cache.keySet().size() returns different value than cache.size() for HotRod client
Key: ISPN-4470
URL: https://issues.jboss.org/browse/ISPN-4470
Project: Infinispan
Issue Type: Bug
Security Level: Public (Everyone can see)
Affects Versions: 7.0.0.Alpha4
Reporter: Martin Gencur
Assignee: Mircea Markus
cache.keySet().size() returns the number of all keys in the cluster (even a distributed one) while cache.size() returns just local cache size (this might be different for a distributed cache)
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
9 years, 10 months