3 p.m.
Hello,
In a recent heap dump analysis I found that
org.infinispan.marshall.core.JBossMarshaller consumes a lot of
memory(about 46 MB) that seems to be unused.
This is due to PerThreadInstanceHolder having ExtendedRiverMarshaller
objects that contain big IdentityIntMap objects. Some of those
IdentityIntMap instances have a size of 2 million entries, but most of
them have sizes of a few 100 thousands.
When I look into these IdentityIntMap instances, it seems that the
entries are all unused.
Is that kind of memory consumption expected or does that indicate a
possibly wrong configuration?
I am using Infinispan 7.2.4.Final on Wildfly 9.0.1.Final.
3:56 p.m.
Those IdentityIntMap are caches meant to speed up serialization if the same objects or
types are marshalled again. It's normal for them to be populated as marshalling
operations are executed. We don't currently have a way to clear these caches.
Cheers,
--
Galder Zamarreño
Infinispan, Red Hat
On 22 Nov 2015, at 15:00, Christian Beikov
<christian(a)sweazer.com> wrote:
Hello,
In a recent heap dump analysis I found that
org.infinispan.marshall.core.JBossMarshaller consumes a lot of
memory(about 46 MB) that seems to be unused.
This is due to PerThreadInstanceHolder having ExtendedRiverMarshaller
objects that contain big IdentityIntMap objects. Some of those
IdentityIntMap instances have a size of 2 million entries, but most of
them have sizes of a few 100 thousands.
When I look into these IdentityIntMap instances, it seems that the
entries are all unused.
Is that kind of memory consumption expected or does that indicate a
possibly wrong configuration?
I am using Infinispan 7.2.4.Final on Wildfly 9.0.1.Final.
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
9:26 a.m.
Are you going to do something about this memory consumption or is there
at least some kind of minimum expected memory usage you can give me?
I ran into an OOMEs the other day and the cluster was unable to recover
from that by restarting single nodes. The nodes couldn't synchronize
because of the OOMEs. I had to (jgroups-)disconnect all nodes from the
cluster and start a separate cluster which of course lead to data loss.
All of this happened because of some wrong memory consumption
estimations I made so in order to avoid that in the future I would like
to plan better ahead. Is there any other way to avoid such a cluster death?
Regards,
Christian
Am 26.11.2015 um 15:56 schrieb Galder Zamarreño:
Those IdentityIntMap are caches meant to speed up serialization if
the same objects or types are marshalled again. It's normal for them to be populated
as marshalling operations are executed. We don't currently have a way to clear these
caches.
Cheers,
--
Galder Zamarreño
Infinispan, Red Hat
> On 22 Nov 2015, at 15:00, Christian Beikov <christian(a)sweazer.com> wrote:
>
> Hello,
>
> In a recent heap dump analysis I found that
> org.infinispan.marshall.core.JBossMarshaller consumes a lot of
> memory(about 46 MB) that seems to be unused.
> This is due to PerThreadInstanceHolder having ExtendedRiverMarshaller
> objects that contain big IdentityIntMap objects. Some of those
> IdentityIntMap instances have a size of 2 million entries, but most of
> them have sizes of a few 100 thousands.
> When I look into these IdentityIntMap instances, it seems that the
> entries are all unused.
>
> Is that kind of memory consumption expected or does that indicate a
> possibly wrong configuration?
>
> I am using Infinispan 7.2.4.Final on Wildfly 9.0.1.Final.
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
5:57 p.m.
We're actively working to reduce our memory footprint.
I can't really provide guidance on memory requirements since it's very dependant
on the types stored and the amount of instances that are stored, which is specific to each
use case.
It's worth investing some time estimating loads and running load tests to adjust
memory parameters before going to production.
Cheers,
--
Galder Zamarreño
Infinispan, Red Hat
On 27 Nov 2015, at 09:26, Christian Beikov
<christian(a)sweazer.com> wrote:
Are you going to do something about this memory consumption or is there
at least some kind of minimum expected memory usage you can give me?
I ran into an OOMEs the other day and the cluster was unable to recover
from that by restarting single nodes. The nodes couldn't synchronize
because of the OOMEs. I had to (jgroups-)disconnect all nodes from the
cluster and start a separate cluster which of course lead to data loss.
All of this happened because of some wrong memory consumption
estimations I made so in order to avoid that in the future I would like
to plan better ahead. Is there any other way to avoid such a cluster death?
Regards,
Christian
Am 26.11.2015 um 15:56 schrieb Galder Zamarreño:
> Those IdentityIntMap are caches meant to speed up serialization if the same objects
or types are marshalled again. It's normal for them to be populated as marshalling
operations are executed. We don't currently have a way to clear these caches.
>
> Cheers,
> --
> Galder Zamarreño
> Infinispan, Red Hat
>
>> On 22 Nov 2015, at 15:00, Christian Beikov <christian(a)sweazer.com> wrote:
>>
>> Hello,
>>
>> In a recent heap dump analysis I found that
>> org.infinispan.marshall.core.JBossMarshaller consumes a lot of
>> memory(about 46 MB) that seems to be unused.
>> This is due to PerThreadInstanceHolder having ExtendedRiverMarshaller
>> objects that contain big IdentityIntMap objects. Some of those
>> IdentityIntMap instances have a size of 2 million entries, but most of
>> them have sizes of a few 100 thousands.
>> When I look into these IdentityIntMap instances, it seems that the
>> entries are all unused.
>>
>> Is that kind of memory consumption expected or does that indicate a
>> possibly wrong configuration?
>>
>> I am using Infinispan 7.2.4.Final on Wildfly 9.0.1.Final.
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
1:44 p.m.
Hi Christian
If I were to guess, I would say your stateTransfer.chunkSize is too
high. The default is 512, but you may need an even lower value.
Those IdentityIntMaps can only get as big as the number of objects
referenced by a single Infinispan command (well, 2x to be more
precise). I doubt your keys and values each aggregate 100000s of
objects, so the culprit is most likely a command that aggregates lots
of entries - like StateResponseCommand, which can hold chunkSize
entries.
I would also recommend taking a heap dump with allocation profiling
enabled (e.g. with FlightRecorder or JProfiler). If the allocation
stacks of those IdentityIntMap's keys and values arrays contain an
OutboundTransferTask frame, then reducing the chunkSize will
definitely help.
Unfortunately there are many things, both in the configuration and in
the way you use Infinispan, which will affect the amount of memory it
uses. E.g. state transfer also keeps track of entries modified since
state transfer started. If your values are big and/or your state
transfer is short, that memory is insignificant, but if your values
are small and for some reason state transfer takes a long time, that
extra memory can become a problem. The only way to protect yourself is
to leave a lot of margin for unexpected stuff, and investigating
whenever your application dips into that margin.
Cheers
Dan
On Mon, Nov 30, 2015 at 6:57 PM, Galder Zamarreño <galder(a)redhat.com> wrote:
We're actively working to reduce our memory footprint.
I can't really provide guidance on memory requirements since it's very dependant
on the types stored and the amount of instances that are stored, which is specific to each
use case.
It's worth investing some time estimating loads and running load tests to adjust
memory parameters before going to production.
Cheers,
--
Galder Zamarreño
Infinispan, Red Hat
> On 27 Nov 2015, at 09:26, Christian Beikov <christian(a)sweazer.com> wrote:
>
> Are you going to do something about this memory consumption or is there
> at least some kind of minimum expected memory usage you can give me?
> I ran into an OOMEs the other day and the cluster was unable to recover
> from that by restarting single nodes. The nodes couldn't synchronize
> because of the OOMEs. I had to (jgroups-)disconnect all nodes from the
> cluster and start a separate cluster which of course lead to data loss.
> All of this happened because of some wrong memory consumption
> estimations I made so in order to avoid that in the future I would like
> to plan better ahead. Is there any other way to avoid such a cluster death?
>
> Regards,
> Christian
>
> Am 26.11.2015 um 15:56 schrieb Galder Zamarreño:
>> Those IdentityIntMap are caches meant to speed up serialization if the same
objects or types are marshalled again. It's normal for them to be populated as
marshalling operations are executed. We don't currently have a way to clear these
caches.
>>
>> Cheers,
>> --
>> Galder Zamarreño
>> Infinispan, Red Hat
>>
>>> On 22 Nov 2015, at 15:00, Christian Beikov <christian(a)sweazer.com>
wrote:
>>>
>>> Hello,
>>>
>>> In a recent heap dump analysis I found that
>>> org.infinispan.marshall.core.JBossMarshaller consumes a lot of
>>> memory(about 46 MB) that seems to be unused.
>>> This is due to PerThreadInstanceHolder having ExtendedRiverMarshaller
>>> objects that contain big IdentityIntMap objects. Some of those
>>> IdentityIntMap instances have a size of 2 million entries, but most of
>>> them have sizes of a few 100 thousands.
>>> When I look into these IdentityIntMap instances, it seems that the
>>> entries are all unused.
>>>
>>> Is that kind of memory consumption expected or does that indicate a
>>> possibly wrong configuration?
>>>
>>> I am using Infinispan 7.2.4.Final on Wildfly 9.0.1.Final.
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
3117
days inactive
3128
days old
infinispan-dev@lists.jboss.org
4 comments
3 participants
participants (3)
-
Christian Beikov -
Dan Berindei -
Galder Zamarreño