[infinispan-dev] Memory consumption of org.infinispan.marshall.core.JBossMarshaller

[Dan Berindei]

Hi Christian

If I were to guess, I would say your stateTransfer.chunkSize is too
high. The default is 512, but you may need an even lower value.

Those IdentityIntMaps can only get as big as the number of objects
referenced by a single Infinispan command (well, 2x to be more
precise). I doubt your keys and values each aggregate 100000s of
objects, so the culprit is most likely a command that aggregates lots
of entries - like StateResponseCommand, which can hold chunkSize
entries.

I would also recommend taking a heap dump with allocation profiling
enabled (e.g. with FlightRecorder or JProfiler). If the allocation
stacks of those IdentityIntMap's keys and values arrays contain an
OutboundTransferTask frame, then reducing the chunkSize will
definitely help.

Unfortunately there are many things, both in the configuration and in
the way you use Infinispan, which will affect the amount of memory it
uses. E.g. state transfer also keeps track of entries modified since
state transfer started. If your values are big and/or your state
transfer is short, that memory is insignificant, but if your values
are small and for some reason state transfer takes a long time, that
extra memory can become a problem. The only way to protect yourself is
to leave a lot of margin for unexpected stuff, and investigating
whenever your application dips into that margin.

Cheers
Dan


On Mon, Nov 30, 2015 at 6:57 PM, Galder Zamarreño <galder at redhat.com> wrote:
> We're actively working to reduce our memory footprint.
>
> I can't really provide guidance on memory requirements since it's very dependant on the types stored and the amount of instances that are stored, which is specific to each use case.
>
> It's worth investing some time estimating loads and running load tests to adjust memory parameters before going to production.
>
> Cheers,
> --
> Galder Zamarreño
> Infinispan, Red Hat
>
>> On 27 Nov 2015, at 09:26, Christian Beikov <christian at sweazer.com> wrote:
>>
>> Are you going to do something about this memory consumption or is there
>> at least some kind of minimum expected memory usage you can give me?
>> I ran into an OOMEs the other day and the cluster was unable to recover
>> from that by restarting single nodes. The nodes couldn't synchronize
>> because of the OOMEs. I had to (jgroups-)disconnect all nodes from the
>> cluster and start a separate cluster which of course lead to data loss.
>> All of this happened because of some wrong memory consumption
>> estimations I made so in order to avoid that in the future I would like
>> to plan better ahead. Is there any other way to avoid such a cluster death?
>>
>> Regards,
>> Christian
>>
>> Am 26.11.2015 um 15:56 schrieb Galder Zamarreño:
>>> Those IdentityIntMap are caches meant to speed up serialization if the same objects or types are marshalled again. It's normal for them to be populated as marshalling operations are executed. We don't currently have a way to clear these caches.
>>>
>>> Cheers,
>>> --
>>> Galder Zamarreño
>>> Infinispan, Red Hat
>>>
>>>> On 22 Nov 2015, at 15:00, Christian Beikov <christian at sweazer.com> wrote:
>>>>
>>>> Hello,
>>>>
>>>> In a recent heap dump analysis I found that
>>>> org.infinispan.marshall.core.JBossMarshaller consumes a lot of
>>>> memory(about 46 MB) that seems to be unused.
>>>> This is due to PerThreadInstanceHolder having ExtendedRiverMarshaller
>>>> objects that contain big IdentityIntMap objects. Some of those
>>>> IdentityIntMap instances have a size of 2 million entries, but most of
>>>> them have sizes of a few 100 thousands.
>>>> When I look into these IdentityIntMap instances, it seems that the
>>>> entries are all unused.
>>>>
>>>> Is that kind of memory consumption expected or does that indicate a
>>>> possibly wrong configuration?
>>>>
>>>> I am using Infinispan 7.2.4.Final on Wildfly 9.0.1.Final.
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev