[infinispan-dev] Testsuite: memory usage?

[Sanne Grinovero]

Thanks Dan.

Do you happen to have observed the memory trend during a build?

After a couple more attempts it passed the build once, so that shows
it's possible to pass.. but even though it's a small sample so far
that's 1 pass vs 3 OOMs on my machine.

Even the one time it successfully completed the tests I see it wasted
~80% of total build time doing GC runs.. it was likely very close to
fall over, and definitely not an efficient setting for regular builds.
Observing trends on my machine I'd guess a reasonable value to be
around 5GB to keep builds fast, or a minimum of 1.3 GB to be able to
complete successfully without often failing.

The memory issues are worse towards the end of the testsuite, and
steadily growing.

I won't be able to investigate further as I need to urgently work on
modules, but I noticed there are quite some MBeans according to
JConsole. I guess it would be good to check if we're not leaking the
MBean registration, and therefore leaking (stopped?) CacheManagers
from there?

Even near the beginning of the tests, when forcing a full GC I see
about 400MB being "not free". That's quite a lot for some simple
tests, no?

Thanks,
Sanne


On 15 February 2018 at 06:51, Dan Berindei <dan.berindei at gmail.com> wrote:
> forkJvmArgs used to be "-Xmx2G" before ISPN-8478. I reduced the heap to 1G
> because we were trying to run the build on agent VMs with only 4GB of RAM,
> and the 2GB heap was making the build run out of native memory.
>
> I've yet to see an OOME in the core tests, locally or in CI. But I also
> included -XX:+HeapDumpOnOutOfMemoryError in forkJvmArgs, so assuming there's
> a new leak it should be easy to track down in the heap dump.
>
> Cheers
> Dan
>
>
> On Wed, Feb 14, 2018 at 11:46 PM, Sanne Grinovero <sanne at infinispan.org>
> wrote:
>>
>> Hey all,
>>
>> I'm having OOMs running the tests of infinispan-core.
>>
>> Initially I thought it was related to limits and security as that's
>> the usual suspect, but no it's really just not enough memory :)
>>
>> Found that the root pom.xml sets a <forkJvmArgs> property to Xmx1G for
>> surefire; I've been observing the growth of heap usage in JConsole and
>> it's clearly not enough.
>>
>> What surprises me is that - as an occasional tester - I shouldn't be
>> the one to notice such a new requirement first. A leak which only
>> manifests in certain conditions?
>>
>> What do others observe?
>>
>> FWIW, I'm running it with 8G heap now and it's working much better;
>> still a couple of failures but at least they're not OOM related.
>>
>> Thanks,
>> Sanne
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev