Re: [infinispan-dev] Testsuite: memory usage?
Thanks Dan.
Do you happen to have observed the memory trend during a build?
After a couple more attempts it passed the build once, so that shows
it's possible to pass.. but even though it's a small sample so far
that's 1 pass vs 3 OOMs on my machine.
Even the one time it successfully completed the tests I see it wasted
~80% of total build time doing GC runs.. it was likely very close to
fall over, and definitely not an efficient setting for regular builds.
Observing trends on my machine I'd guess a reasonable value to be
around 5GB to keep builds fast, or a minimum of 1.3 GB to be able to
complete successfully without often failing.
The memory issues are worse towards the end of the testsuite, and
steadily growing.
I won't be able to investigate further as I need to urgently work on
modules, but I noticed there are quite some MBeans according to
JConsole. I guess it would be good to check if we're not leaking the
MBean registration, and therefore leaking (stopped?) CacheManagers
from there?
Even near the beginning of the tests, when forcing a full GC I see
about 400MB being "not free". That's quite a lot for some simple
tests, no?
Thanks,
Sanne
On 15 February 2018 at 06:51, Dan Berindei <dan.berindei(a)gmail.com> wrote:
forkJvmArgs used to be "-Xmx2G" before ISPN-8478. I reduced
the heap to 1G
because we were trying to run the build on agent VMs with only 4GB of RAM,
and the 2GB heap was making the build run out of native memory.
I've yet to see an OOME in the core tests, locally or in CI. But I also
included -XX:+HeapDumpOnOutOfMemoryError in forkJvmArgs, so assuming there's
a new leak it should be easy to track down in the heap dump.
Cheers
Dan
On Wed, Feb 14, 2018 at 11:46 PM, Sanne Grinovero <sanne(a)infinispan.org>
wrote:
>
> Hey all,
>
> I'm having OOMs running the tests of infinispan-core.
>
> Initially I thought it was related to limits and security as that's
> the usual suspect, but no it's really just not enough memory :)
>
> Found that the root pom.xml sets a <forkJvmArgs> property to Xmx1G for
> surefire; I've been observing the growth of heap usage in JConsole and
> it's clearly not enough.
>
> What surprises me is that - as an occasional tester - I shouldn't be
> the one to notice such a new requirement first. A leak which only
> manifests in certain conditions?
>
> What do others observe?
>
> FWIW, I'm running it with 8G heap now and it's working much better;
> still a couple of failures but at least they're not OOM related.
>
> Thanks,
> Sanne
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev