[infinispan-dev] Testsuite: memory usage?

William Burns mudokonman at gmail.com
Thu Feb 15 08:32:46 EST 2018


So I must admit I had noticed a while back that I was having some issues
with running the core test suite. Unfortunately at the time CI and everyone
else seemed to not have any issues. I just ignored it because at the time I
didn't need to run core tests. But now that Sanne pointed this out, by
increasing the heap variable in the pom.xml, I was for the first time able
to run the test suite completely. It would normally hang for an extremely
long time near the 9k-10K test completed point and never finish for me (at
least I didn't wait long enough).

So it definitely seems there is something leaking in the test suite causing
the GC to use a ton of CPU time.

 - Will

On Thu, Feb 15, 2018 at 5:40 AM Sanne Grinovero <sanne at infinispan.org>
wrote:

> Thanks Dan.
>
> Do you happen to have observed the memory trend during a build?
>
> After a couple more attempts it passed the build once, so that shows
> it's possible to pass.. but even though it's a small sample so far
> that's 1 pass vs 3 OOMs on my machine.
>
> Even the one time it successfully completed the tests I see it wasted
> ~80% of total build time doing GC runs.. it was likely very close to
> fall over, and definitely not an efficient setting for regular builds.
> Observing trends on my machine I'd guess a reasonable value to be
> around 5GB to keep builds fast, or a minimum of 1.3 GB to be able to
> complete successfully without often failing.
>
> The memory issues are worse towards the end of the testsuite, and
> steadily growing.
>
> I won't be able to investigate further as I need to urgently work on
> modules, but I noticed there are quite some MBeans according to
> JConsole. I guess it would be good to check if we're not leaking the
> MBean registration, and therefore leaking (stopped?) CacheManagers
> from there?
>
> Even near the beginning of the tests, when forcing a full GC I see
> about 400MB being "not free". That's quite a lot for some simple
> tests, no?
>
> Thanks,
> Sanne
>
>
> On 15 February 2018 at 06:51, Dan Berindei <dan.berindei at gmail.com> wrote:
> > forkJvmArgs used to be "-Xmx2G" before ISPN-8478. I reduced the heap to
> 1G
> > because we were trying to run the build on agent VMs with only 4GB of
> RAM,
> > and the 2GB heap was making the build run out of native memory.
> >
> > I've yet to see an OOME in the core tests, locally or in CI. But I also
> > included -XX:+HeapDumpOnOutOfMemoryError in forkJvmArgs, so assuming
> there's
> > a new leak it should be easy to track down in the heap dump.
> >
> > Cheers
> > Dan
> >
> >
> > On Wed, Feb 14, 2018 at 11:46 PM, Sanne Grinovero <sanne at infinispan.org>
> > wrote:
> >>
> >> Hey all,
> >>
> >> I'm having OOMs running the tests of infinispan-core.
> >>
> >> Initially I thought it was related to limits and security as that's
> >> the usual suspect, but no it's really just not enough memory :)
> >>
> >> Found that the root pom.xml sets a <forkJvmArgs> property to Xmx1G for
> >> surefire; I've been observing the growth of heap usage in JConsole and
> >> it's clearly not enough.
> >>
> >> What surprises me is that - as an occasional tester - I shouldn't be
> >> the one to notice such a new requirement first. A leak which only
> >> manifests in certain conditions?
> >>
> >> What do others observe?
> >>
> >> FWIW, I'm running it with 8G heap now and it's working much better;
> >> still a couple of failures but at least they're not OOM related.
> >>
> >> Thanks,
> >> Sanne
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20180215/b2731226/attachment.html 


More information about the infinispan-dev mailing list