<div dir="ltr"><div>And here I was thinking that by adding -XX:+<wbr>HeapDumpOnOutOfMemoryError anyone would be able to look into OOMEs and I wouldn't have to reproduce the failures myself :)<br><br></div>Dan<br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 15, 2018 at 1:32 PM, William Burns <span dir="ltr"><<a href="mailto:mudokonman@gmail.com" target="_blank">mudokonman@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">So I must admit I had noticed a while back that I was having some issues with running the core test suite. Unfortunately at the time CI and everyone else seemed to not have any issues. I just ignored it because at the time I didn't need to run core tests. But now that Sanne pointed this out, by increasing the heap variable in the pom.xml, I was for the first time able to run the test suite completely. It would normally hang for an extremely long time near the 9k-10K test completed point and never finish for me (at least I didn't wait long enough).<div><br></div><div>So it definitely seems there is something leaking in the test suite causing the GC to use a ton of CPU time.</div><div><br></div><div> - Will</div></div><div class="HOEnZb"><div class="h5"><br><div class="gmail_quote"><div dir="ltr">On Thu, Feb 15, 2018 at 5:40 AM Sanne Grinovero <<a href="mailto:sanne@infinispan.org" target="_blank">sanne@infinispan.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thanks Dan.<br>
<br>
Do you happen to have observed the memory trend during a build?<br>
<br>
After a couple more attempts it passed the build once, so that shows<br>
it's possible to pass.. but even though it's a small sample so far<br>
that's 1 pass vs 3 OOMs on my machine.<br>
<br>
Even the one time it successfully completed the tests I see it wasted<br>
~80% of total build time doing GC runs.. it was likely very close to<br>
fall over, and definitely not an efficient setting for regular builds.<br>
Observing trends on my machine I'd guess a reasonable value to be<br>
around 5GB to keep builds fast, or a minimum of 1.3 GB to be able to<br>
complete successfully without often failing.<br>
<br>
The memory issues are worse towards the end of the testsuite, and<br>
steadily growing.<br>
<br>
I won't be able to investigate further as I need to urgently work on<br>
modules, but I noticed there are quite some MBeans according to<br>
JConsole. I guess it would be good to check if we're not leaking the<br>
MBean registration, and therefore leaking (stopped?) CacheManagers<br>
from there?<br>
<br>
Even near the beginning of the tests, when forcing a full GC I see<br>
about 400MB being "not free". That's quite a lot for some simple<br>
tests, no?<br>
<br>
Thanks,<br>
Sanne<br>
<br>
<br>
On 15 February 2018 at 06:51, Dan Berindei <<a href="mailto:dan.berindei@gmail.com" target="_blank">dan.berindei@gmail.com</a>> wrote:<br>
> forkJvmArgs used to be "-Xmx2G" before ISPN-8478. I reduced the heap to 1G<br>
> because we were trying to run the build on agent VMs with only 4GB of RAM,<br>
> and the 2GB heap was making the build run out of native memory.<br>
><br>
> I've yet to see an OOME in the core tests, locally or in CI. But I also<br>
> included -XX:+<wbr>HeapDumpOnOutOfMemoryError in forkJvmArgs, so assuming there's<br>
> a new leak it should be easy to track down in the heap dump.<br>
><br>
> Cheers<br>
> Dan<br>
><br>
><br>
> On Wed, Feb 14, 2018 at 11:46 PM, Sanne Grinovero <<a href="mailto:sanne@infinispan.org" target="_blank">sanne@infinispan.org</a>><br>
> wrote:<br>
>><br>
>> Hey all,<br>
>><br>
>> I'm having OOMs running the tests of infinispan-core.<br>
>><br>
>> Initially I thought it was related to limits and security as that's<br>
>> the usual suspect, but no it's really just not enough memory :)<br>
>><br>
>> Found that the root pom.xml sets a <forkJvmArgs> property to Xmx1G for<br>
>> surefire; I've been observing the growth of heap usage in JConsole and<br>
>> it's clearly not enough.<br>
>><br>
>> What surprises me is that - as an occasional tester - I shouldn't be<br>
>> the one to notice such a new requirement first. A leak which only<br>
>> manifests in certain conditions?<br>
>><br>
>> What do others observe?<br>
>><br>
>> FWIW, I'm running it with 8G heap now and it's working much better;<br>
>> still a couple of failures but at least they're not OOM related.<br>
>><br>
>> Thanks,<br>
>> Sanne<br>
>> ______________________________<wbr>_________________<br>
>> infinispan-dev mailing list<br>
>> <a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
>> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/infinispan-<wbr>dev</a><br>
><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> infinispan-dev mailing list<br>
> <a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/infinispan-<wbr>dev</a><br>
______________________________<wbr>_________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/infinispan-<wbr>dev</a><br>
</blockquote></div>
</div></div><br>______________________________<wbr>_________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/infinispan-<wbr>dev</a><br></blockquote></div><br></div>