12:40 p.m.
Hi all,
after reviewing some pull requests, I'm since a couple of days unable
to run the testsuite; since Anna's fixes affect many modules I'm
trying to run the testsuite of the whole project, as we should always
do but I admit I haven't done it in a while because of the core module
failures.
So I run:
$ mvn -fn clean install
using -fn to have it continue after the core failures.
First attempt gave me an OOM, was running with 1G heap.. I'm pretty
sure this was good enough some months back.
Second attempt slowed down like crazy, and I found a warning about
having filled the code cache size, so doubled it to 200M.
Third attempt: OutOfMemoryError: PermGen space! But I'm running with
-XX:MaxPermSize=380M which should be plenty?
This is :
java version "1.6.0_43"
Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
-Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
-XX:ReservedCodeCacheSize=200M
-Dlog4j.configuration=file:/opt/infinispan-log4j.xml
My custom log configuration just disables trace & debug.
Going to try now with larger PermGen and different JVMs but it looks
quite bad.. any other suggestion?
(I do have the security limits setup properly)
Sanne
1:09 p.m.
I've just run it on master and didn't get OOM. well I'm using osx. Are you
running it on master or a particular branch? Which module crashes?
e.g. pedro's ISPN-2808 adds quite some threads to the party - that's the reason it
hasn't been integrated yet.
On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
Hi all,
after reviewing some pull requests, I'm since a couple of days unable
to run the testsuite; since Anna's fixes affect many modules I'm
trying to run the testsuite of the whole project, as we should always
do but I admit I haven't done it in a while because of the core module
failures.
So I run:
$ mvn -fn clean install
using -fn to have it continue after the core failures.
First attempt gave me an OOM, was running with 1G heap.. I'm pretty
sure this was good enough some months back.
Second attempt slowed down like crazy, and I found a warning about
having filled the code cache size, so doubled it to 200M.
Third attempt: OutOfMemoryError: PermGen space! But I'm running with
-XX:MaxPermSize=380M which should be plenty?
This is :
java version "1.6.0_43"
Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
-Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
-XX:ReservedCodeCacheSize=200M
-Dlog4j.configuration=file:/opt/infinispan-log4j.xml
My custom log configuration just disables trace & debug.
Going to try now with larger PermGen and different JVMs but it looks
quite bad.. any other suggestion?
(I do have the security limits setup properly)
Sanne
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
1:27 p.m.
I'm testing master, at da5c3f0
Just killed a run which was using
java version "1.7.0_17"
Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
this time again an OOM (while I have 2GB !), last sign of life came
from the "Rolling Upgrade Tooling"
I'm not going to merge/review any pull request until this works.
Sanne
On 20 March 2013 12:09, Mircea Markus <mmarkus(a)redhat.com> wrote:
I've just run it on master and didn't get OOM. well I'm
using osx. Are you running it on master or a particular branch? Which module crashes?
e.g. pedro's ISPN-2808 adds quite some threads to the party - that's the reason
it hasn't been integrated yet.
On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
> Hi all,
> after reviewing some pull requests, I'm since a couple of days unable
> to run the testsuite; since Anna's fixes affect many modules I'm
> trying to run the testsuite of the whole project, as we should always
> do but I admit I haven't done it in a while because of the core module
> failures.
>
> So I run:
> $ mvn -fn clean install
>
> using -fn to have it continue after the core failures.
>
> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
> sure this was good enough some months back.
>
> Second attempt slowed down like crazy, and I found a warning about
> having filled the code cache size, so doubled it to 200M.
>
> Third attempt: OutOfMemoryError: PermGen space! But I'm running with
> -XX:MaxPermSize=380M which should be plenty?
>
> This is :
> java version "1.6.0_43"
> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>
> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
> -XX:ReservedCodeCacheSize=200M
> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>
> My custom log configuration just disables trace & debug.
>
> Going to try now with larger PermGen and different JVMs but it looks
> quite bad.. any other suggestion?
> (I do have the security limits setup properly)
>
> Sanne
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
1:59 p.m.
Sanne, turn on CompressedOops ? Still those requirements are indeed
ridiculous.
Tristan
On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
I'm testing master, at da5c3f0
Just killed a run which was using
java version "1.7.0_17"
Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
this time again an OOM (while I have 2GB !), last sign of life came
from the "Rolling Upgrade Tooling"
I'm not going to merge/review any pull request until this works.
Sanne
On 20 March 2013 12:09, Mircea Markus <mmarkus(a)redhat.com> wrote:
> I've just run it on master and didn't get OOM. well I'm using osx. Are
you running it on master or a particular branch? Which module crashes?
> e.g. pedro's ISPN-2808 adds quite some threads to the party - that's the
reason it hasn't been integrated yet.
>
> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
>
>> Hi all,
>> after reviewing some pull requests, I'm since a couple of days unable
>> to run the testsuite; since Anna's fixes affect many modules I'm
>> trying to run the testsuite of the whole project, as we should always
>> do but I admit I haven't done it in a while because of the core module
>> failures.
>>
>> So I run:
>> $ mvn -fn clean install
>>
>> using -fn to have it continue after the core failures.
>>
>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
>> sure this was good enough some months back.
>>
>> Second attempt slowed down like crazy, and I found a warning about
>> having filled the code cache size, so doubled it to 200M.
>>
>> Third attempt: OutOfMemoryError: PermGen space! But I'm running with
>> -XX:MaxPermSize=380M which should be plenty?
>>
>> This is :
>> java version "1.6.0_43"
>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>>
>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>> -XX:ReservedCodeCacheSize=200M
>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>>
>> My custom log configuration just disables trace & debug.
>>
>> Going to try now with larger PermGen and different JVMs but it looks
>> quite bad.. any other suggestion?
>> (I do have the security limits setup properly)
>>
>> Sanne
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
4:12 p.m.
The problem is that we still leak threads in almost every module, and that
means we keep a copy of the core classes (and all their dependencies) for
every module. Of course, some modules' dependencies are already oversized,
so keeping only one copy is already too much...
I admit I don't run the whole test suite too often either, but I recently
changed the Cloudbees settings to get rid of the OOM there. It uses about
550MB of permgen by the end of the test suite, without
-XX:+UseCompressedOops. These are the settings I used:
-server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
-XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m
-XX:MaxPermSize=700M
Cheers
Dan
On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant <ttarrant(a)redhat.com>wrote:
Sanne, turn on CompressedOops ? Still those requirements are indeed
ridiculous.
Tristan
On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
> I'm testing master, at da5c3f0
>
> Just killed a run which was using
>
> java version "1.7.0_17"
> Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
>
> this time again an OOM (while I have 2GB !), last sign of life came
> from the "Rolling Upgrade Tooling"
>
> I'm not going to merge/review any pull request until this works.
>
> Sanne
>
> On 20 March 2013 12:09, Mircea Markus <mmarkus(a)redhat.com> wrote:
>> I've just run it on master and didn't get OOM. well I'm using osx.
Are
you running it on master or a particular branch? Which module crashes?
>> e.g. pedro's ISPN-2808 adds quite some threads to the party - that's
the reason it hasn't been integrated yet.
>>
>> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
>>
>>> Hi all,
>>> after reviewing some pull requests, I'm since a couple of days unable
>>> to run the testsuite; since Anna's fixes affect many modules I'm
>>> trying to run the testsuite of the whole project, as we should always
>>> do but I admit I haven't done it in a while because of the core module
>>> failures.
>>>
>>> So I run:
>>> $ mvn -fn clean install
>>>
>>> using -fn to have it continue after the core failures.
>>>
>>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
>>> sure this was good enough some months back.
>>>
>>> Second attempt slowed down like crazy, and I found a warning about
>>> having filled the code cache size, so doubled it to 200M.
>>>
>>> Third attempt: OutOfMemoryError: PermGen space! But I'm running with
>>> -XX:MaxPermSize=380M which should be plenty?
>>>
>>> This is :
>>> java version "1.6.0_43"
>>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>>>
>>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>>> -XX:ReservedCodeCacheSize=200M
>>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>>>
>>> My custom log configuration just disables trace & debug.
>>>
>>> Going to try now with larger PermGen and different JVMs but it looks
>>> quite bad.. any other suggestion?
>>> (I do have the security limits setup properly)
>>>
>>> Sanne
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> Cheers,
>> --
>> Mircea Markus
>> Infinispan lead (www.infinispan.org)
>>
>>
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
4:29 p.m.
I've also tried changing the fork mode of surefire from 'none' to
'once'
and the entire suite runs fine now on jvm 1.6 with 500mb MaxPermSize.
Previously I did not complete, 500mb was not enough.
Anyone knows why surefire was not allowed to fork?
Haven't tried to analyze closely the heap yet but first thing I noticed
is 15% of it is occupied by 190000 ComponentMetadataRepo instances,
which probably is not the root cause of this issue, but is odd anyway :).
On 03/20/2013 05:12 PM, Dan Berindei wrote:
The problem is that we still leak threads in almost every module, and
that means we keep a copy of the core classes (and all their
dependencies) for every module. Of course, some modules' dependencies
are already oversized, so keeping only one copy is already too much...
I admit I don't run the whole test suite too often either, but I
recently changed the Cloudbees settings to get rid of the OOM there.
It uses about 550MB of permgen by the end of the test suite, without
-XX:+UseCompressedOops. These are the settings I used:
-server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
-XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k
-Xms100m -Xmx900m -XX:MaxPermSize=700M
Cheers
Dan
On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant <ttarrant(a)redhat.com
<mailto:ttarrant@redhat.com>> wrote:
Sanne, turn on CompressedOops ? Still those requirements are indeed
ridiculous.
Tristan
On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
> I'm testing master, at da5c3f0
>
> Just killed a run which was using
>
> java version "1.7.0_17"
> Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
>
> this time again an OOM (while I have 2GB !), last sign of life came
> from the "Rolling Upgrade Tooling"
>
> I'm not going to merge/review any pull request until this works.
>
> Sanne
>
> On 20 March 2013 12:09, Mircea Markus <mmarkus(a)redhat.com
<mailto:mmarkus@redhat.com>> wrote:
>> I've just run it on master and didn't get OOM. well I'm using
osx. Are you running it on master or a particular branch? Which
module crashes?
>> e.g. pedro's ISPN-2808 adds quite some threads to the party -
that's the reason it hasn't been integrated yet.
>>
>> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
>>
>>> Hi all,
>>> after reviewing some pull requests, I'm since a couple of days
unable
>>> to run the testsuite; since Anna's fixes affect many modules
I'm
>>> trying to run the testsuite of the whole project, as we should
always
>>> do but I admit I haven't done it in a while because of the
core module
>>> failures.
>>>
>>> So I run:
>>> $ mvn -fn clean install
>>>
>>> using -fn to have it continue after the core failures.
>>>
>>> First attempt gave me an OOM, was running with 1G heap.. I'm
pretty
>>> sure this was good enough some months back.
>>>
>>> Second attempt slowed down like crazy, and I found a warning about
>>> having filled the code cache size, so doubled it to 200M.
>>>
>>> Third attempt: OutOfMemoryError: PermGen space! But I'm
running with
>>> -XX:MaxPermSize=380M which should be plenty?
>>>
>>> This is :
>>> java version "1.6.0_43"
>>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>>>
>>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>>> -XX:ReservedCodeCacheSize=200M
>>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>>>
>>> My custom log configuration just disables trace & debug.
>>>
>>> Going to try now with larger PermGen and different JVMs but it
looks
>>> quite bad.. any other suggestion?
>>> (I do have the security limits setup properly)
>>>
>>> Sanne
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> Cheers,
>> --
>> Mircea Markus
>> Infinispan lead (www.infinispan.org <http://www.infinispan.org>)
>>
>>
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
6:36 p.m.
On 20 Mar 2013, at 15:29, Adrian Nistor <anistor(a)redhat.com> wrote:
I've also tried changing the fork mode of surefire from
'none' to 'once' and the entire suite runs fine now on jvm 1.6 with 500mb
MaxPermSize. Previously I did not complete, 500mb was not enough.
Anyone knows why surefire was not allowed to fork?
Haven't tried to analyze closely the heap yet but first thing I noticed is 15% of it
is occupied by 190000 ComponentMetadataRepo instances, which probably is not the root
cause of this issue, but is odd anyway :).
Yes, very odd. Do you also see 190000 instances of a GlobalComponentRegistry?
On 03/20/2013 05:12 PM, Dan Berindei wrote:
> The problem is that we still leak threads in almost every module, and that means we
keep a copy of the core classes (and all their dependencies) for every module. Of course,
some modules' dependencies are already oversized, so keeping only one copy is already
too much...
>
> I admit I don't run the whole test suite too often either, but I recently changed
the Cloudbees settings to get rid of the OOM there. It uses about 550MB of permgen by the
end of the test suite, without -XX:+UseCompressedOops. These are the settings I used:
>
> -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
-XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m
-XX:MaxPermSize=700M
>
>
> Cheers
> Dan
>
>
>
> On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant <ttarrant(a)redhat.com> wrote:
> Sanne, turn on CompressedOops ? Still those requirements are indeed
> ridiculous.
>
> Tristan
>
> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
> > I'm testing master, at da5c3f0
> >
> > Just killed a run which was using
> >
> > java version "1.7.0_17"
> > Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
> > Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
> >
> > this time again an OOM (while I have 2GB !), last sign of life came
> > from the "Rolling Upgrade Tooling"
> >
> > I'm not going to merge/review any pull request until this works.
> >
> > Sanne
> >
> > On 20 March 2013 12:09, Mircea Markus <mmarkus(a)redhat.com> wrote:
> >> I've just run it on master and didn't get OOM. well I'm using
osx. Are you running it on master or a particular branch? Which module crashes?
> >> e.g. pedro's ISPN-2808 adds quite some threads to the party - that's
the reason it hasn't been integrated yet.
> >>
> >> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
> >>
> >>> Hi all,
> >>> after reviewing some pull requests, I'm since a couple of days
unable
> >>> to run the testsuite; since Anna's fixes affect many modules
I'm
> >>> trying to run the testsuite of the whole project, as we should always
> >>> do but I admit I haven't done it in a while because of the core
module
> >>> failures.
> >>>
> >>> So I run:
> >>> $ mvn -fn clean install
> >>>
> >>> using -fn to have it continue after the core failures.
> >>>
> >>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
> >>> sure this was good enough some months back.
> >>>
> >>> Second attempt slowed down like crazy, and I found a warning about
> >>> having filled the code cache size, so doubled it to 200M.
> >>>
> >>> Third attempt: OutOfMemoryError: PermGen space! But I'm running
with
> >>> -XX:MaxPermSize=380M which should be plenty?
> >>>
> >>> This is :
> >>> java version "1.6.0_43"
> >>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
> >>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
> >>>
> >>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
> >>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
> >>> -XX:ReservedCodeCacheSize=200M
> >>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
> >>>
> >>> My custom log configuration just disables trace & debug.
> >>>
> >>> Going to try now with larger PermGen and different JVMs but it looks
> >>> quite bad.. any other suggestion?
> >>> (I do have the security limits setup properly)
> >>>
> >>> Sanne
> >>> _______________________________________________
> >>> infinispan-dev mailing list
> >>> infinispan-dev(a)lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> Cheers,
> >> --
> >> Mircea Markus
> >> Infinispan lead (www.infinispan.org)
> >>
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev(a)lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Platform Architect, JBoss Data Grid
http://red.ht/data-grid
12:55 a.m.
Thanks Dan,
with the following options it completed the build:
MAVEN_OPTS=-server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
-XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k
-Xmx16G -Xms1G -XX:MaxPermSize=700M -XX:HeapDumpPath=/tmp/java_heap
-Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
-XX:ReservedCodeCacheSize=200M
-Dlog4j.configuration=file:/opt/infinispan-log4j.xml
Sanne
On 20 March 2013 17:36, Manik Surtani <msurtani(a)redhat.com> wrote:
On 20 Mar 2013, at 15:29, Adrian Nistor <anistor(a)redhat.com> wrote:
I've also tried changing the fork mode of surefire from 'none' to
'once' and
the entire suite runs fine now on jvm 1.6 with 500mb MaxPermSize.
Previously I did not complete, 500mb was not enough.
Anyone knows why surefire was not allowed to fork?
Haven't tried to analyze closely the heap yet but first thing I noticed is
15% of it is occupied by 190000 ComponentMetadataRepo instances, which
probably is not the root cause of this issue, but is odd anyway :).
Yes, very odd. Do you also see 190000 instances of a
GlobalComponentRegistry?
On 03/20/2013 05:12 PM, Dan Berindei wrote:
The problem is that we still leak threads in almost every module, and that
means we keep a copy of the core classes (and all their dependencies) for
every module. Of course, some modules' dependencies are already oversized,
so keeping only one copy is already too much...
I admit I don't run the whole test suite too often either, but I recently
changed the Cloudbees settings to get rid of the OOM there. It uses about
550MB of permgen by the end of the test suite, without
-XX:+UseCompressedOops. These are the settings I used:
-server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
-XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m
-XX:MaxPermSize=700M
Cheers
Dan
On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant <ttarrant(a)redhat.com>
wrote:
>
> Sanne, turn on CompressedOops ? Still those requirements are indeed
> ridiculous.
>
> Tristan
>
> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
> > I'm testing master, at da5c3f0
> >
> > Just killed a run which was using
> >
> > java version "1.7.0_17"
> > Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
> > Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
> >
> > this time again an OOM (while I have 2GB !), last sign of life came
> > from the "Rolling Upgrade Tooling"
> >
> > I'm not going to merge/review any pull request until this works.
> >
> > Sanne
> >
> > On 20 March 2013 12:09, Mircea Markus <mmarkus(a)redhat.com> wrote:
> >> I've just run it on master and didn't get OOM. well I'm using
osx. Are
> >> you running it on master or a particular branch? Which module crashes?
> >> e.g. pedro's ISPN-2808 adds quite some threads to the party -
that's
> >> the reason it hasn't been integrated yet.
> >>
> >> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
> >>
> >>> Hi all,
> >>> after reviewing some pull requests, I'm since a couple of days
unable
> >>> to run the testsuite; since Anna's fixes affect many modules
I'm
> >>> trying to run the testsuite of the whole project, as we should always
> >>> do but I admit I haven't done it in a while because of the core
module
> >>> failures.
> >>>
> >>> So I run:
> >>> $ mvn -fn clean install
> >>>
> >>> using -fn to have it continue after the core failures.
> >>>
> >>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
> >>> sure this was good enough some months back.
> >>>
> >>> Second attempt slowed down like crazy, and I found a warning about
> >>> having filled the code cache size, so doubled it to 200M.
> >>>
> >>> Third attempt: OutOfMemoryError: PermGen space! But I'm running
with
> >>> -XX:MaxPermSize=380M which should be plenty?
> >>>
> >>> This is :
> >>> java version "1.6.0_43"
> >>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
> >>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
> >>>
> >>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
> >>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
> >>> -XX:ReservedCodeCacheSize=200M
> >>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
> >>>
> >>> My custom log configuration just disables trace & debug.
> >>>
> >>> Going to try now with larger PermGen and different JVMs but it looks
> >>> quite bad.. any other suggestion?
> >>> (I do have the security limits setup properly)
> >>>
> >>> Sanne
> >>> _______________________________________________
> >>> infinispan-dev mailing list
> >>> infinispan-dev(a)lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> Cheers,
> >> --
> >> Mircea Markus
> >> Infinispan lead (www.infinispan.org)
> >>
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev(a)lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Platform Architect, JBoss Data Grid
http://red.ht/data-grid
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
9:21 a.m.
Sanne, you Xmx setting seems a bit excessive :)
Did you see it fail on your machine with 1GB regular heap and 700MB permgen?
On Thu, Mar 21, 2013 at 1:55 AM, Sanne Grinovero <sanne(a)infinispan.org>wrote:
Thanks Dan,
with the following options it completed the build:
MAVEN_OPTS=-server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
-XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k
-Xmx16G -Xms1G -XX:MaxPermSize=700M -XX:HeapDumpPath=/tmp/java_heap
-Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
-XX:ReservedCodeCacheSize=200M
-Dlog4j.configuration=file:/opt/infinispan-log4j.xml
Sanne
On 20 March 2013 17:36, Manik Surtani <msurtani(a)redhat.com> wrote:
>
> On 20 Mar 2013, at 15:29, Adrian Nistor <anistor(a)redhat.com> wrote:
>
> I've also tried changing the fork mode of surefire from 'none' to
'once'
and
> the entire suite runs fine now on jvm 1.6 with 500mb MaxPermSize.
> Previously I did not complete, 500mb was not enough.
> Anyone knows why surefire was not allowed to fork?
>
> Haven't tried to analyze closely the heap yet but first thing I noticed
is
> 15% of it is occupied by 190000 ComponentMetadataRepo instances, which
> probably is not the root cause of this issue, but is odd anyway :).
>
>
> Yes, very odd. Do you also see 190000 instances of a
> GlobalComponentRegistry?
>
>
> On 03/20/2013 05:12 PM, Dan Berindei wrote:
>
> The problem is that we still leak threads in almost every module, and
that
> means we keep a copy of the core classes (and all their dependencies) for
> every module. Of course, some modules' dependencies are already
oversized,
> so keeping only one copy is already too much...
>
> I admit I don't run the whole test suite too often either, but I recently
> changed the Cloudbees settings to get rid of the OOM there. It uses about
> 550MB of permgen by the end of the test suite, without
> -XX:+UseCompressedOops. These are the settings I used:
>
> -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
> -XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m
> -XX:MaxPermSize=700M
>
>
> Cheers
> Dan
>
>
>
> On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant <ttarrant(a)redhat.com>
> wrote:
>>
>> Sanne, turn on CompressedOops ? Still those requirements are indeed
>> ridiculous.
>>
>> Tristan
>>
>> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
>> > I'm testing master, at da5c3f0
>> >
>> > Just killed a run which was using
>> >
>> > java version "1.7.0_17"
>> > Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
>> > Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
>> >
>> > this time again an OOM (while I have 2GB !), last sign of life came
>> > from the "Rolling Upgrade Tooling"
>> >
>> > I'm not going to merge/review any pull request until this works.
>> >
>> > Sanne
>> >
>> > On 20 March 2013 12:09, Mircea Markus <mmarkus(a)redhat.com> wrote:
>> >> I've just run it on master and didn't get OOM. well I'm
using osx.
Are
>> >> you running it on master or a particular branch? Which module
crashes?
>> >> e.g. pedro's ISPN-2808 adds quite some threads to the party -
that's
>> >> the reason it hasn't been integrated yet.
>> >>
>> >> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
>> >>
>> >>> Hi all,
>> >>> after reviewing some pull requests, I'm since a couple of days
unable
>> >>> to run the testsuite; since Anna's fixes affect many modules
I'm
>> >>> trying to run the testsuite of the whole project, as we should
always
>> >>> do but I admit I haven't done it in a while because of the
core
module
>> >>> failures.
>> >>>
>> >>> So I run:
>> >>> $ mvn -fn clean install
>> >>>
>> >>> using -fn to have it continue after the core failures.
>> >>>
>> >>> First attempt gave me an OOM, was running with 1G heap.. I'm
pretty
>> >>> sure this was good enough some months back.
>> >>>
>> >>> Second attempt slowed down like crazy, and I found a warning about
>> >>> having filled the code cache size, so doubled it to 200M.
>> >>>
>> >>> Third attempt: OutOfMemoryError: PermGen space! But I'm running
with
>> >>> -XX:MaxPermSize=380M which should be plenty?
>> >>>
>> >>> This is :
>> >>> java version "1.6.0_43"
>> >>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>> >>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>> >>>
>> >>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>> >>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>> >>> -XX:ReservedCodeCacheSize=200M
>> >>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>> >>>
>> >>> My custom log configuration just disables trace & debug.
>> >>>
>> >>> Going to try now with larger PermGen and different JVMs but it
looks
>> >>> quite bad.. any other suggestion?
>> >>> (I do have the security limits setup properly)
>> >>>
>> >>> Sanne
>> >>> _______________________________________________
>> >>> infinispan-dev mailing list
>> >>> infinispan-dev(a)lists.jboss.org
>> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >> Cheers,
>> >> --
>> >> Mircea Markus
>> >> Infinispan lead (www.infinispan.org)
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> infinispan-dev mailing list
>> >> infinispan-dev(a)lists.jboss.org
>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> > _______________________________________________
>> > infinispan-dev mailing list
>> > infinispan-dev(a)lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Manik Surtani
> manik(a)jboss.org
> twitter.com/maniksurtani
>
> Platform Architect, JBoss Data Grid
> http://red.ht/data-grid
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
11:43 a.m.
Now the suite is forks a new process for each module. I've complete the run with 128m
MaxPerm.
On 20 Mar 2013, at 23:55, Sanne Grinovero wrote:
Thanks Dan,
with the following options it completed the build:
MAVEN_OPTS=-server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
-XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k
-Xmx16G -Xms1G -XX:MaxPermSize=700M -XX:HeapDumpPath=/tmp/java_heap
-Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
-XX:ReservedCodeCacheSize=200M
-Dlog4j.configuration=file:/opt/infinispan-log4j.xml
Sanne
On 20 March 2013 17:36, Manik Surtani <msurtani(a)redhat.com> wrote:
>
> On 20 Mar 2013, at 15:29, Adrian Nistor <anistor(a)redhat.com> wrote:
>
> I've also tried changing the fork mode of surefire from 'none' to
'once' and
> the entire suite runs fine now on jvm 1.6 with 500mb MaxPermSize.
> Previously I did not complete, 500mb was not enough.
> Anyone knows why surefire was not allowed to fork?
>
> Haven't tried to analyze closely the heap yet but first thing I noticed is
> 15% of it is occupied by 190000 ComponentMetadataRepo instances, which
> probably is not the root cause of this issue, but is odd anyway :).
>
>
> Yes, very odd. Do you also see 190000 instances of a
> GlobalComponentRegistry?
>
>
> On 03/20/2013 05:12 PM, Dan Berindei wrote:
>
> The problem is that we still leak threads in almost every module, and that
> means we keep a copy of the core classes (and all their dependencies) for
> every module. Of course, some modules' dependencies are already oversized,
> so keeping only one copy is already too much...
>
> I admit I don't run the whole test suite too often either, but I recently
> changed the Cloudbees settings to get rid of the OOM there. It uses about
> 550MB of permgen by the end of the test suite, without
> -XX:+UseCompressedOops. These are the settings I used:
>
> -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
> -XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m
> -XX:MaxPermSize=700M
>
>
> Cheers
> Dan
>
>
>
> On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant <ttarrant(a)redhat.com>
> wrote:
>>
>> Sanne, turn on CompressedOops ? Still those requirements are indeed
>> ridiculous.
>>
>> Tristan
>>
>> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
>>> I'm testing master, at da5c3f0
>>>
>>> Just killed a run which was using
>>>
>>> java version "1.7.0_17"
>>> Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
>>> Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
>>>
>>> this time again an OOM (while I have 2GB !), last sign of life came
>>> from the "Rolling Upgrade Tooling"
>>>
>>> I'm not going to merge/review any pull request until this works.
>>>
>>> Sanne
>>>
>>> On 20 March 2013 12:09, Mircea Markus <mmarkus(a)redhat.com> wrote:
>>>> I've just run it on master and didn't get OOM. well I'm using
osx. Are
>>>> you running it on master or a particular branch? Which module crashes?
>>>> e.g. pedro's ISPN-2808 adds quite some threads to the party -
that's
>>>> the reason it hasn't been integrated yet.
>>>>
>>>> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
>>>>
>>>>> Hi all,
>>>>> after reviewing some pull requests, I'm since a couple of days
unable
>>>>> to run the testsuite; since Anna's fixes affect many modules
I'm
>>>>> trying to run the testsuite of the whole project, as we should
always
>>>>> do but I admit I haven't done it in a while because of the core
module
>>>>> failures.
>>>>>
>>>>> So I run:
>>>>> $ mvn -fn clean install
>>>>>
>>>>> using -fn to have it continue after the core failures.
>>>>>
>>>>> First attempt gave me an OOM, was running with 1G heap.. I'm
pretty
>>>>> sure this was good enough some months back.
>>>>>
>>>>> Second attempt slowed down like crazy, and I found a warning about
>>>>> having filled the code cache size, so doubled it to 200M.
>>>>>
>>>>> Third attempt: OutOfMemoryError: PermGen space! But I'm running
with
>>>>> -XX:MaxPermSize=380M which should be plenty?
>>>>>
>>>>> This is :
>>>>> java version "1.6.0_43"
>>>>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>>>>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>>>>>
>>>>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>>>>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>>>>> -XX:ReservedCodeCacheSize=200M
>>>>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>>>>>
>>>>> My custom log configuration just disables trace & debug.
>>>>>
>>>>> Going to try now with larger PermGen and different JVMs but it looks
>>>>> quite bad.. any other suggestion?
>>>>> (I do have the security limits setup properly)
>>>>>
>>>>> Sanne
>>>>> _______________________________________________
>>>>> infinispan-dev mailing list
>>>>> infinispan-dev(a)lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>> Cheers,
>>>> --
>>>> Mircea Markus
>>>> Infinispan lead (www.infinispan.org)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev(a)lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Manik Surtani
> manik(a)jboss.org
> twitter.com/maniksurtani
>
> Platform Architect, JBoss Data Grid
> http://red.ht/data-grid
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
8:10 p.m.
Nice :)
On Apr 18, 2013, at 6:43 AM, Mircea Markus <mmarkus(a)redhat.com> wrote:
Now the suite is forks a new process for each module. I've
complete the run with 128m MaxPerm.
On 20 Mar 2013, at 23:55, Sanne Grinovero wrote:
> Thanks Dan,
> with the following options it completed the build:
>
> MAVEN_OPTS=-server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
> -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k
> -Xmx16G -Xms1G -XX:MaxPermSize=700M -XX:HeapDumpPath=/tmp/java_heap
> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
> -XX:ReservedCodeCacheSize=200M
> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>
> Sanne
>
> On 20 March 2013 17:36, Manik Surtani <msurtani(a)redhat.com> wrote:
>>
>> On 20 Mar 2013, at 15:29, Adrian Nistor <anistor(a)redhat.com> wrote:
>>
>> I've also tried changing the fork mode of surefire from 'none' to
'once' and
>> the entire suite runs fine now on jvm 1.6 with 500mb MaxPermSize.
>> Previously I did not complete, 500mb was not enough.
>> Anyone knows why surefire was not allowed to fork?
>>
>> Haven't tried to analyze closely the heap yet but first thing I noticed is
>> 15% of it is occupied by 190000 ComponentMetadataRepo instances, which
>> probably is not the root cause of this issue, but is odd anyway :).
>>
>>
>> Yes, very odd. Do you also see 190000 instances of a
>> GlobalComponentRegistry?
>>
>>
>> On 03/20/2013 05:12 PM, Dan Berindei wrote:
>>
>> The problem is that we still leak threads in almost every module, and that
>> means we keep a copy of the core classes (and all their dependencies) for
>> every module. Of course, some modules' dependencies are already oversized,
>> so keeping only one copy is already too much...
>>
>> I admit I don't run the whole test suite too often either, but I recently
>> changed the Cloudbees settings to get rid of the OOM there. It uses about
>> 550MB of permgen by the end of the test suite, without
>> -XX:+UseCompressedOops. These are the settings I used:
>>
>> -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
>> -XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m
>> -XX:MaxPermSize=700M
>>
>>
>> Cheers
>> Dan
>>
>>
>>
>> On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant <ttarrant(a)redhat.com>
>> wrote:
>>>
>>> Sanne, turn on CompressedOops ? Still those requirements are indeed
>>> ridiculous.
>>>
>>> Tristan
>>>
>>> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
>>>> I'm testing master, at da5c3f0
>>>>
>>>> Just killed a run which was using
>>>>
>>>> java version "1.7.0_17"
>>>> Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
>>>> Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
>>>>
>>>> this time again an OOM (while I have 2GB !), last sign of life came
>>>> from the "Rolling Upgrade Tooling"
>>>>
>>>> I'm not going to merge/review any pull request until this works.
>>>>
>>>> Sanne
>>>>
>>>> On 20 March 2013 12:09, Mircea Markus <mmarkus(a)redhat.com> wrote:
>>>>> I've just run it on master and didn't get OOM. well I'm
using osx. Are
>>>>> you running it on master or a particular branch? Which module
crashes?
>>>>> e.g. pedro's ISPN-2808 adds quite some threads to the party -
that's
>>>>> the reason it hasn't been integrated yet.
>>>>>
>>>>> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
>>>>>
>>>>>> Hi all,
>>>>>> after reviewing some pull requests, I'm since a couple of
days unable
>>>>>> to run the testsuite; since Anna's fixes affect many modules
I'm
>>>>>> trying to run the testsuite of the whole project, as we should
always
>>>>>> do but I admit I haven't done it in a while because of the
core module
>>>>>> failures.
>>>>>>
>>>>>> So I run:
>>>>>> $ mvn -fn clean install
>>>>>>
>>>>>> using -fn to have it continue after the core failures.
>>>>>>
>>>>>> First attempt gave me an OOM, was running with 1G heap.. I'm
pretty
>>>>>> sure this was good enough some months back.
>>>>>>
>>>>>> Second attempt slowed down like crazy, and I found a warning
about
>>>>>> having filled the code cache size, so doubled it to 200M.
>>>>>>
>>>>>> Third attempt: OutOfMemoryError: PermGen space! But I'm
running with
>>>>>> -XX:MaxPermSize=380M which should be plenty?
>>>>>>
>>>>>> This is :
>>>>>> java version "1.6.0_43"
>>>>>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>>>>>>
>>>>>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>>>>>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>>>>>> -XX:ReservedCodeCacheSize=200M
>>>>>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>>>>>>
>>>>>> My custom log configuration just disables trace & debug.
>>>>>>
>>>>>> Going to try now with larger PermGen and different JVMs but it
looks
>>>>>> quite bad.. any other suggestion?
>>>>>> (I do have the security limits setup properly)
>>>>>>
>>>>>> Sanne
>>>>>> _______________________________________________
>>>>>> infinispan-dev mailing list
>>>>>> infinispan-dev(a)lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>> Cheers,
>>>>> --
>>>>> Mircea Markus
>>>>> Infinispan lead (www.infinispan.org)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> infinispan-dev mailing list
>>>>> infinispan-dev(a)lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev(a)lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>> --
>> Manik Surtani
>> manik(a)jboss.org
>> twitter.com/maniksurtani
>>
>> Platform Architect, JBoss Data Grid
>> http://red.ht/data-grid
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
galder(a)redhat.com
twitter.com/galderz
Project Lead, Escalante
http://escalante.io
Engineer, Infinispan
http://infinispan.org
12:35 p.m.
On Mar 20, 2013, at 3:29 PM, Adrian Nistor <anistor(a)redhat.com> wrote:
I've also tried changing the fork mode of surefire from
'none' to 'once' and the entire suite runs fine now on jvm 1.6 with 500mb
MaxPermSize. Previously I did not complete, 500mb was not enough.
Anyone knows why surefire was not allowed to fork?
Haven't tried to analyze closely the heap yet but first thing I noticed is 15% of it
is occupied by 190000 ComponentMetadataRepo instances, which probably is not the root
cause of this issue, but is odd anyway :).
If you have a heap dump, can you put it somewhere online so that we can inspect it?
Eclipse MAT is great for that...
On 03/20/2013 05:12 PM, Dan Berindei wrote:
> The problem is that we still leak threads in almost every module, and that means we
keep a copy of the core classes (and all their dependencies) for every module. Of course,
some modules' dependencies are already oversized, so keeping only one copy is already
too much...
>
> I admit I don't run the whole test suite too often either, but I recently changed
the Cloudbees settings to get rid of the OOM there. It uses about 550MB of permgen by the
end of the test suite, without -XX:+UseCompressedOops. These are the settings I used:
>
> -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
-XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m
-XX:MaxPermSize=700M
>
>
> Cheers
> Dan
>
>
>
> On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant <ttarrant(a)redhat.com> wrote:
> Sanne, turn on CompressedOops ? Still those requirements are indeed
> ridiculous.
>
> Tristan
>
> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
> > I'm testing master, at da5c3f0
> >
> > Just killed a run which was using
> >
> > java version "1.7.0_17"
> > Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
> > Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
> >
> > this time again an OOM (while I have 2GB !), last sign of life came
> > from the "Rolling Upgrade Tooling"
> >
> > I'm not going to merge/review any pull request until this works.
> >
> > Sanne
> >
> > On 20 March 2013 12:09, Mircea Markus <mmarkus(a)redhat.com> wrote:
> >> I've just run it on master and didn't get OOM. well I'm using
osx. Are you running it on master or a particular branch? Which module crashes?
> >> e.g. pedro's ISPN-2808 adds quite some threads to the party - that's
the reason it hasn't been integrated yet.
> >>
> >> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
> >>
> >>> Hi all,
> >>> after reviewing some pull requests, I'm since a couple of days
unable
> >>> to run the testsuite; since Anna's fixes affect many modules
I'm
> >>> trying to run the testsuite of the whole project, as we should always
> >>> do but I admit I haven't done it in a while because of the core
module
> >>> failures.
> >>>
> >>> So I run:
> >>> $ mvn -fn clean install
> >>>
> >>> using -fn to have it continue after the core failures.
> >>>
> >>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
> >>> sure this was good enough some months back.
> >>>
> >>> Second attempt slowed down like crazy, and I found a warning about
> >>> having filled the code cache size, so doubled it to 200M.
> >>>
> >>> Third attempt: OutOfMemoryError: PermGen space! But I'm running
with
> >>> -XX:MaxPermSize=380M which should be plenty?
> >>>
> >>> This is :
> >>> java version "1.6.0_43"
> >>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
> >>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
> >>>
> >>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
> >>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
> >>> -XX:ReservedCodeCacheSize=200M
> >>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
> >>>
> >>> My custom log configuration just disables trace & debug.
> >>>
> >>> Going to try now with larger PermGen and different JVMs but it looks
> >>> quite bad.. any other suggestion?
> >>> (I do have the security limits setup properly)
> >>>
> >>> Sanne
> >>> _______________________________________________
> >>> infinispan-dev mailing list
> >>> infinispan-dev(a)lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> Cheers,
> >> --
> >> Mircea Markus
> >> Infinispan lead (www.infinispan.org)
> >>
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev(a)lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
>
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
galder(a)redhat.com
twitter.com/galderz
Project Lead, Escalante
http://escalante.io
Engineer, Infinispan
http://infinispan.org
3:47 p.m.
A correction is needed. I realized later that those 19000 instances were
actually ComponentMetadata not ComponentMetadataRepo!
There were a few hundred ComponentMetadataRepo instances and the same
number of GlobalComponentRegistry - nothing very unusual. I counted them
using MAT but I lost the hprof files.
On 03/21/2013 01:35 PM, Galder Zamarreño wrote:
On Mar 20, 2013, at 3:29 PM, Adrian Nistor <anistor(a)redhat.com>
wrote:
> I've also tried changing the fork mode of surefire from 'none' to
'once' and the entire suite runs fine now on jvm 1.6 with 500mb MaxPermSize.
Previously I did not complete, 500mb was not enough.
> Anyone knows why surefire was not allowed to fork?
>
> Haven't tried to analyze closely the heap yet but first thing I noticed is 15% of
it is occupied by 190000 ComponentMetadataRepo instances, which probably is not the root
cause of this issue, but is odd anyway :).
If you have a heap dump, can you put it somewhere online so that we can inspect it?
Eclipse MAT is great for that...
> On 03/20/2013 05:12 PM, Dan Berindei wrote:
>> The problem is that we still leak threads in almost every module, and that means
we keep a copy of the core classes (and all their dependencies) for every module. Of
course, some modules' dependencies are already oversized, so keeping only one copy is
already too much...
>>
>> I admit I don't run the whole test suite too often either, but I recently
changed the Cloudbees settings to get rid of the OOM there. It uses about 550MB of permgen
by the end of the test suite, without -XX:+UseCompressedOops. These are the settings I
used:
>>
>> -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
-XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m
-XX:MaxPermSize=700M
>>
>>
>> Cheers
>> Dan
>>
>>
>>
>> On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant <ttarrant(a)redhat.com>
wrote:
>> Sanne, turn on CompressedOops ? Still those requirements are indeed
>> ridiculous.
>>
>> Tristan
>>
>> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
>>> I'm testing master, at da5c3f0
>>>
>>> Just killed a run which was using
>>>
>>> java version "1.7.0_17"
>>> Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
>>> Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
>>>
>>> this time again an OOM (while I have 2GB !), last sign of life came
>>> from the "Rolling Upgrade Tooling"
>>>
>>> I'm not going to merge/review any pull request until this works.
>>>
>>> Sanne
>>>
>>> On 20 March 2013 12:09, Mircea Markus <mmarkus(a)redhat.com> wrote:
>>>> I've just run it on master and didn't get OOM. well I'm using
osx. Are you running it on master or a particular branch? Which module crashes?
>>>> e.g. pedro's ISPN-2808 adds quite some threads to the party -
that's the reason it hasn't been integrated yet.
>>>>
>>>> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
>>>>
>>>>> Hi all,
>>>>> after reviewing some pull requests, I'm since a couple of days
unable
>>>>> to run the testsuite; since Anna's fixes affect many modules
I'm
>>>>> trying to run the testsuite of the whole project, as we should
always
>>>>> do but I admit I haven't done it in a while because of the core
module
>>>>> failures.
>>>>>
>>>>> So I run:
>>>>> $ mvn -fn clean install
>>>>>
>>>>> using -fn to have it continue after the core failures.
>>>>>
>>>>> First attempt gave me an OOM, was running with 1G heap.. I'm
pretty
>>>>> sure this was good enough some months back.
>>>>>
>>>>> Second attempt slowed down like crazy, and I found a warning about
>>>>> having filled the code cache size, so doubled it to 200M.
>>>>>
>>>>> Third attempt: OutOfMemoryError: PermGen space! But I'm running
with
>>>>> -XX:MaxPermSize=380M which should be plenty?
>>>>>
>>>>> This is :
>>>>> java version "1.6.0_43"
>>>>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>>>>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>>>>>
>>>>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>>>>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>>>>> -XX:ReservedCodeCacheSize=200M
>>>>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>>>>>
>>>>> My custom log configuration just disables trace & debug.
>>>>>
>>>>> Going to try now with larger PermGen and different JVMs but it looks
>>>>> quite bad.. any other suggestion?
>>>>> (I do have the security limits setup properly)
>>>>>
>>>>> Sanne
>>>>> _______________________________________________
>>>>> infinispan-dev mailing list
>>>>> infinispan-dev(a)lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>> Cheers,
>>>> --
>>>> Mircea Markus
>>>> Infinispan lead (www.infinispan.org)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev(a)lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>>
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
galder(a)redhat.com
twitter.com/galderz
Project Lead, Escalante
http://escalante.io
Engineer, Infinispan
http://infinispan.org
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
6:31 p.m.
If anyone wants to analyze the heap dump, I've recreated it:
http://dl.dropbox.com/u/29415206/java_pid27113.zip
On 03/25/2013 04:47 PM, Adrian Nistor wrote:
A correction is needed. I realized later that those 19000 instances
were
actually ComponentMetadata not ComponentMetadataRepo!
There were a few hundred ComponentMetadataRepo instances and the same
number of GlobalComponentRegistry - nothing very unusual. I counted them
using MAT but I lost the hprof files.
On 03/21/2013 01:35 PM, Galder Zamarreño wrote:
> On Mar 20, 2013, at 3:29 PM, Adrian Nistor <anistor(a)redhat.com> wrote:
>
>> I've also tried changing the fork mode of surefire from 'none' to
'once' and the entire suite runs fine now on jvm 1.6 with 500mb MaxPermSize.
Previously I did not complete, 500mb was not enough.
>> Anyone knows why surefire was not allowed to fork?
>>
>> Haven't tried to analyze closely the heap yet but first thing I noticed is
15% of it is occupied by 190000 ComponentMetadataRepo instances, which probably is not the
root cause of this issue, but is odd anyway :).
> If you have a heap dump, can you put it somewhere online so that we can inspect it?
Eclipse MAT is great for that...
>
>> On 03/20/2013 05:12 PM, Dan Berindei wrote:
>>> The problem is that we still leak threads in almost every module, and that
means we keep a copy of the core classes (and all their dependencies) for every module. Of
course, some modules' dependencies are already oversized, so keeping only one copy is
already too much...
>>>
>>> I admit I don't run the whole test suite too often either, but I recently
changed the Cloudbees settings to get rid of the OOM there. It uses about 550MB of permgen
by the end of the test suite, without -XX:+UseCompressedOops. These are the settings I
used:
>>>
>>> -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
-XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m
-XX:MaxPermSize=700M
>>>
>>>
>>> Cheers
>>> Dan
>>>
>>>
>>>
>>> On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant <ttarrant(a)redhat.com>
wrote:
>>> Sanne, turn on CompressedOops ? Still those requirements are indeed
>>> ridiculous.
>>>
>>> Tristan
>>>
>>> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
>>>> I'm testing master, at da5c3f0
>>>>
>>>> Just killed a run which was using
>>>>
>>>> java version "1.7.0_17"
>>>> Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
>>>> Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
>>>>
>>>> this time again an OOM (while I have 2GB !), last sign of life came
>>>> from the "Rolling Upgrade Tooling"
>>>>
>>>> I'm not going to merge/review any pull request until this works.
>>>>
>>>> Sanne
>>>>
>>>> On 20 March 2013 12:09, Mircea Markus <mmarkus(a)redhat.com> wrote:
>>>>> I've just run it on master and didn't get OOM. well I'm
using osx. Are you running it on master or a particular branch? Which module crashes?
>>>>> e.g. pedro's ISPN-2808 adds quite some threads to the party -
that's the reason it hasn't been integrated yet.
>>>>>
>>>>> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
>>>>>
>>>>>> Hi all,
>>>>>> after reviewing some pull requests, I'm since a couple of
days unable
>>>>>> to run the testsuite; since Anna's fixes affect many modules
I'm
>>>>>> trying to run the testsuite of the whole project, as we should
always
>>>>>> do but I admit I haven't done it in a while because of the
core module
>>>>>> failures.
>>>>>>
>>>>>> So I run:
>>>>>> $ mvn -fn clean install
>>>>>>
>>>>>> using -fn to have it continue after the core failures.
>>>>>>
>>>>>> First attempt gave me an OOM, was running with 1G heap.. I'm
pretty
>>>>>> sure this was good enough some months back.
>>>>>>
>>>>>> Second attempt slowed down like crazy, and I found a warning
about
>>>>>> having filled the code cache size, so doubled it to 200M.
>>>>>>
>>>>>> Third attempt: OutOfMemoryError: PermGen space! But I'm
running with
>>>>>> -XX:MaxPermSize=380M which should be plenty?
>>>>>>
>>>>>> This is :
>>>>>> java version "1.6.0_43"
>>>>>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>>>>>>
>>>>>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>>>>>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>>>>>> -XX:ReservedCodeCacheSize=200M
>>>>>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>>>>>>
>>>>>> My custom log configuration just disables trace & debug.
>>>>>>
>>>>>> Going to try now with larger PermGen and different JVMs but it
looks
>>>>>> quite bad.. any other suggestion?
>>>>>> (I do have the security limits setup properly)
>>>>>>
>>>>>> Sanne
>>>>>> _______________________________________________
>>>>>> infinispan-dev mailing list
>>>>>> infinispan-dev(a)lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>> Cheers,
>>>>> --
>>>>> Mircea Markus
>>>>> Infinispan lead (www.infinispan.org)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> infinispan-dev mailing list
>>>>> infinispan-dev(a)lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev(a)lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>>
>>> infinispan-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> --
> Galder Zamarreño
> galder(a)redhat.com
> twitter.com/galderz
>
> Project Lead, Escalante
> http://escalante.io
>
> Engineer, Infinispan
> http://infinispan.org
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
6:01 p.m.
I've upgrade to mvn 2.14 and configured: forkCount=1/reuseForks=true (default
settings) "which means that Surefire creates *one new JVM* process to execute all
tests in one *maven module*." [1]
It doesn't do that: it reuses the same process to execute all the tests in *all*
modules.
[1]
http://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-opti...
On 20 Mar 2013, at 15:29, Adrian Nistor wrote:
I've also tried changing the fork mode of surefire from
'none' to 'once' and the entire suite runs fine now on jvm 1.6 with 500mb
MaxPermSize. Previously I did not complete, 500mb was not enough.
Anyone knows why surefire was not allowed to fork?
Haven't tried to analyze closely the heap yet but first thing I noticed is 15% of it
is occupied by 190000 ComponentMetadataRepo instances, which probably is not the root
cause of this issue, but is odd anyway :).
On 03/20/2013 05:12 PM, Dan Berindei wrote:
> The problem is that we still leak threads in almost every module, and that means we
keep a copy of the core classes (and all their dependencies) for every module. Of course,
some modules' dependencies are already oversized, so keeping only one copy is already
too much...
>
> I admit I don't run the whole test suite too often either, but I recently changed
the Cloudbees settings to get rid of the OOM there. It uses about 550MB of permgen by the
end of the test suite, without -XX:+UseCompressedOops. These are the settings I used:
>
> -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
-XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m
-XX:MaxPermSize=700M
>
>
> Cheers
> Dan
>
>
>
> On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant <ttarrant(a)redhat.com> wrote:
> Sanne, turn on CompressedOops ? Still those requirements are indeed
> ridiculous.
>
> Tristan
>
> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
> > I'm testing master, at da5c3f0
> >
> > Just killed a run which was using
> >
> > java version "1.7.0_17"
> > Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
> > Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
> >
> > this time again an OOM (while I have 2GB !), last sign of life came
> > from the "Rolling Upgrade Tooling"
> >
> > I'm not going to merge/review any pull request until this works.
> >
> > Sanne
> >
> > On 20 March 2013 12:09, Mircea Markus <mmarkus(a)redhat.com> wrote:
> >> I've just run it on master and didn't get OOM. well I'm using
osx. Are you running it on master or a particular branch? Which module crashes?
> >> e.g. pedro's ISPN-2808 adds quite some threads to the party - that's
the reason it hasn't been integrated yet.
> >>
> >> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
> >>
> >>> Hi all,
> >>> after reviewing some pull requests, I'm since a couple of days
unable
> >>> to run the testsuite; since Anna's fixes affect many modules
I'm
> >>> trying to run the testsuite of the whole project, as we should always
> >>> do but I admit I haven't done it in a while because of the core
module
> >>> failures.
> >>>
> >>> So I run:
> >>> $ mvn -fn clean install
> >>>
> >>> using -fn to have it continue after the core failures.
> >>>
> >>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
> >>> sure this was good enough some months back.
> >>>
> >>> Second attempt slowed down like crazy, and I found a warning about
> >>> having filled the code cache size, so doubled it to 200M.
> >>>
> >>> Third attempt: OutOfMemoryError: PermGen space! But I'm running
with
> >>> -XX:MaxPermSize=380M which should be plenty?
> >>>
> >>> This is :
> >>> java version "1.6.0_43"
> >>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
> >>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
> >>>
> >>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
> >>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
> >>> -XX:ReservedCodeCacheSize=200M
> >>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
> >>>
> >>> My custom log configuration just disables trace & debug.
> >>>
> >>> Going to try now with larger PermGen and different JVMs but it looks
> >>> quite bad.. any other suggestion?
> >>> (I do have the security limits setup properly)
> >>>
> >>> Sanne
> >>> _______________________________________________
> >>> infinispan-dev mailing list
> >>> infinispan-dev(a)lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> Cheers,
> >> --
> >> Mircea Markus
> >> Infinispan lead (www.infinispan.org)
> >>
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev(a)lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
>
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
6:30 p.m.
On 5 Apr 2013, at 17:01, Mircea Markus <mmarkus(a)redhat.com> wrote:
I've upgrade to mvn 2.14
You mean Surefire 2.14. You had me confused for a bit, since I'm pretty sure we
enforce mvn 3.x. ;)
and configured: forkCount=1/reuseForks=true (default settings)
"which means that Surefire creates *one new JVM* process to execute all tests in one
*maven module*." [1]
It doesn't do that: it reuses the same process to execute all the tests in *all*
modules.
[1]
http://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-opti...
On 20 Mar 2013, at 15:29, Adrian Nistor wrote:
> I've also tried changing the fork mode of surefire from 'none' to
'once' and the entire suite runs fine now on jvm 1.6 with 500mb MaxPermSize.
Previously I did not complete, 500mb was not enough.
> Anyone knows why surefire was not allowed to fork?
>
> Haven't tried to analyze closely the heap yet but first thing I noticed is 15% of
it is occupied by 190000 ComponentMetadataRepo instances, which probably is not the root
cause of this issue, but is odd anyway :).
>
> On 03/20/2013 05:12 PM, Dan Berindei wrote:
>> The problem is that we still leak threads in almost every module, and that means
we keep a copy of the core classes (and all their dependencies) for every module. Of
course, some modules' dependencies are already oversized, so keeping only one copy is
already too much...
>>
>> I admit I don't run the whole test suite too often either, but I recently
changed the Cloudbees settings to get rid of the OOM there. It uses about 550MB of permgen
by the end of the test suite, without -XX:+UseCompressedOops. These are the settings I
used:
>>
>> -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
-XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m
-XX:MaxPermSize=700M
>>
>>
>> Cheers
>> Dan
>>
>>
>>
>> On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant <ttarrant(a)redhat.com>
wrote:
>> Sanne, turn on CompressedOops ? Still those requirements are indeed
>> ridiculous.
>>
>> Tristan
>>
>> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
>>> I'm testing master, at da5c3f0
>>>
>>> Just killed a run which was using
>>>
>>> java version "1.7.0_17"
>>> Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
>>> Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
>>>
>>> this time again an OOM (while I have 2GB !), last sign of life came
>>> from the "Rolling Upgrade Tooling"
>>>
>>> I'm not going to merge/review any pull request until this works.
>>>
>>> Sanne
>>>
>>> On 20 March 2013 12:09, Mircea Markus <mmarkus(a)redhat.com> wrote:
>>>> I've just run it on master and didn't get OOM. well I'm using
osx. Are you running it on master or a particular branch? Which module crashes?
>>>> e.g. pedro's ISPN-2808 adds quite some threads to the party -
that's the reason it hasn't been integrated yet.
>>>>
>>>> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
>>>>
>>>>> Hi all,
>>>>> after reviewing some pull requests, I'm since a couple of days
unable
>>>>> to run the testsuite; since Anna's fixes affect many modules
I'm
>>>>> trying to run the testsuite of the whole project, as we should
always
>>>>> do but I admit I haven't done it in a while because of the core
module
>>>>> failures.
>>>>>
>>>>> So I run:
>>>>> $ mvn -fn clean install
>>>>>
>>>>> using -fn to have it continue after the core failures.
>>>>>
>>>>> First attempt gave me an OOM, was running with 1G heap.. I'm
pretty
>>>>> sure this was good enough some months back.
>>>>>
>>>>> Second attempt slowed down like crazy, and I found a warning about
>>>>> having filled the code cache size, so doubled it to 200M.
>>>>>
>>>>> Third attempt: OutOfMemoryError: PermGen space! But I'm running
with
>>>>> -XX:MaxPermSize=380M which should be plenty?
>>>>>
>>>>> This is :
>>>>> java version "1.6.0_43"
>>>>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>>>>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>>>>>
>>>>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>>>>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>>>>> -XX:ReservedCodeCacheSize=200M
>>>>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>>>>>
>>>>> My custom log configuration just disables trace & debug.
>>>>>
>>>>> Going to try now with larger PermGen and different JVMs but it looks
>>>>> quite bad.. any other suggestion?
>>>>> (I do have the security limits setup properly)
>>>>>
>>>>> Sanne
>>>>> _______________________________________________
>>>>> infinispan-dev mailing list
>>>>> infinispan-dev(a)lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>> Cheers,
>>>> --
>>>> Mircea Markus
>>>> Infinispan lead (www.infinispan.org)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev(a)lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>>
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Platform Architect, JBoss Data Grid
http://red.ht/data-grid
11:02 a.m.
On 5 Apr 2013, at 17:01, Mircea Markus <mmarkus(a)redhat.com> wrote:
I've upgrade to mvn 2.14
You mean Surefire 2.14. You had me confused for a bit, since I'm pretty sure we
enforce mvn 3.x. ;)
and configured: forkCount=1/reuseForks=true (default settings)
"which means that Surefire creates *one new JVM* process to execute all tests in one
*maven module*." [1]
It doesn't do that: it reuses the same process to execute all the tests in *all*
modules.
[1]
http://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-opti...
On 20 Mar 2013, at 15:29, Adrian Nistor wrote:
> I've also tried changing the fork mode of surefire from 'none' to
'once' and the entire suite runs fine now on jvm 1.6 with 500mb MaxPermSize.
Previously I did not complete, 500mb was not enough.
> Anyone knows why surefire was not allowed to fork?
>
> Haven't tried to analyze closely the heap yet but first thing I noticed is 15% of
it is occupied by 190000 ComponentMetadataRepo instances, which probably is not the root
cause of this issue, but is odd anyway :).
>
> On 03/20/2013 05:12 PM, Dan Berindei wrote:
>> The problem is that we still leak threads in almost every module, and that means
we keep a copy of the core classes (and all their dependencies) for every module. Of
course, some modules' dependencies are already oversized, so keeping only one copy is
already too much...
>>
>> I admit I don't run the whole test suite too often either, but I recently
changed the Cloudbees settings to get rid of the OOM there. It uses about 550MB of permgen
by the end of the test suite, without -XX:+UseCompressedOops. These are the settings I
used:
>>
>> -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
-XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m
-XX:MaxPermSize=700M
>>
>>
>> Cheers
>> Dan
>>
>>
>>
>> On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant <ttarrant(a)redhat.com>
wrote:
>> Sanne, turn on CompressedOops ? Still those requirements are indeed
>> ridiculous.
>>
>> Tristan
>>
>> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
>>> I'm testing master, at da5c3f0
>>>
>>> Just killed a run which was using
>>>
>>> java version "1.7.0_17"
>>> Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
>>> Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
>>>
>>> this time again an OOM (while I have 2GB !), last sign of life came
>>> from the "Rolling Upgrade Tooling"
>>>
>>> I'm not going to merge/review any pull request until this works.
>>>
>>> Sanne
>>>
>>> On 20 March 2013 12:09, Mircea Markus <mmarkus(a)redhat.com> wrote:
>>>> I've just run it on master and didn't get OOM. well I'm using
osx. Are you running it on master or a particular branch? Which module crashes?
>>>> e.g. pedro's ISPN-2808 adds quite some threads to the party -
that's the reason it hasn't been integrated yet.
>>>>
>>>> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
>>>>
>>>>> Hi all,
>>>>> after reviewing some pull requests, I'm since a couple of days
unable
>>>>> to run the testsuite; since Anna's fixes affect many modules
I'm
>>>>> trying to run the testsuite of the whole project, as we should
always
>>>>> do but I admit I haven't done it in a while because of the core
module
>>>>> failures.
>>>>>
>>>>> So I run:
>>>>> $ mvn -fn clean install
>>>>>
>>>>> using -fn to have it continue after the core failures.
>>>>>
>>>>> First attempt gave me an OOM, was running with 1G heap.. I'm
pretty
>>>>> sure this was good enough some months back.
>>>>>
>>>>> Second attempt slowed down like crazy, and I found a warning about
>>>>> having filled the code cache size, so doubled it to 200M.
>>>>>
>>>>> Third attempt: OutOfMemoryError: PermGen space! But I'm running
with
>>>>> -XX:MaxPermSize=380M which should be plenty?
>>>>>
>>>>> This is :
>>>>> java version "1.6.0_43"
>>>>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>>>>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>>>>>
>>>>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>>>>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>>>>> -XX:ReservedCodeCacheSize=200M
>>>>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>>>>>
>>>>> My custom log configuration just disables trace & debug.
>>>>>
>>>>> Going to try now with larger PermGen and different JVMs but it looks
>>>>> quite bad.. any other suggestion?
>>>>> (I do have the security limits setup properly)
>>>>>
>>>>> Sanne
>>>>> _______________________________________________
>>>>> infinispan-dev mailing list
>>>>> infinispan-dev(a)lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>> Cheers,
>>>> --
>>>> Mircea Markus
>>>> Infinispan lead (www.infinispan.org)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev(a)lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>>
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Platform Architect, JBoss Data Grid
http://red.ht/data-grid
1:59 p.m.
On 8 Apr 2013, at 10:02, Manik Surtani wrote:
> I've upgrade to mvn 2.14
You mean Surefire 2.14. You had me confused for a bit, since I'm pretty sure we
enforce mvn 3.x. ;)
yep, surefire 2.14 :-)
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
4:33 p.m.
On 20 Mar 2013, at 15:12, Dan Berindei wrote:
The problem is that we still leak threads in almost every module, and
that means we keep a copy of the core classes (and all their dependencies) for every
module. Of course, some modules' dependencies are already oversized, so keeping only
one copy is already too much...
do we have a JIRA for this?
I admit I don't run the whole test suite too often either, but I recently changed the
Cloudbees settings to get rid of the OOM there. It uses about 550MB of permgen by the end
of the test suite, without -XX:+UseCompressedOops. These are the settings I used:
-server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
-XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m
-XX:MaxPermSize=700M
Thanks for sharing this Dan!
Cheers
Dan
On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant <ttarrant(a)redhat.com> wrote:
Sanne, turn on CompressedOops ? Still those requirements are indeed
ridiculous.
Tristan
On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
> I'm testing master, at da5c3f0
>
> Just killed a run which was using
>
> java version "1.7.0_17"
> Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
>
> this time again an OOM (while I have 2GB !), last sign of life came
> from the "Rolling Upgrade Tooling"
>
> I'm not going to merge/review any pull request until this works.
>
> Sanne
>
> On 20 March 2013 12:09, Mircea Markus <mmarkus(a)redhat.com> wrote:
>> I've just run it on master and didn't get OOM. well I'm using osx.
Are you running it on master or a particular branch? Which module crashes?
>> e.g. pedro's ISPN-2808 adds quite some threads to the party - that's the
reason it hasn't been integrated yet.
>>
>> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
>>
>>> Hi all,
>>> after reviewing some pull requests, I'm since a couple of days unable
>>> to run the testsuite; since Anna's fixes affect many modules I'm
>>> trying to run the testsuite of the whole project, as we should always
>>> do but I admit I haven't done it in a while because of the core module
>>> failures.
>>>
>>> So I run:
>>> $ mvn -fn clean install
>>>
>>> using -fn to have it continue after the core failures.
>>>
>>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
>>> sure this was good enough some months back.
>>>
>>> Second attempt slowed down like crazy, and I found a warning about
>>> having filled the code cache size, so doubled it to 200M.
>>>
>>> Third attempt: OutOfMemoryError: PermGen space! But I'm running with
>>> -XX:MaxPermSize=380M which should be plenty?
>>>
>>> This is :
>>> java version "1.6.0_43"
>>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>>>
>>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>>> -XX:ReservedCodeCacheSize=200M
>>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>>>
>>> My custom log configuration just disables trace & debug.
>>>
>>> Going to try now with larger PermGen and different JVMs but it looks
>>> quite bad.. any other suggestion?
>>> (I do have the security limits setup properly)
>>>
>>> Sanne
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> Cheers,
>> --
>> Mircea Markus
>> Infinispan lead (www.infinispan.org)
>>
>>
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
5:08 p.m.
On Wed, Mar 20, 2013 at 5:33 PM, Mircea Markus <mmarkus(a)redhat.com> wrote:
On 20 Mar 2013, at 15:12, Dan Berindei wrote:
> The problem is that we still leak threads in almost every module, and
that means we keep a copy of the core classes (and all their dependencies)
for every module. Of course, some modules' dependencies are already
oversized, so keeping only one copy is already too much...
do we have a JIRA for this?
There is https://issues.jboss.org/browse/ISPN-2477, but we probably need a
new issue to cover the test suite specifically.
> I admit I don't run the whole test suite too often either, but I
recently changed the Cloudbees settings to get rid of the OOM there. It
uses about 550MB of permgen by the end of the test suite, without
-XX:+UseCompressedOops. These are the settings I used:
>
> -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
-XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m
-XX:MaxPermSize=700M
Thanks for sharing this Dan!
>
>
> Cheers
> Dan
>
>
>
> On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant <ttarrant(a)redhat.com>
wrote:
> Sanne, turn on CompressedOops ? Still those requirements are indeed
> ridiculous.
>
> Tristan
>
> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
> > I'm testing master, at da5c3f0
> >
> > Just killed a run which was using
> >
> > java version "1.7.0_17"
> > Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
> > Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
> >
> > this time again an OOM (while I have 2GB !), last sign of life came
> > from the "Rolling Upgrade Tooling"
> >
> > I'm not going to merge/review any pull request until this works.
> >
> > Sanne
> >
> > On 20 March 2013 12:09, Mircea Markus <mmarkus(a)redhat.com> wrote:
> >> I've just run it on master and didn't get OOM. well I'm using
osx.
Are you running it on master or a particular branch? Which module crashes?
> >> e.g. pedro's ISPN-2808 adds quite some threads to the party -
that's
the reason it hasn't been integrated yet.
> >>
> >> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
> >>
> >>> Hi all,
> >>> after reviewing some pull requests, I'm since a couple of days
unable
> >>> to run the testsuite; since Anna's fixes affect many modules
I'm
> >>> trying to run the testsuite of the whole project, as we should always
> >>> do but I admit I haven't done it in a while because of the core
module
> >>> failures.
> >>>
> >>> So I run:
> >>> $ mvn -fn clean install
> >>>
> >>> using -fn to have it continue after the core failures.
> >>>
> >>> First attempt gave me an OOM, was running with 1G heap.. I'm
pretty
> >>> sure this was good enough some months back.
> >>>
> >>> Second attempt slowed down like crazy, and I found a warning about
> >>> having filled the code cache size, so doubled it to 200M.
> >>>
> >>> Third attempt: OutOfMemoryError: PermGen space! But I'm running
with
> >>> -XX:MaxPermSize=380M which should be plenty?
> >>>
> >>> This is :
> >>> java version "1.6.0_43"
> >>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
> >>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
> >>>
> >>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
> >>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
> >>> -XX:ReservedCodeCacheSize=200M
> >>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
> >>>
> >>> My custom log configuration just disables trace & debug.
> >>>
> >>> Going to try now with larger PermGen and different JVMs but it looks
> >>> quite bad.. any other suggestion?
> >>> (I do have the security limits setup properly)
> >>>
> >>> Sanne
> >>> _______________________________________________
> >>> infinispan-dev mailing list
> >>> infinispan-dev(a)lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> Cheers,
> >> --
> >> Mircea Markus
> >> Infinispan lead (www.infinispan.org)
> >>
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev(a)lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
5:05 p.m.
On 03/20/2013 04:12 PM, Dan Berindei wrote:
The problem is that we still leak threads in almost every module, and
that means we keep a copy of the core classes (and all their
dependencies) for every module. Of course, some modules' dependencies
are already oversized, so keeping only one copy is already too much...
I did do
quite a bit of cleanup in the server-core and server-hotrod
modules a while back, but things might have regressed since then. I
think we need to go on a cleaning spree.
Tristan
4027
days inactive
4061
days old
infinispan-dev@lists.jboss.org
20 comments
7 participants
participants (7)
-
Adrian Nistor -
Dan Berindei -
Galder Zamarreño -
Manik Surtani -
Mircea Markus -
Sanne Grinovero -
Tristan Tarrant