Kabir Khan wrote: Wild guess - Perhaps, JBoss Messaging takes time to create the JBM tables for the first run. The second run the tables are already present in JBOSS_HOME/data folder so i guess it's skipped. Maybe deleting the data folder before starting the server, the second time, will show no significant difference between the two runs. regards, -Jaikiran

BTW, my guess is that it is file caching. Good job though! Almost a full second shaved off. Nothing to sneeze about. So what's next for optimization? * Is VFS3 in trunk/ now? * Did somebody look into whether code guards are being done for logging? (isDebugEnabled(), etc.) * Any possible improvements to creating BeanMetaData? * How about making our bean.xml files more coarse grain? Meaning *A LOT* less implementation details exposed through XML. You can do IoC in Java you know. The vast majority of details within all our beans.xml files will never ever change, nor will we want to support users changing this stuff. If our beans.xml file are reduced to a few bean definitions and classes, would make parsing and creating bean metadata much much faster. * I also could resurrect Fast-JAXB, but I didn't want to do that until we got under 10 seconds. For the MC team, I think the biggest focus should be profiling and optimizing Minimal. Kabir Khan wrote: -- Bill Burke JBoss, a division of Red Hat http://bill.burkecentral.com

Brian Stansberry wrote: EJB and Webservices create an insane amount of beans that can probably be collapsed into a few beans. -- Bill Burke JBoss, a division of Red Hat http://bill.burkecentral.com

Although I agree that we have user configuration and internal configuration mixed up in a lot of places, I don't agree on using other IoC constructs to resolve internal concerns. We want flexibility on every level and I would need an IoC construct anyway. So we should be able to use MC for all scenarios and fix the scalability issue where it lies. Carlo On 02/12/2010 02:34 PM, Dimitris Andreadis wrote:

Bill Burke wrote: Amen Brother Bill :) Just because we develop cool interesting features does not mean we need to use them all over the place for our internal implementations. Part of being a framework developer is having to sacrifice code creativity and reusability in the name of speed. -- Jason T. Greene JBoss, a division of Red Hat

I did an alternative dependency resolution prototype [1] where I did basic indexing of depenencies. It performed much better than the existing dependency resolution algorithm when everything is deployed in the wrong order, but I did not really see the performance gains I was hoping for in AS, since dependencies are not all deployed in the wrong order there. When things are deployed more or less in the right order, it did not make much difference. Then again, it was just a proof of concept so it is doable, but still not as efficient as it could be. For example I only try to resolve the dependencies of contexts who match a dependency of a recently incremented context, but I still try to resolve all those dependencies rather than tracking all the dependencies. I might pick it up again once some of the other "perfomance noise" from VFS, MDR and other core projects is gone, since there are definitely a few of the fixes that I have done the past week or so that should have an impact on this as well. [1] http://community.jboss.org/thread/146838?tstart=0 On 18 Feb 2010, at 02:06, David M. Lloyd wrote:

On 11 Feb 2010, at 23:23, Bill Burke wrote: In dependency the things which had the biggest impact were not breaking out of the resolveContexts() loop and that AbstractDependencyItem was calling controller.getContext() three times when a dependency is not there yet. For me, profiling the whole app server startup gives too much information and different hotspots don't really stand out that much. I'm creating a series of micro-benchmarks to profile different areas in isolation. Next is trying the different things in jboss-kernel, which is more interesting since it pulls in things like MDR and jboss-reflect. I'll have a look at the ScopeKey lookups I mentioned in http://community.jboss.org/message/525925#525925 today, before trying out other parts of kernel. I believe Ales did this for MC If somebody wants to look into that, some help would be welcome. I've got a feeling my hands will be full the next few weeks :-)

Bill, Brian and everyone else involved: great work on putting a concerted effort into this! Mark. On 12 Feb 2010, at 10:19, Kabir Khan wrote: --- Mark Little mlittle(a)redhat.com JBoss, by Red Hat Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in UK and Wales under Company Registration No. 3798903 Directors: Michael Cunningham (USA), Charlie Peters (USA), Matt Parsons (USA) and Brendan Lane (Ireland).

I found that ScopeKey's hashCode was not being calculated correctly (http://community.jboss.org/message/526058#526058), and fixed this in a local jboss-mdr snapshot, which gives another tiny enhancement. Here are today's numbers, same setup as last time. kernel 2.2.0-Alpha5 mdr 2.2.0-Alpha1 =============== 34s:696ms (1) 20s:562ms (2) 20s:349ms (7) 20s:463ms (10) 20s:622ms (13) 20s:582ms (16) 20s:35ms (19) ------------------------- Average ignoring (1): 20s436ms kernel 2.2.0-SNAPSHOT mdr 2.2.0-Alpha1 =================== 29s:637ms (3) 19s:627ms (4) 19s:786ms (8) 20s:2ms (11) 20s:432ms (14) 19s:827ms (17) 19s:731ms (20) ------------------------------- Average ignoring (3): 19s900ms kernel 2.2.0-SNAPSHOT mdr 2.2.0-SNAPSHOT ============================ 27s:167ms (5) 19s:834ms (6) 19s:702ms (9) 19s:349ms (12) 20s:228ms (15) 19s:346ms (18) 19s:315ms (21) ---------------------------- Average ignoring (5): 19s629ms Now that I'm more into profiling stuff, I'll see if profiling minimal as suggested by Bill makes more sense to me, if not I'll do a few more cycles of these. On 11 Feb 2010, at 12:25, Kabir Khan wrote:

Another 200ms taken off by this change indexing the ControllerStates in the controller making state comparisons more efficient: http://community.jboss.org/message/526380#526380 Average startup time for me is 18s687ms compared with 18s856ms with this morning's setup. For minimal, the startup times are about 4s780ms with both setups down from 5s224ms with AS-6.0.0.M2 On 15 Feb 2010, at 12:22, Kabir Khan wrote:

I need some input from Adrian/Ales on [1] regarding changes to MDR. Following from Jaikiran's thread [2], I found [3] and [4] and I created a few benchmarks for those areas and found that getting a component MetaDataRetrieval for the AnnotatedElementMetaDataLoader is quite costly, and found that AnnotatedElementMetaDataLoader.getComponentMetaDataRetrieval() creates a return value whether or not there are any annotations. More on that in [1], but doing that fix in my local benchmark has shown the biggest improvement for any single fix: From: 18s878ms with yesterday's setup To: 18s207ms with the fix from [1] Full list of times here, including the first time under 18 seconds :-) Yesterday's previously best setup =========================== 18s:920ms (1) 19s:301ms (3) 18s:464ms (5) 18s:866ms (7) 18s:893ms (9) 18s:826ms (11) With Component metadata fix =========================== 18s:73ms (2) 18s:473ms (4) 18s:244ms (6) 18s:276ms (8) 18s:214ms (10) 17s:965ms (12) Regarding the issues raised in [2], I found that 1) findMethod() overhead was negligible, but I might have been benchmarking the wrong thing 2) Was done a while ago for https://jira.jboss.org/jira/browse/JBKERNEL-64 3) I can try to look into that 4) Was handled already [1] http://community.jboss.org/thread/148203 [2] http://community.jboss.org/message/434125#434125 [3] http://community.jboss.org/thread/96937?tstart=0 [4] http://community.jboss.org/message/316208#316208 On 15 Feb 2010, at 15:48, Kabir Khan wrote:

I did some more optimizations on [1] and while profiling this area modified the most frequent user [2] so compared with the setup from 16 Feb there are some more improvements. The average time with today's fixes is: 17s932ms With setup from 16 Feb the average time is: 18s147ms 16 Feb ====== 18s:193ms (1) 18s:199ms (4) 18s:114ms (5) 18s:114ms (8) 18s:131ms (10) 18s:130ms (12) 18 Feb ====== 18s:136ms (2) 18s:96ms (3) 18s:45ms (6) 17s:745ms (7) 17s:536ms (9) 18s:31ms (11) [2] http://community.jboss.org/message/527084#527084 On 16 Feb 2010, at 20:10, Kabir Khan wrote: > I need some input from Adrian/Ales on [1] regarding changes to MDR. Following from Jaikiran's thread [2], I found [3] and [4] and I created a few benchmarks for those areas and found that getting a component MetaDataRetrieval for the AnnotatedElementMetaDataLoader is quite costly, and found that AnnotatedElementMetaDataLoader.getComponentMetaDataRetrieval() creates a return value whether or not there are any annotations. More on that in [1], but doing that fix in my local benchmark has shown the biggest improvement for any single fix: > > From: 18s878ms with yesterday's setup > To: 18s207ms with the fix from [1] > > Full list of times here, including the first time under 18 seconds :-) > > Yesterday's previously best setup > =========================== > 18s:920ms (1) > 19s:301ms (3) > 18s:464ms (5) > 18s:866ms (7) > 18s:893ms (9) > 18s:826ms (11) > > With Component metadata fix > =========================== > 18s:73ms (2) > 18s:473ms (4) > 18s:244ms (6) > 18s:276ms (8) > 18s:214ms (10) > 17s:965ms (12) > > Regarding the issues raised in [2], I found that > 1) findMethod() overhead was negligible, but I might have been benchmarking the wrong thing > 2) Was done a while ago for https://jira.jboss.org/jira/browse/JBKERNEL-64 > 3) I can try to look into that > 4) Was handled already > > [2] http://community.jboss.org/message/434125#434125 > [3] http://community.jboss.org/thread/96937?tstart=0 > [4] http://community.jboss.org/message/316208#316208 > > On 15 Feb 2010, at 15:48, Kabir Khan wrote: > >> Another 200ms taken off by this change indexing the ControllerStates in the controller making state comparisons more efficient: http://community.jboss.org/message/526380#526380 >> >> Average startup time for me is 18s687ms compared with 18s856ms with this morning's setup. >> For minimal, the startup times are about 4s780ms with both setups down from 5s224ms with AS-6.0.0.M2 >> >> >> On 15 Feb 2010, at 12:22, Kabir Khan wrote: >> >>> Getting rid of the @Dependency annotation checks from AOPDependencyBuilder https://jira.jboss.org/jira/browse/JBKERNEL-98 brings the startup time down to the 18s range. >>> >>> kernel 2.2.0-SNAPSHOT, from a few days ago (Alpha6 with the extra resolveContexts loop optimization) >>> mdr 2.2.0-Alpha1 >>> ================================================================================ >>> 20s:192ms (1) >>> 19s:733ms (3) >>> 19s:968ms (4) >>> >>> kernel 2.2.0-SNAPSHOT, includes the @Dependency stuff >>> mdr 2.2.0-SNAPSHOT >>> ============================================ >>> 18s:905ms (2) >>> 18s:604ms (4) >>> 18s:657ms (6) >>> >>> >>> >>> On 12 Feb 2010, at 15:50, Kabir Khan wrote: >>> >>>> I found that ScopeKey's hashCode was not being calculated correctly (http://community.jboss.org/message/526058#526058), and fixed this in a local jboss-mdr snapshot, which gives another tiny enhancement. Here are today's numbers, same setup as last time. >>>> >>>> kernel 2.2.0-Alpha5 >>>> mdr 2.2.0-Alpha1 >>>> =============== >>>> 34s:696ms (1) >>>> 20s:562ms (2) >>>> 20s:349ms (7) >>>> 20s:463ms (10) >>>> 20s:622ms (13) >>>> 20s:582ms (16) >>>> 20s:35ms (19) >>>> ------------------------- >>>> Average ignoring (1): 20s436ms >>>> >>>> kernel 2.2.0-SNAPSHOT >>>> mdr 2.2.0-Alpha1 >>>> =================== >>>> 29s:637ms (3) >>>> 19s:627ms (4) >>>> 19s:786ms (8) >>>> 20s:2ms (11) >>>> 20s:432ms (14) >>>> 19s:827ms (17) >>>> 19s:731ms (20) >>>> ------------------------------- >>>> Average ignoring (3): 19s900ms >>>> >>>> kernel 2.2.0-SNAPSHOT >>>> mdr 2.2.0-SNAPSHOT >>>> ============================ >>>> 27s:167ms (5) >>>> 19s:834ms (6) >>>> 19s:702ms (9) >>>> 19s:349ms (12) >>>> 20s:228ms (15) >>>> 19s:346ms (18) >>>> 19s:315ms (21) >>>> ---------------------------- >>>> Average ignoring (5): 19s629ms >>>> >>>> Now that I'm more into profiling stuff, I'll see if profiling minimal as suggested by Bill makes more sense to me, if not I'll do a few more cycles of these. >>>> >>>> >>>> On 11 Feb 2010, at 12:25, Kabir Khan wrote: >>>> >>>>> I have been doing some work on profiling and optimizing the jboss-dependency project [1] for kernel 2.2.0.Alpha6 and it seems to have had a small, but still measurable impact :-) >>>>> >>>>> I did a few startups of AS with jboss kernel 2.2.0.Alpha5 and with a local snapshot. This snapshot contains the work done for Alpha6 with the addition of removing the break in the resolveContexts loop as mentioned in the forum thread. >>>>> >>>>> The startups were done on a freshly rebooted machine with nothing else running. The startup orders are given in brackets, so I started Alpha5 twice, then SNAPSHOT twice and so on. >>>>> >>>>> 2.2.0-Alpha5 >>>>> ---------------------- >>>>> 37s:247ms (1) >>>>> 20s:573ms (2) >>>>> 20s:275ms (5) >>>>> 20s:127ms (7) >>>>> 20s:433ms (9) >>>>> 20s:329ms (10) >>>>> 20s:890ms (13) >>>>> >>>>> >>>>> 2.2.0-SNAPSHOT >>>>> ------------------------ >>>>> 29s:563ms (3) >>>>> 19s:965ms (4) >>>>> 19s:821ms (6) >>>>> 20s:332ms (8) >>>>> 19s:618ms (10) >>>>> 19s:905ms (12) >>>>> 19s:900ms (14) >>>>> >>>>> One strange thing is that the first time I started each server they took loads longer than the other times, does somebody know the reason for that? >>>>> >>>>> Ignoring those initial times leaves me with average start times of: >>>>> Alpha5: 20s437ms >>>>> SNAPSHOT: 19s923ms >>>>> >>>>> I'm going to have a final look at jboss-dependency to see if there are any other obvious and easy fixes before moving on to have a look at jboss-kernel. >>>>> >>>>> [1] http://community.jboss.org/message/525047#525047 >>>>> _______________________________________________ >>>>> jboss-development mailing list >>>>> jboss-development(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/jboss-development >>>> >>>> >>>> _______________________________________________ >>>> jboss-development mailing list >>>> jboss-development(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-development >>> >>> >>> _______________________________________________ >>> jboss-development mailing list >>> jboss-development(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-development >> >> >> _______________________________________________ >> jboss-development mailing list >> jboss-development(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-development > > > _______________________________________________ > jboss-development mailing list > jboss-development(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-development

I can account for a couple differences between minimal and jbossweb-standalone: 1) conf/jboss-service.xml in minimal does not include this huge addition to the server classpath: <classpath codebase="${jboss.common.lib.url}" archives="*"/> Adding that to minimal increases it's startup time for me from ~ 6.4s to ~ 6.8s. BTW, Dimitris, I want a machine like his 2.6s one. ;-) 2) The ServiceBindingManager config in minimal is just an empty placeholder; in jbossweb-standalone it's the real thing. Putting that one in minimal increases its boot time to ~ 7.2s. On 02/23/2010 11:30 AM, Rémy Maucherat wrote: -- Brian Stansberry Lead, AS Clustering JBoss by Red Hat

On 02/23/2010 06:13 PM, Rémy Maucherat wrote: SBM configuration should have its own schema[1]. Not sure what effect that will have on boot time -- maybe good because the MC isn't dealing with a bunch of beans it doesn't really need to know about; maybe bad as another new set of xml parsing overhead is added. Also relevant to the jbossweb-standalone numbers (but not too minimal) is SBM is doing that XSL transform on server.xml. There's a JIRA to get rid of that, although I'm not too happy about using system properties to pass these values. We do it elsewhere though. [1] https://jira.jboss.org/jira/browse/JBAS-6829 [2] https://jira.jboss.org/jira/browse/JBAS-7466 -- Brian Stansberry Lead, AS Clustering JBoss by Red Hat

On Wed, Feb 24, 2010 at 5:45 PM, Brian Stansberry <brian.stansberry(a)redhat.com&gt; wrote: Dropping XSL does improve the numbers. Should I commit that ? [I'm not committing yet the simplified jbossweb-standalone due to JBKERNEL-100] Rémy

On Wed, Feb 24, 2010 at 8:02 PM, Brian Stansberry <brian.stansberry(a)redhat.com&gt; wrote: With the change, I have the naming.test.SimpleUnitTestCase.testHttpInvoker from the smoke tests which is failing and I don't really understand why. Strange. Rémy

On Wed, Feb 24, 2010 at 10:35 PM, Brian Stansberry <brian.stansberry(a)redhat.com&gt; wrote: After killing off JARs in the web release packaging (down to a 20MB download - from 70MB for the beta1), I get: JBoss Web 2.1: 0.6s / 6.8MB minimal: 2.6s / 35MB jbossweb-standalone (with the jbw release packaging): 3.9s / 65.5MB default: 9.5s / 161.5MB all: 20s / 237MB Although improving the startup time further would be nice, some effort would also be needed on memory use I think ? Rémy

On Fri, Feb 26, 2010 at 3:56 PM, Brian Stansberry <brian.stansberry(a)redhat.com&gt; wrote: I didn't have the impression that VFS was a factor here, but we'll see when this is integrated. Rémy

I have upgraded trunk to use kernel 2.2.0.Alpha7 and mdr 2.2.0.Alpha2 which contain the performance fixes I have been working on. For my own record the last hudson run before this was http://hudson.qa.jboss.com/hudson/view/JBoss%20AS/job/JBoss-AS-6.0.x-test... On 26 Feb 2010, at 17:52, Scott Marlow wrote:

Cool :-) It seems to have broken quite a few of the scoped aop tests http://hudson.qa.jboss.com/hudson/view/JBoss%20AS/job/JBoss-AS-6.0.x-test... It is hopefully something simple, so I'll fix those tomorrow before John's VFS upgrade On 28 Feb 2010, at 09:24, Jaikiran Pai wrote:

I am seeing the same thing. I won't be able to get a fix in before John starts the VFS stuff so I will revert to the previous mdr + kernel versions and investigate locally On 1 Mar 2010, at 12:36, Rémy Maucherat wrote:

Yes, I'll fix and do a new release once VFS 3 is integrated :-) On 1 Mar 2010, at 17:25, Rémy Maucherat wrote:

That's being fixed https://jira.jboss.org/jira/browse/JBBOOT-131 -Jaikiran Ales Justin wrote:

https://jira.jboss.org/jira/browse/JBKERNEL-107 On 1 Mar 2010, at 12:36, Rémy Maucherat wrote:

On Thu, Mar 4, 2010 at 7:10 PM, Kabir Khan <kabir.khan(a)jboss.com&gt; wrote: Ok, thanks for the fix. Rémy

On Thu, Mar 4, 2010 at 9:00 PM, Kabir Khan <kabir.khan(a)jboss.com&gt; wrote: Cool, my optional dependency is now ok :) default: 8.5s all: 18.93s jbossweb-standalone (with the jbw release packaging): 3.5s Nice :) Rémy

On Fri, Feb 26, 2010 at 1:55 PM, Rémy Maucherat <remy.maucherat(a)gmail.com&gt; wrote: Results before/after VFS 3. minimal: 2.3s / 30.2MB jbossweb-standalone (with the jbw release packaging): 3.8s / 56.4MB (small gain here, so some of the benefit from VFS 3 probably comes from better handling of the large amount of common JARs, which are heavily trimmed in that packaging) On the plus side, in 2000, it took Tomcat 4 that amount of time to start, and the memory use was relatively much bigger than this ;) default: 8.95s / 144.5MB all: 19.2s / 209.8MB So there's an improvement (and the next MC update will also bring another improvement). Very good job on the lack of huge regressions. Rémy

Jason T. Greene wrote: This is post-full-gc. Remy said the above. Thunderbird loves to mangle replies. -- Jason T. Greene JBoss, a division of Red Hat

On 03/02/2010 04:20 AM, Kabir Khan wrote: John, your work on this has really been outstanding. -- Brian Stansberry Lead, AS Clustering JBoss by Red Hat

Kabir Khan wrote: You also might want to try removing stuff in deploy/ and just deploying deployers (you'll need to remove the war and weld deployers though). That may shed more light on things too. Its how I found the deployer sorting issue. Correct me if I'm wrong, but after VFS3 is in, the next big thing I saw is probably BeanInfo creation (all the reflection stuff). I personally don't think the JDK hacks that Carlo came up with for reflection are a good idea. This is why I thought all projects might want to consider making their beans more coarse grain and do IoC within code. -- Bill Burke JBoss, a division of Red Hat http://bill.burkecentral.com

Kabir Khan wrote: I haven't looked in the latest profiler snapshots, but the last time i checked, we noticed this http://community.jboss.org/message/434125#434125 -Jaikiran

It's this piece of code: package org.jboss.hack; import java.lang.reflect.InvocationTargetException; import java.lang.reflect.Method; /** * Fast method finder. * * @author <a href="mailto:cdewolf@redhat.com">Carlo de Wolf</a> */ public class ClassHack { private static Method COPY; private static Method PRIVATE_GET_DECLARED_METHODS; private static Method SEARCH_METHODS; static { try { COPY = Method.class.getDeclaredMethod("copy"); COPY.setAccessible(true); PRIVATE_GET_DECLARED_METHODS = Class.class.getDeclaredMethod("privateGetDeclaredMethods", Boolean.TYPE); PRIVATE_GET_DECLARED_METHODS.setAccessible(true); SEARCH_METHODS = Class.class.getDeclaredMethod("searchMethods", Method[].class, String.class, Class[].class); SEARCH_METHODS.setAccessible(true); } catch (SecurityException e) { throw new RuntimeException(e); } catch (NoSuchMethodException e) { throw new RuntimeException(e); } } /** * Returns a safe copy of the method. * * @param unsafeMethod * @return */ private static Method copy(Method unsafeMethod) { // TODO: should really use ReflectAccess / LangReflectAccess try { return (Method) COPY.invoke(unsafeMethod); } catch (IllegalArgumentException e) { throw new RuntimeException(e); } catch (IllegalAccessException e) { throw new RuntimeException(e); } catch (InvocationTargetException e) { throw new RuntimeException(e); } } public static Method findMethod(Class<?> cls, String name, Class<?>... parameterTypes) { if(cls == null) return null; Method unsafeMethods[] = getDeclaredMethods(cls, false); Method unsafeMethod = searchMethods(unsafeMethods, name, parameterTypes); if(unsafeMethod != null) return copy(unsafeMethod); return findMethod(cls.getSuperclass(), name, parameterTypes); } /** * Very unsafe, returns references to the cache. * * @param publicOnly * @return */ private static Method[] getDeclaredMethods(Class<?> cls, boolean publicOnly) { try { return (Method[]) PRIVATE_GET_DECLARED_METHODS.invoke(cls, publicOnly); } catch (IllegalArgumentException e) { throw new RuntimeException(e); } catch (IllegalAccessException e) { throw new RuntimeException(e); } catch (InvocationTargetException e) { throw new RuntimeException(e); } } public static Method searchMethods(Method[] methods, String name, Class<?>[] parameterTypes) { try { return (Method) SEARCH_METHODS.invoke(null, methods, name, parameterTypes); } catch (IllegalArgumentException e) { throw new RuntimeException(e); } catch (IllegalAccessException e) { throw new RuntimeException(e); } catch (InvocationTargetException e) { throw new RuntimeException(e); } } } I never committed it, because as Bill says it's a Bad Thing. Not only does it abuse internal methods it also exposes Methods from a cache that lead to huge security holes. So don't commit it, ever. :-) The important bit is that scanning annotations using reflection is extremely slow, so never do that either. Carlo On 02/16/2010 02:20 PM, Kabir Khan wrote:

Something that starts a non-daemon thread? (or is it the other way around) Kabir Khan wrote: -- Bill Burke JBoss, a division of Red Hat http://bill.burkecentral.com

Kabir Khan wrote: Hmm, that looks like a System.out.println statement from some piece of code. -Jaikiran