[wildfly-dev] Speeding up WildFly boot time

[Brian Stansberry]

+1 re: being careful about ServiceLoader.

We were doing some perf testing work last week and that's one thing that showed up. Unfortunately it was in the app we were testing rather than in the server code but I could easily imagine similar things happening in the server.

In case people are curious, the issue is the javax.json.Json class, which provides a bunch of static utility methods to create JSON related objects. The problem is they are implemented via "JsonProvider.provider().doXXX”. And that JsonProvider.provider() call uses a ServiceLoader to try and find any custom JsonProvider impls before falling back using the default. WildFly doesn’t ship any such impls. Those ServiceLoader calls all result in a FileNotFoundException as the classloader checks for META-INF/services/…JsonProvider. So IO access plus the cost of creating exception.

The solution is to not use  javax.json.Json.xxx all the time in the app, but to call JsonProvider.provider() once and cache it.

> On May 15, 2017, at 9:23 AM, Sanne Grinovero <sanne at hibernate.org> wrote:
> 
> Very interesting.
> 
>> From a different perspective, but closely related, I was recently
> trying to profile the testsuites of Hibernate projects and also in our
> case ClassLoader time is significant portion of the bootstrap time.
> 
> In the Hibernate case I noticed it spends quite some time to locate
> Service implementations over the ServiceLoader pattern.
> 
> The problem seems to be that a lot of internal code has been
> refactored in recent versions to be "replaceable" so Hibernate ORM
> includes a default implementation for each internal Service but it
> will first check if it can find an alternative somewhere else on the
> classpath: looking both among its own dependencies and among the
> classes provided by the deployment.
> 
> I'll see if we can do better in Hibernate ORM (not sure yet!), but
> raising it here as I suspect several other libraries could be guilty
> of the same approach.
> 
> I also hope we'll be able to curate (trim) the dependencies more; the
> current JPA subsystem is including many dependencies of questionable
> usefulness. That should help?
> 
> Thanks,
> Sanne
> 
> 
> 
> On 15 May 2017 at 13:27, Rostislav Svoboda <rsvoboda at redhat.com> wrote:
>> Hi.
>> 
>> I can confirm I see improvements in boot time with your changes.
>> My HW is Lenovo T440s with Fedora 25, Intel(R) Core(TM) i7-4600U CPU (Base Frequency 2.10 GHz, Max Turbo 3.30 GHz)
>> 
>> I executed 50 iterations of start - stop sequence [1], before execution 5x start - stop for "warmup"
>> 
>> With your changes
>>       Min: 3116        Max: 3761       Average: 3247.640000
>> 
>> Without:
>>       Min: 3442        Max: 4081       Average: 3580.840000
>> 
>> 
>>> 1) A hard coded list of class names that we generate before a release
>> 
>> This will improve first boot impression, little bit harder for maintaining the list for the final build.
>> 
>> Property files could be located inside properties directory of dedicated module (<resource-root path="properties"/>). Properties directory could contain property files for delivered profiles.
>> 
>> Layered products or customer modifications could deliver own property file.
>> e.g. predefined property file for standalone-openshift.xml in EAP image in OpenShift environment, I think they boot the server just once and throw away the whole docker image when something changes.
>> 
>> 
>>> 2) Generate the list dynamically on first boot, and store it in the temp
>> 
>> This looks like the most elegant thing to do. Question is how it will slow down the initial boot. People care about first boot impression, some blog writers do the mistake too.
>> This would also block boot time improvements for use-cases when you start the server just once - e.g. Docker, OpenShift.
>> 
>> Also the logic should take into account which profile is loaded - e.g standalone.xml vs. standalone-full-ha.xml
>> 
>> Rostislav
>> 
>> [1]
>> rm wildfly-11.0.0.Beta1-SNAPSHOT-preload/standalone/log/server.log
>> rm wildfly-11.0.0.Beta1-SNAPSHOT/standalone/log/server.log
>> 
>> for i in {1..50}; do
>>  echo $i
>>  wildfly-11.0.0.Beta1-SNAPSHOT-preload/bin/standalone.sh 1>/dev/null 2>&1 &
>>  sleep 8
>>  wildfly-11.0.0.Beta1-SNAPSHOT-preload/bin/jboss-cli.sh -c :shutdown 1>/dev/null 2>&1
>> done
>> grep WFLYSRV0025 wildfly-11.0.0.Beta1-SNAPSHOT-preload/standalone/log/server.log | sed "s/.*\(....\)ms.*/\1/g"  | awk 'NR == 1 { max=$1; min=$1; sum=0 }
>>   { if ($1>max) max=$1; if ($1<min) min=$1; sum+=$1;}
>>   END {printf "Min: %d\tMax: %d\tAverage: %f\n", min, max, sum/NR}'
>> 
>> 
>> for i in {1..50}; do
>>  echo $i
>>  wildfly-11.0.0.Beta1-SNAPSHOT/bin/standalone.sh 1>/dev/null 2>&1 &
>>  sleep 8
>>  wildfly-11.0.0.Beta1-SNAPSHOT/bin/jboss-cli.sh -c :shutdown 1>/dev/null 2>&1
>> done
>> grep WFLYSRV0025 wildfly-11.0.0.Beta1-SNAPSHOT/standalone/log/server.log | sed "s/.*\(....\)ms.*/\1/g"  | awk 'NR == 1 { max=$1; min=$1; sum=0 }
>>   { if ($1>max) max=$1; if ($1<min) min=$1; sum+=$1;}
>>   END {printf "Min: %d\tMax: %d\tAverage: %f\n", min, max, sum/NR}'
>> 
>> 
>> ----- Original Message -----
>>> When JIRA was being screwy on Friday I used the time to investigate an idea I
>>> have had for a while about improving our boot time performance. According to
>>> Yourkit the majority of our time is spent in class loading. It seems very
>>> unlikely that we will be able to reduce the number of classes we load on
>>> boot (or at the very least it would be a massive amount of work) so I
>>> investigated a different approach.
>>> 
>>> I modified ModuleClassLoader to spit out the name and module of every class
>>> that is loaded at boot time, and stored this in a properties file. I then
>>> created a simple Service that starts immediately that uses two threads to
>>> eagerly load every class on this list (I used two threads because that
>>> seemed to work well on my laptop, I think Runtime.availableProcessors()/4 is
>>> probably the best amount, but that assumption would need to be tested on
>>> different hardware).
>>> 
>>> The idea behind this is that we know the classes will be used at some point,
>>> and we generally do not fully utilise all CPU's during boot, so we can use
>>> the unused CPU to pre load these classes so they are ready when they are
>>> actually required.
>>> 
>>> Using this approach I saw the boot time for standalone.xml drop from ~2.9s to
>>> ~2.3s on my laptop. The (super hacky) code I used to perform this test is at
>>> https://github.com/wildfly/wildfly-core/compare/master...stuartwdouglas:boot-performance-hack
>>> 
>>> I think these initial results are encouraging, and it is a big enough gain
>>> that I think it is worth investigating further.
>>> 
>>> Firstly it would be great if I could get others to try it out and see if they
>>> see similar gains to boot time, it may be that the gain is very system
>>> dependent.
>>> 
>>> Secondly if we do decide to do this there are two approach that we can use
>>> that I can see:
>>> 
>>> 1) A hard coded list of class names that we generate before a release
>>> (basically what the hack already does), this is simplest, but does add a
>>> little bit of additional work to the release process (although if it is
>>> missed it would be no big deal, as ClassNotFoundException's would be
>>> suppressed, and if a few classes are missing the performance impact is
>>> negligible as long as the majority of the list is correct).
>>> 
>>> 2) Generate the list dynamically on first boot, and store it in the temp
>>> directory. This would require the addition of a hook into JBoss Modules to
>>> generate the list, but is the approach I would prefer (as first boot is
>>> always a bit slower anyway).
>>> 
>>> Thoughts?
>>> 
>>> Stuart
>>> 
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> wildfly-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev

-- 
Brian Stansberry
Manager, Senior Principal Software Engineer
JBoss by Red Hat