[wildfly-dev] Speeding up WildFly boot time

[David M. Lloyd]

I have a few thoughts that might be of interest.

Firstly, I'd be interested to see when you are logging the class name 
being loaded.  If you are logging it in loadClass, you might not be 
seeing the actual correct load order because that method is ultimately 
recursive.  To get an accurate picture of what order that classes are 
actually defined - and thus what order you can load them in order to 
prevent contention on per-class locks within the CL - you should log 
immediately _after_ defineClass completes for each class.

Secondly, while debugging a resource iteration performance problem a 
user was having with a large number of deployments, I discovered that 
contention for the lock on JarFile and ZipFile was a primary cause.  The 
workaround I employed was to keep a RAM-based List of the files in the 
JAR, which can be iterated over without touching the lock.

When we're preloading classes, we're definitely going to see this same 
kind of contention come up, because there's only one lock per JarFile 
instance so you can only ever read one entry at a time, thus preventing 
any kind of useful concurrency on a per-module basis.

Exploding the files out of the JarFile could expose this contention and 
therefore might be useful as a test - but it would also skew the results 
a little because you have no decompression overhead, and creating the 
separate file streams hypothetically might be somewhat more (or less) 
expensive.  I joked about resurrecting jzipfile (which I killed off 
because it was something like 20% slower at decompressing entries than 
Jar/ZipFile) but it might be worth considering having our own JAR 
extractor at some point with a view towards concurrency gains.  If we go 
this route, we could go even further and create an optimized module 
format, which is an idea I think we've looked at a little bit in the 
past; there are a few avenues of exploration here which could be 
interesting.

At some point we also need to see how jaotc might improve things.  It 
probably won't improve class loading time directly, but it might improve 
the processes by which class loading is done because all the one-off 
bits would be precompiled.  Also it's worth exploring whether the jimage 
format has contention issues like this.

On 05/14/2017 06:36 PM, Stuart Douglas wrote:
> When JIRA was being screwy on Friday I used the time to investigate an 
> idea I have had for a while about improving our boot time performance. 
> According to Yourkit the majority of our time is spent in class loading. 
> It seems very unlikely that we will be able to reduce the number of 
> classes we load on boot (or at the very least it would be a massive 
> amount of work) so I investigated a different approach.
> 
> I modified ModuleClassLoader to spit out the name and module of every 
> class that is loaded at boot time, and stored this in a properties file. 
> I then created a simple Service that starts immediately that uses two 
> threads to eagerly load every class on this list (I used two threads 
> because that seemed to work well on my laptop, I think 
> Runtime.availableProcessors()/4 is probably the best amount, but that 
> assumption would need to be tested on different hardware).
> 
> The idea behind this is that we know the classes will be used at some 
> point, and we generally do not fully utilise all CPU's during boot, so 
> we can use the unused CPU to pre load these classes so they are ready 
> when they are actually required.
> 
> Using this approach I saw the boot time for standalone.xml drop from 
> ~2.9s to ~2.3s on my laptop. The (super hacky) code I used to perform 
> this test is at 
> https://github.com/wildfly/wildfly-core/compare/master...stuartwdouglas:boot-performance-hack
> 
> I think these initial results are encouraging, and it is a big enough 
> gain that I think it is worth investigating further.
> 
> Firstly it would be great if I could get others to try it out and see if 
> they see similar gains to boot time, it may be that the gain is very 
> system dependent.
> 
> Secondly if we do decide to do this there are two approach that we can 
> use that I can see:
> 
> 1) A hard coded list of class names that we generate before a release 
> (basically what the hack already does), this is simplest, but does add a 
> little bit of additional work to the release process (although if it is 
> missed it would be no big deal, as ClassNotFoundException's would be 
> suppressed, and if a few classes are missing the performance impact is 
> negligible as long as the majority of the list is correct).
> 
> 2) Generate the list dynamically on first boot, and store it in the temp 
> directory. This would require the addition of a hook into JBoss Modules 
> to generate the list, but is the approach I would prefer (as first boot 
> is always a bit slower anyway).
> 
> Thoughts?
> 
> Stuart
> 
> 
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> 




-- 
- DML