[wildfly-dev] Speeding up WildFly boot time

[Stuart Douglas]

On Tue, May 16, 2017 at 1:34 AM, David M. Lloyd <david.lloyd at redhat.com>
wrote:

> I have a few thoughts that might be of interest.
>
> Firstly, I'd be interested to see when you are logging the class name
> being loaded.  If you are logging it in loadClass, you might not be
> seeing the actual correct load order because that method is ultimately
> recursive.  To get an accurate picture of what order that classes are
> actually defined - and thus what order you can load them in order to
> prevent contention on per-class locks within the CL - you should log
> immediately _after_ defineClass completes for each class.
>

I set a breakpoint in loadClassLocal to print off the information.


>
> Secondly, while debugging a resource iteration performance problem a
> user was having with a large number of deployments, I discovered that
> contention for the lock on JarFile and ZipFile was a primary cause.  The
> workaround I employed was to keep a RAM-based List of the files in the
> JAR, which can be iterated over without touching the lock.
>
> When we're preloading classes, we're definitely going to see this same
> kind of contention come up, because there's only one lock per JarFile
> instance so you can only ever read one entry at a time, thus preventing
> any kind of useful concurrency on a per-module basis.
>

I think this is why I see an even bigger gain when pre-loading classes one
module at a time.


>
> Exploding the files out of the JarFile could expose this contention and
> therefore might be useful as a test - but it would also skew the results
> a little because you have no decompression overhead, and creating the
> separate file streams hypothetically might be somewhat more (or less)
> expensive.  I joked about resurrecting jzipfile (which I killed off
> because it was something like 20% slower at decompressing entries than
> Jar/ZipFile) but it might be worth considering having our own JAR
> extractor at some point with a view towards concurrency gains.  If we go
> this route, we could go even further and create an optimized module
> format, which is an idea I think we've looked at a little bit in the
> past; there are a few avenues of exploration here which could be
> interesting.
>

This could be worth investigating.

Stuart


>
> At some point we also need to see how jaotc might improve things.  It
> probably won't improve class loading time directly, but it might improve
> the processes by which class loading is done because all the one-off
> bits would be precompiled.  Also it's worth exploring whether the jimage
> format has contention issues like this.
>
> On 05/14/2017 06:36 PM, Stuart Douglas wrote:
> > When JIRA was being screwy on Friday I used the time to investigate an
> > idea I have had for a while about improving our boot time performance.
> > According to Yourkit the majority of our time is spent in class loading.
> > It seems very unlikely that we will be able to reduce the number of
> > classes we load on boot (or at the very least it would be a massive
> > amount of work) so I investigated a different approach.
> >
> > I modified ModuleClassLoader to spit out the name and module of every
> > class that is loaded at boot time, and stored this in a properties file.
> > I then created a simple Service that starts immediately that uses two
> > threads to eagerly load every class on this list (I used two threads
> > because that seemed to work well on my laptop, I think
> > Runtime.availableProcessors()/4 is probably the best amount, but that
> > assumption would need to be tested on different hardware).
> >
> > The idea behind this is that we know the classes will be used at some
> > point, and we generally do not fully utilise all CPU's during boot, so
> > we can use the unused CPU to pre load these classes so they are ready
> > when they are actually required.
> >
> > Using this approach I saw the boot time for standalone.xml drop from
> > ~2.9s to ~2.3s on my laptop. The (super hacky) code I used to perform
> > this test is at
> > https://github.com/wildfly/wildfly-core/compare/master...
> stuartwdouglas:boot-performance-hack
> >
> > I think these initial results are encouraging, and it is a big enough
> > gain that I think it is worth investigating further.
> >
> > Firstly it would be great if I could get others to try it out and see if
> > they see similar gains to boot time, it may be that the gain is very
> > system dependent.
> >
> > Secondly if we do decide to do this there are two approach that we can
> > use that I can see:
> >
> > 1) A hard coded list of class names that we generate before a release
> > (basically what the hack already does), this is simplest, but does add a
> > little bit of additional work to the release process (although if it is
> > missed it would be no big deal, as ClassNotFoundException's would be
> > suppressed, and if a few classes are missing the performance impact is
> > negligible as long as the majority of the list is correct).
> >
> > 2) Generate the list dynamically on first boot, and store it in the temp
> > directory. This would require the addition of a hook into JBoss Modules
> > to generate the list, but is the approach I would prefer (as first boot
> > is always a bit slower anyway).
> >
> > Thoughts?
> >
> > Stuart
> >
> >
> > _______________________________________________
> > wildfly-dev mailing list
> > wildfly-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/wildfly-dev
> >
>
>
>
>
> --
> - DML
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/wildfly-dev/attachments/20170516/db8f7c2a/attachment-0001.html