[wildfly-dev] Speeding up WildFly boot time

[David M. Lloyd]

Off list discussion: the NIO.2 JAR file provider appears to have 
substantially better performance (and no central locking).  This JIRA 
[1] covers using this by default within JBoss Modules.

https://issues.jboss.org/browse/MODULES-285

On 05/15/2017 10:34 AM, David M. Lloyd wrote:
> I have a few thoughts that might be of interest.
> 
> Firstly, I'd be interested to see when you are logging the class name
> being loaded.  If you are logging it in loadClass, you might not be
> seeing the actual correct load order because that method is ultimately
> recursive.  To get an accurate picture of what order that classes are
> actually defined - and thus what order you can load them in order to
> prevent contention on per-class locks within the CL - you should log
> immediately _after_ defineClass completes for each class.
> 
> Secondly, while debugging a resource iteration performance problem a
> user was having with a large number of deployments, I discovered that
> contention for the lock on JarFile and ZipFile was a primary cause.  The
> workaround I employed was to keep a RAM-based List of the files in the
> JAR, which can be iterated over without touching the lock.
> 
> When we're preloading classes, we're definitely going to see this same
> kind of contention come up, because there's only one lock per JarFile
> instance so you can only ever read one entry at a time, thus preventing
> any kind of useful concurrency on a per-module basis.
> 
> Exploding the files out of the JarFile could expose this contention and
> therefore might be useful as a test - but it would also skew the results
> a little because you have no decompression overhead, and creating the
> separate file streams hypothetically might be somewhat more (or less)
> expensive.  I joked about resurrecting jzipfile (which I killed off
> because it was something like 20% slower at decompressing entries than
> Jar/ZipFile) but it might be worth considering having our own JAR
> extractor at some point with a view towards concurrency gains.  If we go
> this route, we could go even further and create an optimized module
> format, which is an idea I think we've looked at a little bit in the
> past; there are a few avenues of exploration here which could be
> interesting.
> 
> At some point we also need to see how jaotc might improve things.  It
> probably won't improve class loading time directly, but it might improve
> the processes by which class loading is done because all the one-off
> bits would be precompiled.  Also it's worth exploring whether the jimage
> format has contention issues like this.
> 
> On 05/14/2017 06:36 PM, Stuart Douglas wrote:
>> When JIRA was being screwy on Friday I used the time to investigate an
>> idea I have had for a while about improving our boot time performance.
>> According to Yourkit the majority of our time is spent in class loading.
>> It seems very unlikely that we will be able to reduce the number of
>> classes we load on boot (or at the very least it would be a massive
>> amount of work) so I investigated a different approach.
>>
>> I modified ModuleClassLoader to spit out the name and module of every
>> class that is loaded at boot time, and stored this in a properties file.
>> I then created a simple Service that starts immediately that uses two
>> threads to eagerly load every class on this list (I used two threads
>> because that seemed to work well on my laptop, I think
>> Runtime.availableProcessors()/4 is probably the best amount, but that
>> assumption would need to be tested on different hardware).
>>
>> The idea behind this is that we know the classes will be used at some
>> point, and we generally do not fully utilise all CPU's during boot, so
>> we can use the unused CPU to pre load these classes so they are ready
>> when they are actually required.
>>
>> Using this approach I saw the boot time for standalone.xml drop from
>> ~2.9s to ~2.3s on my laptop. The (super hacky) code I used to perform
>> this test is at
>> https://github.com/wildfly/wildfly-core/compare/master...stuartwdouglas:boot-performance-hack
>>
>> I think these initial results are encouraging, and it is a big enough
>> gain that I think it is worth investigating further.
>>
>> Firstly it would be great if I could get others to try it out and see if
>> they see similar gains to boot time, it may be that the gain is very
>> system dependent.
>>
>> Secondly if we do decide to do this there are two approach that we can
>> use that I can see:
>>
>> 1) A hard coded list of class names that we generate before a release
>> (basically what the hack already does), this is simplest, but does add a
>> little bit of additional work to the release process (although if it is
>> missed it would be no big deal, as ClassNotFoundException's would be
>> suppressed, and if a few classes are missing the performance impact is
>> negligible as long as the majority of the list is correct).
>>
>> 2) Generate the list dynamically on first boot, and store it in the temp
>> directory. This would require the addition of a hook into JBoss Modules
>> to generate the list, but is the approach I would prefer (as first boot
>> is always a bit slower anyway).
>>
>> Thoughts?
>>
>> Stuart
>>
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
> 
> 
> 
> 


-- 
- DML