<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 16, 2017 at 1:34 AM, David M. Lloyd <span dir="ltr"><<a href="mailto:david.lloyd@redhat.com" target="_blank">david.lloyd@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I have a few thoughts that might be of interest.<br>
<br>
Firstly, I'd be interested to see when you are logging the class name<br>
being loaded. If you are logging it in loadClass, you might not be<br>
seeing the actual correct load order because that method is ultimately<br>
recursive. To get an accurate picture of what order that classes are<br>
actually defined - and thus what order you can load them in order to<br>
prevent contention on per-class locks within the CL - you should log<br>
immediately _after_ defineClass completes for each class.<br></blockquote><div><br></div><div>I set a breakpoint in loadClassLocal to print off the information.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Secondly, while debugging a resource iteration performance problem a<br>
user was having with a large number of deployments, I discovered that<br>
contention for the lock on JarFile and ZipFile was a primary cause. The<br>
workaround I employed was to keep a RAM-based List of the files in the<br>
JAR, which can be iterated over without touching the lock.<br>
<br>
When we're preloading classes, we're definitely going to see this same<br>
kind of contention come up, because there's only one lock per JarFile<br>
instance so you can only ever read one entry at a time, thus preventing<br>
any kind of useful concurrency on a per-module basis.<br></blockquote><div><br></div><div>I think this is why I see an even bigger gain when pre-loading classes one module at a time.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Exploding the files out of the JarFile could expose this contention and<br>
therefore might be useful as a test - but it would also skew the results<br>
a little because you have no decompression overhead, and creating the<br>
separate file streams hypothetically might be somewhat more (or less)<br>
expensive. I joked about resurrecting jzipfile (which I killed off<br>
because it was something like 20% slower at decompressing entries than<br>
Jar/ZipFile) but it might be worth considering having our own JAR<br>
extractor at some point with a view towards concurrency gains. If we go<br>
this route, we could go even further and create an optimized module<br>
format, which is an idea I think we've looked at a little bit in the<br>
past; there are a few avenues of exploration here which could be<br>
interesting.<br></blockquote><div><br></div><div>This could be worth investigating. </div><div><br></div><div>Stuart</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
At some point we also need to see how jaotc might improve things. It<br>
probably won't improve class loading time directly, but it might improve<br>
the processes by which class loading is done because all the one-off<br>
bits would be precompiled. Also it's worth exploring whether the jimage<br>
format has contention issues like this.<br>
<div class="HOEnZb"><div class="h5"><br>
On 05/14/2017 06:36 PM, Stuart Douglas wrote:<br>
> When JIRA was being screwy on Friday I used the time to investigate an<br>
> idea I have had for a while about improving our boot time performance.<br>
> According to Yourkit the majority of our time is spent in class loading.<br>
> It seems very unlikely that we will be able to reduce the number of<br>
> classes we load on boot (or at the very least it would be a massive<br>
> amount of work) so I investigated a different approach.<br>
><br>
> I modified ModuleClassLoader to spit out the name and module of every<br>
> class that is loaded at boot time, and stored this in a properties file.<br>
> I then created a simple Service that starts immediately that uses two<br>
> threads to eagerly load every class on this list (I used two threads<br>
> because that seemed to work well on my laptop, I think<br>
> Runtime.availableProcessors()/<wbr>4 is probably the best amount, but that<br>
> assumption would need to be tested on different hardware).<br>
><br>
> The idea behind this is that we know the classes will be used at some<br>
> point, and we generally do not fully utilise all CPU's during boot, so<br>
> we can use the unused CPU to pre load these classes so they are ready<br>
> when they are actually required.<br>
><br>
> Using this approach I saw the boot time for standalone.xml drop from<br>
> ~2.9s to ~2.3s on my laptop. The (super hacky) code I used to perform<br>
> this test is at<br>
> <a href="https://github.com/wildfly/wildfly-core/compare/master...stuartwdouglas:boot-performance-hack" rel="noreferrer" target="_blank">https://github.com/wildfly/<wbr>wildfly-core/compare/master...<wbr>stuartwdouglas:boot-<wbr>performance-hack</a><br>
><br>
> I think these initial results are encouraging, and it is a big enough<br>
> gain that I think it is worth investigating further.<br>
><br>
> Firstly it would be great if I could get others to try it out and see if<br>
> they see similar gains to boot time, it may be that the gain is very<br>
> system dependent.<br>
><br>
> Secondly if we do decide to do this there are two approach that we can<br>
> use that I can see:<br>
><br>
> 1) A hard coded list of class names that we generate before a release<br>
> (basically what the hack already does), this is simplest, but does add a<br>
> little bit of additional work to the release process (although if it is<br>
> missed it would be no big deal, as ClassNotFoundException's would be<br>
> suppressed, and if a few classes are missing the performance impact is<br>
> negligible as long as the majority of the list is correct).<br>
><br>
> 2) Generate the list dynamically on first boot, and store it in the temp<br>
> directory. This would require the addition of a hook into JBoss Modules<br>
> to generate the list, but is the approach I would prefer (as first boot<br>
> is always a bit slower anyway).<br>
><br>
> Thoughts?<br>
><br>
> Stuart<br>
><br>
><br>
</div></div><span class="im HOEnZb">> ______________________________<wbr>_________________<br>
> wildfly-dev mailing list<br>
> <a href="mailto:wildfly-dev@lists.jboss.org">wildfly-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/wildfly-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/wildfly-dev</a><br>
><br>
<br>
<br>
<br>
<br>
--<br>
</span><span class="HOEnZb"><font color="#888888">- DML<br>
</font></span><div class="HOEnZb"><div class="h5">______________________________<wbr>_________________<br>
wildfly-dev mailing list<br>
<a href="mailto:wildfly-dev@lists.jboss.org">wildfly-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/wildfly-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/wildfly-dev</a><br>
</div></div></blockquote></div><br></div></div>