[jboss-jira] [JBoss JIRA] Updated: (JBRULES-2067) Compilation performance for rules degrades as the classpath increases

[Edson Tirelli (JIRA)]

     [ https://jira.jboss.org/jira/browse/JBRULES-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Edson Tirelli updated JBRULES-2067:
-----------------------------------

    Fix Version/s: 5.1.0.M2
                       (was: 5.1.0.M1)


> Compilation performance for rules degrades as the classpath increases
> ---------------------------------------------------------------------
>
>                 Key: JBRULES-2067
>                 URL: https://jira.jboss.org/jira/browse/JBRULES-2067
>             Project: Drools
>          Issue Type: Bug
>      Security Level: Public(Everyone can see) 
>          Components: drools-compiler
>    Affects Versions: 5.0.0.CR1
>            Reporter: Edson Tirelli
>            Assignee: Edson Tirelli
>             Fix For: 5.1.0.M2
>
>
> REPORTED BY: Christian Nedregård 
> =============================
> This is now solved.
> I noticed that the problem became worse the bigger the classpath was, even if the extra jars contained totally irrellevant classes.
> Profiling revealed that the rule compilation generated an enomous amount of IO traffic from ClassLoader.loadClass originating from
> org.mvel.ParserContext.checkForDynamicImport and org.drools.rule.PackageCompilationData$PackageClassLoader.
> In both these places Drools generates a lot of loadClass calls for classes that does not exist.
> It seems like the Sun classloaders does not cache the result of failed class lookups, but scans the whole classpath again and again looking for these non-existing classes.
> That is why the problem gets bigger the larger your classpath is. Our classpath contains 100+ jars, so we got hit pretty bad.
> I solved the problem by creating a caching classloader that also caches the result of failed lookups. When I added this as the context classloader scoped around all my compilation I saw a performance increase of a factor of 10 - 20 in our various environments.
> Here are the stats from the caching classloader after compiling 850 rules:
> Caching loader called 200608 times
> 119 actual successful loads
> 932 actual failed loads
> 73738 cache hits for successfull loads
> 125819 cache hits for failed loads
> In other words: Drools generated 126751 loads for 932 non-existing classes. We saved 125819 full classapth scans by caching the failed lookups.
> As mentioned above the caching classloader can be hooked into the system by setting it at the context classloader.
> org.mvel.ParserContext.checkForDynamicImport uses the context classloader directly.
> org.drools.rule.PackageCompilationData$PackageClassLoader uses the classloader configured in PackageBuilderConfiguration which is the context classloader by default.
> This means you get the full effect only when you scope the caching class loader around all your PackageBuilderConfiguration and PackageBuilder usage.
> Below is the code for our caching classloader for the reference of anybody else who might encounter this problem.
> Regards
> Chr
> PS Edson:
> This problem is related to compilation. The RETE building is usually very fast.
> ----------------------------------------------
> package domain.rules;
> import java.util.HashMap;
> import java.util.Map;
> /**
>  * Classloader that intercepts calls to the current context class loader and caches the result of the calls to {@link #loadClass}
>  * on it.
>  * <p/>
>  * The interception will start automagically when this class is instantiated. Make sure that you call {@link #stop()} from a finally
>  * block when you want the loader interception to stop, to avoid unnecessary caching.
>  *
>  * @author Christian Nedregård
>  * @see Thread#getContextClassLoader()
>  */
> public class CachingContextClassLoader extends ClassLoader {
>     private Map classLoaderResultMap = new HashMap();
>     private ClassLoader originalClassLoader;
>     private int successfulCalls = 0;
>     private int failedCalls = 0;
>     private int cacheHits = 0;
>     private int successfulLoads = 0;
>     private int failedLoads = 0;
>     /**
>      * Create an instance. The instance will automagically start intercepting {@link #loadClass(String)} calls to the current
>      * context class loader.
>      *
>      * @see Thread#getContextClassLoader()
>      */
>     CachingContextClassLoader() {
>         super(Thread.currentThread().getContextClassLoader());
>         this.originalClassLoader = Thread.currentThread().getContextClassLoader();
>         Thread.currentThread().setContextClassLoader(this);
>     }
>     /**
>      * Try to load the class from the original context class loader and chache the result (also <code>ClassNotFoundException</code>s).
>      * Subsequent calls for the same class name will return the cached version, or throw a cached ClassNotFoundException if it was
>      * not found initially.
>      *
>      * @param name the name of the class to find.
>      * @return the cached result of a call to <code>loadClass</code> on the original context class loader.
>      * @throws ClassNotFoundException when the class of the given name could not be found.
>      */
>     public Class loadClass(String name) throws ClassNotFoundException {
>         Object result;
>         if (classLoaderResultMap.containsKey(name)) {
>             ++cacheHits;
>             result = classLoaderResultMap.get(name);
>         } else {
>             try {
>                 result = super.loadClass(name);
>                 ++successfulLoads;
>             } catch (ClassNotFoundException e) {
>                 ++failedLoads;
>                 result = e;
>             }
>             classLoaderResultMap.put(name, result);
>         }
>         if (result instanceof ClassNotFoundException) {
>             ++failedCalls;
>             throw (ClassNotFoundException) result;
>         } else {
>             ++successfulCalls;
>             return (Class) result;
>         }
>     }
>     /**
>      * Stop intercepting by resetting the original context classloader using {@link Thread#setContextClassLoader(ClassLoader)}.
>      */
>     public void stop() {
>         Thread.currentThread().setContextClassLoader(originalClassLoader);
>     }
>     /**
>      * Get a textual report of the caching that was performed.
>      *
>      * @return a textual report of the caching that was performed.
>      */
>     public String getReport() {
>         return new StringBuffer("Loader called ").append(successfulCalls + failedCalls).append(" times, ").append(successfulCalls)
>                 .append(" successfull, ").append(failedCalls).append(" failed. Cache hits: ").append(cacheHits)
>                 .append(", successful loads:  ").append(successfulLoads).append(", Failed loads: ").append(failedLoads)
>                 .append('.').toString();
>     }
> }

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira