[rules-users] CompositiveClassLoader$CachingLoader$load method intermittently hangs

[mikerod]

I was finally able to take some time and find out where multiple threads
could be showing up
in the environment where my application was not doing any multithreaded work
with the Drools
knowledge base or session.

One example I have can be seen by looking at the
`org.drools.rule.constraint.MvelConstraint` class.
This method `#jitEvaluator(Object, InternalWorkingMemory, LeftTuple)` has a
line:
`ExecutorHolder.executor.execute(new ConditionJitter(this, object,
workingMemory, leftTuple));`

This is roughly around this line in 5.5.0.Final, for example.
https://github.com/droolsjbpm/drools/blob/5.5.0.Final/drools-core/src/main/java/org/drools/rule/constraint/MvelConstraint.java#L232

My understanding of this method functionality is that a
`java.util.concurrent.Executor` is used to execute a Runnable.
The Executor returned is configurable via the
`org.drools.util.ServiceRegistry`.  There is a related Drools
RuleBaseConfiguration option
called "executorService" that allows the user to configure the details of
how this Runnable is executed.
e.g. a custom application thread pool, a simple thread, no new thread at all
- synchronously.

Drools has a default for this executorService, which is
@
https://github.com/droolsjbpm/drools/blob/5.5.0.Final/drools-core/src/main/java/org/drools/concurrent/ExecutorProviderImpl.java#L11

This implementation runs the Runnable in a new thread, this is where a
concurrency issue is able to come into the picture.

So in the MvelConstraint#jitEvaluator example I mention above, uses this
executor to perform some of the Jit compilation
on MVEL constraints in a new thread, to not "burden" the current thread with
this work.
The issue is that the Runnable work that is done on another thread by the
ConditionJitter, ultimately works with the
same `org.drools.util.CompositeClassLoader$CachingLoader` as the main/other
thread(s).

This CachingLoader is not safe for concurrent access to its
`classLoaderResultMap` field, since it is implemented as a
`j.u.HashMap`.  
e.g. if one thread is reading from the cache at the same time as another
thread is doing a #put into the cache and 
      the cache resizes, this can lead to the reading going into a
non-terminating loop.

So, in summary, my application has no multithreading being done with regard
to the main thread working with the Drools
knowledge base and session.  However, when some of the knowledge base is
being compiled/Jitted there are multiple threads
doing work, and on some occasions they we run into a concurrent access to
the CachingLoader cache problem resulting in
a non-terminating loop.

I believe this would fall into the category of a "race condition" in that
I've only observed the issue happening about 1-5% of the time
of the same knowledge base being built.  I'll am being general on saying
"built" here, as I am not sure this is the correct
terminology to describe the time that MvelConstraint's are being/attempted
to be Jitted and this is happening.

I'll note that "turning off" jitting didn't fix the problem.
There is a chance that I did not effectively stop all jitting though.
I attempted to turn it off, by what is explained @
http://mvel.codehaus.org/Disabling+the+JIT+Compiler.
I cannot remember if I've tried to turn it off via "drools.permgenThreshold"
in my experiments with this issue though.
(That one is described @
http://drools.46999.n3.nabble.com/rules-users-Out-of-Mem-and-thousand-threads-Disabling-JIT-td4026694.html)

I noticed that with jitting "turned off" this method was still being called
when a MvelConstraint was found to be "dynamic"
via the `isDynamic` boolean flag of the MvelConstraint class.  I'm not clear
on what causes a constraint to be considered dynamic or not.

The workaround that *did* work for me was to just disable the CacheLoader
via the configuration property "drools.classLoaderCacheEnabled".
I do not believe I've seen significant performance degradation with this in
my use-case.
Another alternative to fixing this issue (I'd imagine) could be by
configuring the executorService to not use multiple-threads.
However, I'd assume that this is higher impact and more complicated than
disabling the CacheLoader.

I believe that using the `java.util.concurrent.ConcurrentHashMap` may solve
this particular issue.
However, I haven't done any further evaluation/analysis of the CacheLoader
class to ensure that
it is truly safe for use by multiple threads concurrently performaing reads
and writes.

I would say that the CacheLoader feature seems a bit broken right now in its
default use, due to the problem of Drools using multiple
threads internally when building the knowledge base.

I'm not sure how any of this relates to the Drools v6.x family.




--
View this message in context: http://drools.46999.n3.nabble.com/CompositiveClassLoader-CachingLoader-load-method-intermittently-hangs-tp4028780p4029597.html
Sent from the Drools: User forum mailing list archive at Nabble.com.