Hi all,
at work we have a huge rule base (>2k rules with an average size of ~2Mib).
I've found the building this RB with the latest kie-maven-plugin
troublesome at best.
I want to share my experience hoping that some Drools developer could
help me.
I'm using the latest Drools 6.x from the master branch.
The first problem I encounterd was that the compilation process retain
in memory all the compilation caches until the end.
This would lead to a OOM very soon and we can't increase the heap size
indefinitely. So the first thing I do was to extract
the code from KieModuleMetaInfoBuilder and create a custom builder that
would populate a guava cache for every KnowledgeBase compiled.
The aforementioned cache would write the bytestreams when the objects
get evicted.
That worked well and leaded finally to a full compilation. The only
problem was the long time one has to wait until completion.
Some few rules take up to 30 min. to complete but the majority takes
from 5 to 20 secs.
So I extended the custom compiler to be multithreaded and finally the
compilation time is acceptable again.
I also extended the compiler to perform incremental compilations.
Now my problem is that the generated kjar is rather huge and when I load
it I get an OOM again.
Peeking at the code again confirm (if I get this correctly) that all the
caches gets loaded altogether, moving the problem to compile time to run
time.
Is it possible to force the KieContainer to load the caches lazily,
based on calls to getKieBase? If not, would it be usefull?
I understand that my case is rather on the edge but any help would be
appreciated.
Thank you very much :)
Federico.