[rules-users] Problems compiling huge rule base with kie-maven-plugin

[Federico Bertola]

Hi all,
at work we have a huge rule base (>2k rules with an average size of ~2Mib).
I've found the building this RB with the latest kie-maven-plugin 
troublesome at best.
I want to share my experience hoping that some Drools developer could 
help me.

I'm using the latest Drools 6.x from the master branch.

The first problem I encounterd was that the compilation process retain 
in memory all the compilation caches until the end.
This would lead to a OOM very soon and we can't increase the heap size 
indefinitely. So the first thing I do was to extract
the code from KieModuleMetaInfoBuilder and create a custom builder that 
would populate a guava cache for every KnowledgeBase compiled.
The aforementioned cache would write the bytestreams when the objects 
get evicted.
That worked well and leaded finally to a full compilation. The only 
problem was the long time one has to wait until completion.
Some few rules take up to 30 min. to complete but the majority takes 
from 5 to 20 secs.
So I extended the custom compiler to be multithreaded and finally the 
compilation time is acceptable again.
I also extended the compiler to perform incremental compilations.

Now my problem is that the generated kjar is rather huge and when I load 
it I get an OOM again.
Peeking at the code again confirm (if I get this correctly) that all the 
caches gets loaded altogether, moving the problem to compile time to run 
time.
Is it possible to force the KieContainer to load the caches lazily, 
based on calls to getKieBase? If not, would it be usefull?

I understand that my case is rather on the edge but any help would be 
appreciated.

Thank you very much :)

Federico.