[jboss-jira] [JBoss JIRA] (JBRULES-2782) Potential memory leak when rules are often reloaded

Potential memory leak when rules are often reloaded --------------------------------------------------- Key: JBRULES-2782 URL: https://issues.jboss.org/browse/JBRULES-2782 Project: Drools Issue Type: Bug Security Level: Public(Everyone can see) Components: drools-core Affects Versions: 5.1.1.FINAL Environment: Windows 7 x64; JVM: Hotspot; JRE build 1.6.0_22-b04 and JRE build 1.5.0_11-b03 Reporter: Konstantin Pentarakis Assignee: Mark Proctor Fix For: 5.4.0.Beta2 Hi! There could be a small heap memory leak. I encountered the problem when I tried to reload rules very often. The use of heap is permanently growing, eventhough the number of rules is constant! Since the heapsize grows very slowly, this leads to an OutOfMemoryError only, if I have a very large number of rules or if I reload the rules very often. (code-example see below) What do I mean by "reloading rules"? I add rules to an existing KnowledgeBase which is hold in memory. The rules come from an external source. For example, I add 100 rules into a KnowledgeBase (first run). Now, I redo this and add the same 100 rules again to the KnowledgeBase (2nd run). And so on. If I try this using a reduced heapsize of 64 MB, this works fine 71 times! But the 72nd time leads to an OutOfMemoryError, because there is no free heapsize left! It doesn't metter whether I remove the previously added rules first and add them afterwords again, or I directly add them. My test-ruleset has no chaining at all! But I have to mention that the rules have quit a lot conditions ... each rule has 18 different conditions. This seems to doesn't happen with very simple rules like: rule "rule1" when then end; I'm currently trying to find out what a rule must have to cause this. I'll add this information as soon as I know it. It seems to has something to do with the number of conditions! I first tested with very much conditions ... each rule has 18 different conditions. With so much conditions, this effect occures much faster! With simple rules, it happens much slower. For example, when I tested this with 10 simple rules like this: rule "rule1" when then end; ... more than 10.000 iterations were possible!!! So - 10.000 x 10 rule changes - this is quit a lot at only 64 MB. This is no problem at all! But: When I tested using my complex rules with 18 conditions per rule, I could only have about 280 iterations! - So, only 2800 rule changes. Using a profiler, I can see, that the heap-consumption constandly increases! - The more rules I use, the less interations are possible (closley linearly). - The more max. heapsize I use, the more iterations are possible (of course). - The more conditions the rules have, the less iterations are possible. It seams like the memory is hold by the RETE itself. The KnowledgeBase itself holds the references! If all references to the KnowledgeBase are deleted, the garbage collections frees the lost memory! I also tried to use sequential mode. But there was no difference at all! This happens with JRE 1.5 and JRE 1.6. This also happens with the Drools 5.1.1 KnowledgeBase-Implemantion and the still in Drools 5 existing RuleBase-Implemention. But I didn't tested it yet with previous versions of Drools. Why do I test this? We are building an application where rules will be created programmatically. There will be a large number of rules (some thousands) and rules can often change. The reload of rules which I tested here is something like a "simulation" of lots of rule changes. Here is my example. For this example I used a simple Class called "RuleRepository" which has a static String-Array holding the rules as DRL-Strings. I also tried it with files in filesystem - no difference. See the RuleRepository-Class below. MassivRuleChangeTest.java: import java.io.IOException; import java.util.Collection; import org.drools.KnowledgeBase; import org.drools.KnowledgeBaseFactory; import org.drools.builder.KnowledgeBuilder; import org.drools.builder.KnowledgeBuilderFactory; import org.drools.builder.ResourceType; import org.drools.compiler.DroolsParserException; import org.drools.definition.KnowledgePackage; import org.drools.io.ResourceFactory; public class MassivRuleChangeTest { public void run() throws Exception { KnowledgeBase kBase = createRuleBase(); Collection<KnowledgePackage> packages; long counter = 0; int delCounter = 0; int ruleCounter = 0; while (true) { System.out.println(++counter + ":"); for(int iLoop = 0; iLoop < RuleRepository.getSize(); iLoop++) { packages = this.getRulePackage(RuleRepository.getRule(iLoop)); delCounter = this.cleanKBase(kBase, packages, delCounter); kBase.addKnowledgePackages(packages); ruleCounter++; } System.out.println(" - Deleted rules: " + delCounter); System.out.println(" - Added rules: " + ruleCounter); delCounter = 0; ruleCounter = 0; } } private int cleanKBase(KnowledgeBase kBase, Collection<KnowledgePackage> packages, int delCounter) { if(kBase.getKnowledgePackages().size() > 0) { for(KnowledgePackage kPkg : packages) { if(kBase.getKnowledgePackage(kPkg.getName()) != null && kBase.getKnowledgePackage(kPkg.getName()).getRules().size() > 0) { for(org.drools.definition.rule.Rule rule : kPkg.getRules()) { if(kBase.getRule(kPkg.getName(), rule.getName()) != null) { kBase.removeRule(kPkg.getName(), rule.getName()); delCounter++; } } if(kPkg.getRules().size() == 0) kBase.removeKnowledgePackage(kPkg.getName()); } } } return delCounter; } private Collection<KnowledgePackage> getRulePackage(String rule) throws DroolsParserException, IOException { KnowledgeBuilder builder = KnowledgeBuilderFactory.newKnowledgeBuilder(); builder.add(ResourceFactory.newByteArrayResource(rule.getBytes()), ResourceType.DRL); if (builder.hasErrors()) { System.out.println(builder.getErrors().toString()); throw new RuntimeException("Unable to compile " + rule); } return builder.getKnowledgePackages(); } private KnowledgeBase createRuleBase() throws Exception { KnowledgeBase kBase = KnowledgeBaseFactory.newKnowledgeBase(); return kBase; } public static void main(String[] args) throws Exception { new MassivRuleChangeTest().run(); } } ---------------------------------------------------------------------------------------------------------------------------------- RuleRepository.java: public class RuleRepository { private static final String[] rules = { "package testpkg2 import BOM.*; rule \"rule_1\" when Article ((((name == \"N00\") && (status == \"active\") && (manufacturer == \"Man1\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"DE\")) || ((name == \"N00\") && (status == \"active\") && (manufacturer == \"Man1\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"AT\"))) || ((name == \"N00\") && (status == \"active\") && (manufacturer == \"Man1\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"GR\"))) then Description desc = new Description(); desc.setName(\"article1\"); end", "package testpkg2 import BOM.*; rule \"rule_2\" when Article ((((name == \"N01\") && (status == \"active\") && (manufacturer == \"Man2\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"DE\")) || ((name == \"N01\") && (status == \"active\") && (manufacturer == \"Man2\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"AT\"))) || ((name == \"N01\") && (status == \"active\") && (manufacturer == \"Man2\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"GR\"))) then Description desc = new Description(); desc.setName(\"article2\"); end", "package testpkg2 import BOM.*; rule \"rule_3\" when Article ((((name == \"N02\") && (status == \"active\") && (manufacturer == \"Man3\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"DE\")) || ((name == \"N02\") && (status == \"active\") && (manufacturer == \"Man3\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"AT\"))) || ((name == \"N02\") && (status == \"active\") && (manufacturer == \"Man3\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"GR\"))) then Description desc = new Description(); desc.setName(\"article3\"); end", "package testpkg2 import BOM.*; rule \"rule_4\" when Article ((((name == \"N03\") && (status == \"active\") && (manufacturer == \"Man4\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"DE\")) || ((name == \"N03\") && (status == \"active\") && (manufacturer == \"Man4\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"AT\"))) || ((name == \"N03\") && (status == \"active\") && (manufacturer == \"Man4\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"GR\"))) then Description desc = new Description(); desc.setName(\"article4\"); end", "package testpkg2 import BOM.*; rule \"rule_5\" when Article ((((name == \"N04\") && (status == \"active\") && (manufacturer == \"Man5\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"DE\")) || ((name == \"N04\") && (status == \"active\") && (manufacturer == \"Man5\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"AT\"))) || ((name == \"N04\") && (status == \"active\") && (manufacturer == \"Man5\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"GR\"))) then Description desc = new Description(); desc.setName(\"article5\"); end", "package testpkg2 import BOM.*; rule \"rule_6\" when Article ((((name == \"N05\") && (status == \"active\") && (manufacturer == \"Man6\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"DE\")) || ((name == \"N05\") && (status == \"active\") && (manufacturer == \"Man6\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"AT\"))) || ((name == \"N05\") && (status == \"active\") && (manufacturer == \"Man6\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"GR\"))) then Description desc = new Description(); desc.setName(\"article6\"); end", "package testpkg2 import BOM.*; rule \"rule_7\" when Article ((((name == \"N06\") && (status == \"active\") && (manufacturer == \"Man7\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"DE\")) || ((name == \"N06\") && (status == \"active\") && (manufacturer == \"Man7\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"AT\"))) || ((name == \"N06\") && (status == \"active\") && (manufacturer == \"Man7\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"GR\"))) then Description desc = new Description(); desc.setName(\"article7\"); end", "package testpkg2 import BOM.*; rule \"rule_8\" when Article ((((name == \"N07\") && (status == \"active\") && (manufacturer == \"Man8\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"DE\")) || ((name == \"N07\") && (status == \"active\") && (manufacturer == \"Man8\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"AT\"))) || ((name == \"N07\") && (status == \"active\") && (manufacturer == \"Man8\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"GR\"))) then Description desc = new Description(); desc.setName(\"article8\"); end", "package testpkg2 import BOM.*; rule \"rule_9\" when Article ((((name == \"N08\") && (status == \"active\") && (manufacturer == \"Man9\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"DE\")) || ((name == \"N08\") && (status == \"active\") && (manufacturer == \"Man9\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"AT\"))) || ((name == \"N08\") && (status == \"active\") && (manufacturer == \"Man9\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"GR\"))) then Description desc = new Description(); desc.setName(\"article9\"); end", "package testpkg2 import BOM.*; rule \"rule_10\" when Article ((((name == \"N09\") && (status == \"active\") && (manufacturer == \"Man10\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"DE\")) || ((name == \"N09\") && (status == \"active\") && (manufacturer == \"Man10\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"AT\"))) || ((name == \"N09\") && (status == \"active\") && (manufacturer == \"Man10\") && (category == \"cell phones/phones\") && (commodityGroup == \"cell phones\") && (countryCode == \"GR\"))) then Description desc = new Description(); desc.setName(\"article10\"); end" }; public static String getRule(int index) { return rules[index]; } public static int getSize() { return rules.length; } } ---------------------------------------------------------------------------------------------------------------------------------- Artilcle.java: package BOM; public class Article { private String productID; private String status; private String name; private String manufacturer; private String type; private String category; private String commodityGroup; private String countryCode; private String EAN; private String manufacturerPartNumber; private String supplierName; private String creationDate; private String changeDate; private boolean isActive; private int length; private int height; private int width; public String getName() { return name; } public void setName(String name) { this.name = name; } public String getManufacturer() { return manufacturer; } public void setManufacturer(String manufacturer) { this.manufacturer = manufacturer; } public int getLength() { return length; } public void setLength(int length) { this.length = length; } public int getHeight() { return height; } public void setHeight(int height) { this.height = height; } public int getWidth() { return width; } public void setWidth(int width) { this.width = width; } public String getType() { return type; } public void setType(String type) { this.type = type; } public String getCategory() { return category; } public void setCategory(String category) { this.category = category; } public String getProductID() { return productID; } public void setProductID(String productID) { this.productID = productID; } public String getStatus() { return status; } public void setStatus(String status) { this.status = status; } public String getCommodityGroup() { return commodityGroup; } public void setCommodityGroup(String commodityGroup) { this.commodityGroup = commodityGroup; } public String getCountryCode() { return countryCode; } public void setCountryCode(String countryCode) { this.countryCode = countryCode; } public String getEAN() { return EAN; } public void setEAN(String ean) { EAN = ean; } public String getManufacturerPartNumber() { return manufacturerPartNumber; } public void setManufacturerPartNumber(String manufacturerPartNumber) { this.manufacturerPartNumber = manufacturerPartNumber; } public String getSupplierName() { return supplierName; } public void setSupplierName(String supplierName) { this.supplierName = supplierName; } public String getCreationDate() { return creationDate; } public void setCreationDate(String creationDate) { this.creationDate = creationDate; } public String getChangeDate() { return changeDate; } public void setChangeDate(String changeDate) { this.changeDate = changeDate; } public boolean isActive() { return isActive; } public void setActive(boolean isActive) { this.isActive = isActive; } } ---------------------------------------------------------------------------------------------------------------------------------- Description.java: package BOM; public class Description { private String name; private String type; public String getName() { return name; } public void setName(String name) { this.name = name; } public String getType() { return type; } public void setType(String type) { this.type = type; } }