Re: [rules-users] Drools memory consumption

I am sending this message again because maybe the last wasn’t sent because of the attached snapshots.**** I removed them now.**** Thanks.**** ** ** Hi all,**** ** ** I am using Drools Fusion. I am getting OutOfMemoryError rather fast. My JVM is running with –Xmx4g flag.**** I have rules defined in another (not Drools) language.**** Every rule is translated programmatically to a drl file. This is because the user can add and remove rules (in the other language) dynamically.**** The default configuration contains 125 rules.**** ** ** For example, one rule is supposed to identify a port scan event.**** The basic fact is connection log. For each combination of src (source IP) and dst (destination IP) , detect a port scan event, if over 60 seconds there were at least 20 connection logs with different service and protocol. **** The event will stay closed for 10 minute - no event will be sent during this time for this combination of src and dst. The event the connection logs’ ids (markers).**** (other rules are very similar in structure, but different in logic, of course) **** ** ** This is its programmatic drl file:**** ** ** package com.checkpoint.correlation.impl.drools.package30;**** ** ** import java.util.Date**** import java.util.HashMap**** import java.util.Set**** import com.checkpoint.correlation.impl.drools.Log**** import com.checkpoint.correlation.impl.drools.CorrelatedEvent**** ** ** global com.checkpoint.correlation.server.EventsHandler externalEventsHandler;**** ** ** import function com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInDayHourRange **** import function com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInIpRange**** ** ** function boolean filter(Log log) { **** return (!((log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "10.80.0.0", "10.80.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "124.0.0.0", "124.255.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "192.168.0.0", "192.168.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "195.158.7.0", "195.158.7.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "11.25.0.0", "11.25.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "128.157.0.0", "128.157.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "213.114.0.0", "213.114.255.255"))));**** }**** ** ** function String markersToString(Set markersSet) { **** int i = 0;**** String markersString = "";**** for (Object marker : markersSet) { **** if (i == 25) break;**** String markerStr = marker.toString();**** if (i > 0) markersString += "\n";**** markersString += markerStr;**** }**** return markersString;**** }**** ** ** function String calcSeverity(Log log) { **** return "High";**** }**** ** ** function String getUniqueId(Log log) { **** String uniqueId="";**** uniqueId += (log.fieldsMap.get("service") != null ? log.fieldsMap.get("service").toString() : "null");**** uniqueId += (log.fieldsMap.get("proto") != null ? log.fieldsMap.get("proto").toString() : "null");**** return uniqueId;**** }**** **** declare Log**** @role(event)**** end**** ** ** declare CorrelatedEvent**** @role(event)**** @expires(600s)**** end**** ** ** rule "Port scan from external network"**** enabled true**** dialect "java"**** no-loop**** when**** $log : Log(eval(filter($log)))**** not CorrelatedEvent(getId() == "{8AC52BA8-1EE8-4f18-9BB4-54492116501C}", groupByFieldsMap.get("src") == $log.fieldsMap.get("src"), groupByFieldsMap.get("dst") == $log.fieldsMap.get("dst"))**** accumulate($accumulatedLog : Log(eval(filter($accumulatedLog)), this after[0s,60s] $log, fieldsMap.get("src") == $log.fieldsMap.get("src"), fieldsMap.get("dst") == $log.fieldsMap.get("dst"), $id : getUniqueId(this));**** $idSet : collectSet($id);**** $idSet.size > 19)**** accumulate($accumulatedLog : Log(eval(filter($accumulatedLog)), this after[0s,60s] $log, fieldsMap.get("src") == $log.fieldsMap.get("src"), fieldsMap.get("dst") == $log.fieldsMap.get("dst"), $idSet.contains(getUniqueId(this)), $marker : fieldsMap.get("marker"));**** $markerSet : collectSet($marker))**** then**** CorrelatedEvent $ce = new CorrelatedEvent("{8AC52BA8-1EE8-4f18-9BB4-54492116501C}");**** $ce.groupByFieldsMap.put("src", $log.fieldsMap.get("src")); **** $ce.groupByFieldsMap.put("dst", $log.fieldsMap.get("dst")); **** insert($ce);**** HashMap<String,Object> fieldsMap = new HashMap<String,Object>();**** fieldsMap.put("cu_rule_id", "{8AC52BA8-1EE8-4f18-9BB4-54492116501C}");**** fieldsMap.put("event_name", "Port scan from external network");**** fieldsMap.put("cu_rule_severity", calcSeverity($log));**** fieldsMap.put("cu_rule_category", "Scans");**** fieldsMap.put("cu_log_count", $markerSet.size());**** fieldsMap.put("time", new Date());**** fieldsMap.put("cu_markers_list", markersToString($markerSet));**** fieldsMap.put("src", $log.fieldsMap.get("src"));**** fieldsMap.put("src_machine_name", $log.fieldsMap.get("src_machine_name"));**** fieldsMap.put("src_user_name", $log.fieldsMap.get("src_user_name"));**** fieldsMap.put("dst", $log.fieldsMap.get("dst"));**** fieldsMap.put("dst_machine_name", $log.fieldsMap.get("dst_machine_name"));**** fieldsMap.put("dst_user_name", $log.fieldsMap.get("dst_user_name"));**** fieldsMap.put("service", $log.fieldsMap.get("service"));** ** fieldsMap.put("proto", $log.fieldsMap.get("proto"));**** fieldsMap.put("product", $log.fieldsMap.get("product"));** ** externalEventsHandler.handleEvent(fieldsMap);**** end **** ** ** I am sending logs in a rate of up to 200 logs/sec. After about 3 minutes, my application starts to be unresponsive. **** I monitored the JVM with VisualVM. Two snapshots of VisualVM are attached. **** I found out that the class consuming most memory is FromNodeLeftTuple of drools (as can be seen in “instances.png”).**** ** ** **1) **Is my inserting rate is too high?**** **2) **Is There a way I can make my rules more memory efficient?**** ** ** Thanks.**** ** ** Inserting logs:**** public void insertEvents(Collection<Map<String, Object>> logs)**** {**** for (Map<String, Object> map : logs) {**** Log log = new Log();**** Log.fieldsMap.putAll(map);**** session.insert(log);**** session.fireAllRules();**** }**** }**** ** ** Log class:**** public class Log**** {**** public HashMap<String, Object> fieldsMap = new HashMap<>();**** }**** ** ** CorrelatedEvent class:**** ** ** public class CorrelatedEvent**** {**** public Map<String, Object> groupByFieldsMap;**** ** ** private String id;**** ** ** public CorrelatedEvent(String id)**** {**** groupByFieldsMap = new HashMap<>();**** this.id = id;**** }**** ** ** public String getId()**** {**** return id;**** }**** }**** ** ** ** ** _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users