Re: [rules-users] Drools memory consumption

Wolfgang, Thanks again for your responses. I have changed the rule as you recommended me. The main change is using "over window:time" rather than "this after". The heap size still gets to 4 GB. The new drl looks like that (for the port scan event): -------------------------------------------------------------------- package com.checkpoint.correlation.impl.drools.package30; import java.util.Date import java.util.HashMap import java.util.HashSet import java.util.Collection import java.util.ArrayList import com.checkpoint.correlation.impl.drools.Log import com.checkpoint.correlation.impl.drools.CorrelatedEvent global com.checkpoint.correlation.server.EventsHandler externalEventsHandler; global com.checkpoint.correlation.impl.drools.future.Debugger debugger; import function com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInDayHourRange import function com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInIpRange function boolean filter(Log log) { return (!((log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "10.80.0.0", "10.80.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "124.0.0.0", "124.255.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "192.168.0.0", "192.168.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "195.158.7.0", "195.158.7.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "11.25.0.0", "11.25.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "128.157.0.0", "128.157.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "213.114.0.0", "213.114.255.255")))); } function Collection getMarkers(Collection matchedLogs) { ArrayList<String> markers = new ArrayList<String>(); HashSet<String> idSet = new HashSet<String>(); for (Object matchedLogObj : matchedLogs) { Log matchedLog = (Log) matchedLogObj; String id = getUniqueId(matchedLog); if (!idSet.contains(id)) { idSet.add(id); markers.add(matchedLog.fieldsMap.get("marker").toString()); if (markers.size() == 25) break; } } return markers; } function String calcSeverity(Log log) { return "High"; } function String getUniqueId(Log log) { String uniqueId=""; uniqueId += (log.fieldsMap.get("service") != null ? log.fieldsMap.get("service").toString() : "null"); uniqueId += (log.fieldsMap.get("proto") != null ? log.fieldsMap.get("proto").toString() : "null"); return uniqueId; } declare Log @role(event) end declare CorrelatedEvent @role(event) @expires(600s) end rule "Port scan from external network" enabled true dialect "java" no-loop when $log : Log(eval(filter($log))) not CorrelatedEvent(getId() == "{8AC52BA8-1EE8-4f18-9BB4-54492116501C}", groupByFieldsMap.get("src") == $log.fieldsMap.get("src"), groupByFieldsMap.get("dst") == $log.fieldsMap.get("dst")) accumulate($accumulatedLog : Log(eval(filter($accumulatedLog)), fieldsMap.get("src") == $log.fieldsMap.get("src"), fieldsMap.get("dst") == $log.fieldsMap.get("dst"), $id : getUniqueId(this)) over window:time(60s); $matchedLogs: collectList($accumulatedLog), $idSet : collectSet($id); $idSet.size > 19) then Collection markers = getMarkers($matchedLogs); CorrelatedEvent $ce = new CorrelatedEvent("{8AC52BA8-1EE8-4f18-9BB4-54492116501C}"); $ce.groupByFieldsMap.put("src", $log.fieldsMap.get("src")); $ce.groupByFieldsMap.put("dst", $log.fieldsMap.get("dst")); insert($ce); HashMap<String,Object> fieldsMap = new HashMap<String,Object>(); fieldsMap.put("cu_rule_id", "{8AC52BA8-1EE8-4f18-9BB4-54492116501C}"); fieldsMap.put("event_name", "Port scan from external network"); fieldsMap.put("cu_rule_severity", calcSeverity($log)); fieldsMap.put("cu_rule_category", "Scans"); fieldsMap.put("cu_log_count", markers.size()); fieldsMap.put("time", new Date()); fieldsMap.put("cu_markers_list", markers); fieldsMap.put("src", $log.fieldsMap.get("src")); fieldsMap.put("src_machine_name", $log.fieldsMap.get("src_machine_name")); fieldsMap.put("src_user_name", $log.fieldsMap.get("src_user_name")); fieldsMap.put("dst", $log.fieldsMap.get("dst")); fieldsMap.put("dst_machine_name", $log.fieldsMap.get("dst_machine_name")); fieldsMap.put("dst_user_name", $log.fieldsMap.get("dst_user_name")); fieldsMap.put("service", $log.fieldsMap.get("service")); fieldsMap.put("proto", $log.fieldsMap.get("proto")); fieldsMap.put("product", $log.fieldsMap.get("product")); externalEventsHandler.handleEvent(fieldsMap); end --------------------------------------------------------------------------------- I have tested it by sending logs in a rate of about 200 logs/sec with my 125 rules. The class consuming most memory is still "FromNodeLeftTuple". I have logged Drools activity. The compressed log file size is 750KB. How can I send it to you? Thank you very much. -----Original Message----- From: rules-users-bounces(a)lists.jboss.org [mailto:rules-users-bounces@lists.jboss.org] On Behalf Of Wolfgang Laun Sent: Monday, October 28, 2013 7:52 PM To: Rules Users List Subject: Re: [rules-users] Drools memory consumption On 28/10/2013, Elran Dvir <elrand(a)checkpoint.com&gt; wrote: I think that window:time would be more appropriate. Or implement an equivalent functionality using an auxiliary container. There's no reason why it should be empty if $idSet.size > 0. -W _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users Email secured by Check Point