[rules-users] Drools memory consumption
Matteo Cusmai
cusmaimatteo at gmail.com
Thu Oct 24 02:32:12 EDT 2013
Are you using any entrypoint?
Time or length based?
It could be possible you have to retract manually the events.
On 24 Oct 2013 08:23, "Elran Dvir" <elrand at checkpoint.com> wrote:
> I am sending this message again because maybe the last wasn’t sent
> because of the attached snapshots.****
>
> I removed them now.****
>
> Thanks.****
>
> ** **
>
> Hi all,****
>
> ** **
>
> I am using Drools Fusion. I am getting OutOfMemoryError rather fast. My
> JVM is running with –Xmx4g flag.****
>
> I have rules defined in another (not Drools) language.****
>
> Every rule is translated programmatically to a drl file. This is because
> the user can add and remove rules (in the other language) dynamically.****
>
> The default configuration contains 125 rules.****
>
> ** **
>
> For example, one rule is supposed to identify a port scan event.****
>
> The basic fact is connection log. For each combination of src (source IP)
> and dst (destination IP) , detect a port scan event, if over 60 seconds
> there were at least 20 connection logs with different service and protocol.
> ****
>
> The event will stay closed for 10 minute - no event will be sent during
> this time for this combination of src and dst. The event the connection
> logs’ ids (markers).****
>
> (other rules are very similar in structure, but different in logic, of
> course) ****
>
> ** **
>
> This is its programmatic drl file:****
>
> ** **
>
> package com.checkpoint.correlation.impl.drools.package30;****
>
> ** **
>
> import java.util.Date****
>
> import java.util.HashMap****
>
> import java.util.Set****
>
> import com.checkpoint.correlation.impl.drools.Log****
>
> import com.checkpoint.correlation.impl.drools.CorrelatedEvent****
>
> ** **
>
> global com.checkpoint.correlation.server.EventsHandler
> externalEventsHandler;****
>
> ** **
>
> import function
> com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInDayHourRange
> ****
>
> import function
> com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInIpRange****
>
> ** **
>
> function boolean filter(Log log) { ****
>
> return (!((log.fieldsMap.get("src")!= null &&
> isInIpRange(log.fieldsMap.get("src").toString(), "10.80.0.0",
> "10.80.255.255")) || (log.fieldsMap.get("src")!= null &&
> isInIpRange(log.fieldsMap.get("src").toString(), "124.0.0.0",
> "124.255.255.255")) || (log.fieldsMap.get("src")!= null &&
> isInIpRange(log.fieldsMap.get("src").toString(), "192.168.0.0",
> "192.168.255.255")) || (log.fieldsMap.get("src")!= null &&
> isInIpRange(log.fieldsMap.get("src").toString(), "195.158.7.0",
> "195.158.7.255")) || (log.fieldsMap.get("src")!= null &&
> isInIpRange(log.fieldsMap.get("src").toString(), "11.25.0.0",
> "11.25.255.255")) || (log.fieldsMap.get("src")!= null &&
> isInIpRange(log.fieldsMap.get("src").toString(), "128.157.0.0",
> "128.157.255.255")) || (log.fieldsMap.get("src")!= null &&
> isInIpRange(log.fieldsMap.get("src").toString(), "213.114.0.0",
> "213.114.255.255"))));****
>
> }****
>
> ** **
>
> function String markersToString(Set markersSet) { ****
>
> int i = 0;****
>
> String markersString = "";****
>
> for (Object marker : markersSet) { ****
>
> if (i == 25) break;****
>
> String markerStr = marker.toString();****
>
> if (i > 0) markersString += "\n";****
>
> markersString += markerStr;****
>
> }****
>
> return markersString;****
>
> }****
>
> ** **
>
> function String calcSeverity(Log log) { ****
>
> return "High";****
>
> }****
>
> ** **
>
> function String getUniqueId(Log log) { ****
>
> String uniqueId="";****
>
> uniqueId += (log.fieldsMap.get("service") != null ?
> log.fieldsMap.get("service").toString() : "null");****
>
> uniqueId += (log.fieldsMap.get("proto") != null ?
> log.fieldsMap.get("proto").toString() : "null");****
>
> return uniqueId;****
>
> }****
>
>
> ****
>
> declare Log****
>
> @role(event)****
>
> end****
>
> ** **
>
> declare CorrelatedEvent****
>
> @role(event)****
>
> @expires(600s)****
>
> end****
>
> ** **
>
> rule "Port scan from external network"****
>
> enabled true****
>
> dialect "java"****
>
> no-loop****
>
> when****
>
> $log : Log(eval(filter($log)))****
>
> not CorrelatedEvent(getId() ==
> "{8AC52BA8-1EE8-4f18-9BB4-54492116501C}", groupByFieldsMap.get("src") ==
> $log.fieldsMap.get("src"), groupByFieldsMap.get("dst") ==
> $log.fieldsMap.get("dst"))****
>
> accumulate($accumulatedLog :
> Log(eval(filter($accumulatedLog)), this after[0s,60s] $log,
> fieldsMap.get("src") == $log.fieldsMap.get("src"), fieldsMap.get("dst") ==
> $log.fieldsMap.get("dst"), $id : getUniqueId(this));****
>
> $idSet : collectSet($id);****
>
> $idSet.size > 19)****
>
> accumulate($accumulatedLog :
> Log(eval(filter($accumulatedLog)), this after[0s,60s] $log,
> fieldsMap.get("src") == $log.fieldsMap.get("src"), fieldsMap.get("dst") ==
> $log.fieldsMap.get("dst"), $idSet.contains(getUniqueId(this)), $marker :
> fieldsMap.get("marker"));****
>
> $markerSet : collectSet($marker))****
>
> then****
>
> CorrelatedEvent $ce = new
> CorrelatedEvent("{8AC52BA8-1EE8-4f18-9BB4-54492116501C}");****
>
> $ce.groupByFieldsMap.put("src", $log.fieldsMap.get("src"));
> ****
>
> $ce.groupByFieldsMap.put("dst", $log.fieldsMap.get("dst"));
> ****
>
> insert($ce);****
>
> HashMap<String,Object> fieldsMap = new
> HashMap<String,Object>();****
>
> fieldsMap.put("cu_rule_id",
> "{8AC52BA8-1EE8-4f18-9BB4-54492116501C}");****
>
> fieldsMap.put("event_name", "Port scan from external
> network");****
>
> fieldsMap.put("cu_rule_severity", calcSeverity($log));****
>
> fieldsMap.put("cu_rule_category", "Scans");****
>
> fieldsMap.put("cu_log_count", $markerSet.size());****
>
> fieldsMap.put("time", new Date());****
>
> fieldsMap.put("cu_markers_list",
> markersToString($markerSet));****
>
> fieldsMap.put("src", $log.fieldsMap.get("src"));****
>
> fieldsMap.put("src_machine_name",
> $log.fieldsMap.get("src_machine_name"));****
>
> fieldsMap.put("src_user_name",
> $log.fieldsMap.get("src_user_name"));****
>
> fieldsMap.put("dst", $log.fieldsMap.get("dst"));****
>
> fieldsMap.put("dst_machine_name",
> $log.fieldsMap.get("dst_machine_name"));****
>
> fieldsMap.put("dst_user_name",
> $log.fieldsMap.get("dst_user_name"));****
>
> fieldsMap.put("service", $log.fieldsMap.get("service"));**
> **
>
> fieldsMap.put("proto", $log.fieldsMap.get("proto"));****
>
> fieldsMap.put("product", $log.fieldsMap.get("product"));**
> **
>
> externalEventsHandler.handleEvent(fieldsMap);****
>
> end ****
>
> ** **
>
> I am sending logs in a rate of up to 200 logs/sec. After about 3 minutes,
> my application starts to be unresponsive. ****
>
> I monitored the JVM with VisualVM. Two snapshots of VisualVM are attached.
> ****
>
> I found out that the class consuming most memory is FromNodeLeftTuple of
> drools (as can be seen in “instances.png”).****
>
> ** **
>
> **1) **Is my inserting rate is too high?****
>
> **2) **Is There a way I can make my rules more memory efficient?****
>
> ** **
>
> Thanks.****
>
> ** **
>
> Inserting logs:****
>
> public void insertEvents(Collection<Map<String, Object>> logs)****
>
> {****
>
> for (Map<String, Object> map : logs) {****
>
> Log log = new Log();****
>
> Log.fieldsMap.putAll(map);****
>
> session.insert(log);****
>
> session.fireAllRules();****
>
> }****
>
> }****
>
> ** **
>
> Log class:****
>
> public class Log****
>
> {****
>
> public HashMap<String, Object> fieldsMap = new HashMap<>();****
>
> }****
>
> ** **
>
> CorrelatedEvent class:****
>
> ** **
>
> public class CorrelatedEvent****
>
> {****
>
> public Map<String, Object> groupByFieldsMap;****
>
> ** **
>
> private String id;****
>
> ** **
>
> public CorrelatedEvent(String id)****
>
> {****
>
> groupByFieldsMap = new HashMap<>();****
>
> this.id = id;****
>
> }****
>
> ** **
>
> public String getId()****
>
> {****
>
> return id;****
>
> }****
>
> }****
>
> ** **
>
> ** **
>
> _______________________________________________
> rules-users mailing list
> rules-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/rules-users/attachments/20131024/41b3f0ac/attachment-0001.html
More information about the rules-users
mailing list