[rules-users] Drools memory consumption
Elran Dvir
elrand at checkpoint.com
Mon Oct 28 05:31:26 EDT 2013
Thanks for the valuable feedback!
Regarding remarks about the rule itself:
> a) Each Log without a CorrelatedEvent creates an activation. A CorrelatedEvent will not apear until there are 20 Logs with the same src/dst combination. This means that there are up to 19 pending accumulates until the first
> one reaches the threshold, inserts the CorrelatedEvent and cancels the others.
How can I make more efficient?
> b) Additional inefficiency is produced by the separate accumulation of the markers. This set does not participate in LHS constraints, and hence could be easily computed on the RHS.
I tried to remove accumulation of the markers.
These are the changes:
New function:
function Set getMarkers(Set matchedLogs) {
HashSet<String> markerSet = new HashSet<String>();
HashSet<String> idSet = new HashSet<String>();
for (Object matchedLogObj : matchedLogs) {
Log matchedLog = (Log) matchedLogObj;
String id = getUniqueId(matchedLog);
if (!idSet.contains(id)) {
idSet.add(id);
markerSet.add(matchedLog.fieldsMap.get("marker").toString());
if (markerSet.size() == 25) break;
}
}
return markerSet;
}
In LHS:
accumulate($accumulatedLog : Log(eval(filter($accumulatedLog)), this after[0s,60s] $log, fieldsMap.get("src") == $log.fieldsMap.get("src"), fieldsMap.get("dst") == $log.fieldsMap.get("dst"), $id : getUniqueId(this));
$idSet : collectSet($id) , $matchedLogs: collectSet($accumulatedLog);
$idSet.size > 19)
In RHS:
fieldsMap.put("cu_markers_list", getMarkers($matchedLogs));
$matchedLogs return as empty set. Why? How can I make it work?
Thanks.
-----Original Message-----
From: rules-users-bounces at lists.jboss.org [mailto:rules-users-bounces at lists.jboss.org] On Behalf Of Wolfgang Laun
Sent: Sunday, October 27, 2013 7:56 PM
To: Rules Users List
Subject: Re: [rules-users] Drools memory consumption
A few remarks on the code snippets.
* function boolean filter: Is it a good idea to do a lot of operations that could be done on 32-bit quantities (using bit operations) based on String values? Converting the log values to int will pay back nicely.
* function String markersToString: i is never changed. StringBuffer? "\n"?
* String getUniqueId(Log log)
return log.fieldsMap.get("service") + log.fieldsMap.get("service").toString()
would do.
* The rule.
a) Each Log without a CorrelatedEvent creates an activation. A CorrelatedEvent will not apear until there are 20 Logs with the same src/dst combination. This means that there are up to 19 pending accumulates until the first one reaches the threshold, inserts the CorrelatedEvent and cancels the others.
b) Additional inefficiency is produced by the separate accumulation of the markers. This set does not participate in LHS constraints, and hence could be easily computed on the RHS.
This looks like a scenario where Drools is kept overly busy which, in turn, will create some extra GC effort. Analysing the data and counting activations coming and going and firing would shed some more light on this.
-W
On 27/10/2013, Elran Dvir <elrand at checkpoint.com> wrote:
> Thanks for the response.
>
> I will try to implement eval(filter(...)) in DRL.
> I tried to change insert to insertLogical and the rule isn't working
> correctly. The condition not CorrelatedEvent... is always true.
> If I set expiration to CorrelatedEvent isn't Drools supposed to manage
> CorrelatedEvent's memory consumption by itself?
>
> Thanks.
>
> From: rules-users-bounces at lists.jboss.org
> [mailto:rules-users-bounces at lists.jboss.org] On Behalf Of Stephen
> Masters
> Sent: Thursday, October 24, 2013 5:49 PM
> To: Rules Users List
> Subject: Re: [rules-users] Drools memory consumption
>
> I haven't dug through your code in serious detail, but I did notice
> that you're using insert instead of insertLogical.
>
> That could lead to huge numbers of CorrelatedEvent instances in
> working memory, unless you have rules elsewhere to retract them.
>
> >From a performance perspective, I would be very suspicious of your
> eval(filter(...)) condition. It looks as though it could easily be
> implemented in DRL.
>
> And we all know that "eval" is only one vowel away from "evil". ;)
>
>
> On 24 Oct 2013, at 15:03, Elran Dvir
> <elrand at checkpoint.com<mailto:elrand at checkpoint.com>> wrote:
>
>
> Hi All,
>
> Can someone please take a look at my drl file and say if and how I can
> improve its memory consumption?
> I attach my case again.
>
> For example, one rule is supposed to identify a port scan event.
> The basic fact is connection log. For each combination of src (source
> IP) and dst (destination IP) , detect a port scan event, if over 60
> seconds there were at least 20 connection logs with different service and protocol.
> The event will stay closed for 10 minute - no event will be sent
> during this time for this combination of src and dst. The event the connection logs'
> ids (markers).
>
> This is its drl file:
>
> package com.checkpoint.correlation.impl.drools.package30;
>
> import java.util.Date
> import java.util.HashMap
> import java.util.Set
> import com.checkpoint.correlation.impl.drools.Log
> import com.checkpoint.correlation.impl.drools.CorrelatedEvent
>
> global com.checkpoint.correlation.server.EventsHandler
> externalEventsHandler;
>
> import function
> com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInDayHour
> Range
> import function
> com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInIpRange
>
> function boolean filter(Log log) {
> return (!((log.fieldsMap.get("src")!= null &&
> isInIpRange(log.fieldsMap.get("src").toString(), "10.80.0.0",
> "10.80.255.255")) || (log.fieldsMap.get("src")!= null &&
> isInIpRange(log.fieldsMap.get("src").toString(), "124.0.0.0",
> "124.255.255.255")) || (log.fieldsMap.get("src")!= null &&
> isInIpRange(log.fieldsMap.get("src").toString(), "192.168.0.0",
> "192.168.255.255")) || (log.fieldsMap.get("src")!= null &&
> isInIpRange(log.fieldsMap.get("src").toString(), "195.158.7.0",
> "195.158.7.255")) || (log.fieldsMap.get("src")!= null &&
> isInIpRange(log.fieldsMap.get("src").toString(), "11.25.0.0",
> "11.25.255.255")) || (log.fieldsMap.get("src")!= null &&
> isInIpRange(log.fieldsMap.get("src").toString(), "128.157.0.0",
> "128.157.255.255")) || (log.fieldsMap.get("src")!= null &&
> isInIpRange(log.fieldsMap.get("src").toString(), "213.114.0.0",
> "213.114.255.255")))); }
>
> function String markersToString(Set markersSet) {
> int i = 0;
> String markersString = "";
> for (Object marker : markersSet) {
> if (i == 25) break;
> String markerStr = marker.toString();
> if (i > 0) markersString += "\n";
> markersString += markerStr;
> }
> return markersString;
> }
>
> function String calcSeverity(Log log) {
> return "High";
> }
>
> function String getUniqueId(Log log) {
> String uniqueId="";
> uniqueId += (log.fieldsMap.get("service") != null ?
> log.fieldsMap.get("service").toString() : "null");
> uniqueId += (log.fieldsMap.get("proto") != null ?
> log.fieldsMap.get("proto").toString() : "null");
> return uniqueId;
> }
>
> declare Log
> @role(event)
> end
>
> declare CorrelatedEvent
> @role(event)
> @expires(600s)
> end
>
> rule "Port scan from external network"
> enabled true
> dialect "java"
> no-loop
> when
> $log : Log(eval(filter($log)))
> not CorrelatedEvent(getId() ==
> "{8AC52BA8-1EE8-4f18-9BB4-54492116501C}", groupByFieldsMap.get("src")
> == $log.fieldsMap.get("src"), groupByFieldsMap.get("dst") ==
> $log.fieldsMap.get("dst"))
> accumulate($accumulatedLog :
> Log(eval(filter($accumulatedLog)), this after[0s,60s] $log,
> fieldsMap.get("src") == $log.fieldsMap.get("src"),
> fieldsMap.get("dst") == $log.fieldsMap.get("dst"), $id : getUniqueId(this));
> $idSet : collectSet($id);
> $idSet.size > 19)
> accumulate($accumulatedLog :
> Log(eval(filter($accumulatedLog)), this after[0s,60s] $log,
> fieldsMap.get("src") == $log.fieldsMap.get("src"),
> fieldsMap.get("dst") == $log.fieldsMap.get("dst"), $idSet.contains(getUniqueId(this)), $marker :
> fieldsMap.get("marker"));
> $markerSet : collectSet($marker))
> then
> CorrelatedEvent $ce = new
> CorrelatedEvent("{8AC52BA8-1EE8-4f18-9BB4-54492116501C}");
> $ce.groupByFieldsMap.put("src", $log.fieldsMap.get("src"));
> $ce.groupByFieldsMap.put("dst", $log.fieldsMap.get("dst"));
> insert($ce);
> HashMap<String,Object> fieldsMap = new
> HashMap<String,Object>();
> fieldsMap.put("cu_rule_id",
> "{8AC52BA8-1EE8-4f18-9BB4-54492116501C}");
> fieldsMap.put("event_name", "Port scan from external
> network");
> fieldsMap.put("cu_rule_severity", calcSeverity($log));
> fieldsMap.put("cu_rule_category", "Scans");
> fieldsMap.put("cu_log_count", $markerSet.size());
> fieldsMap.put("time", new Date());
> fieldsMap.put("cu_markers_list",
> markersToString($markerSet));
> fieldsMap.put("src", $log.fieldsMap.get("src"));
> fieldsMap.put("src_machine_name",
> $log.fieldsMap.get("src_machine_name"));
> fieldsMap.put("src_user_name",
> $log.fieldsMap.get("src_user_name"));
> fieldsMap.put("dst", $log.fieldsMap.get("dst"));
> fieldsMap.put("dst_machine_name",
> $log.fieldsMap.get("dst_machine_name"));
> fieldsMap.put("dst_user_name",
> $log.fieldsMap.get("dst_user_name"));
> fieldsMap.put("service", $log.fieldsMap.get("service"));
> fieldsMap.put("proto", $log.fieldsMap.get("proto"));
> fieldsMap.put("product", $log.fieldsMap.get("product"));
> externalEventsHandler.handleEvent(fieldsMap);
> end
>
> Thanks.
>
>
> From: Elran Dvir
> Sent: Thursday, October 24, 2013 11:23 AM
> To: Rules Users List
> Subject: RE: [rules-users] Drools memory consumption
>
> I am using 5.5.0.Final.
>
> Thanks.
>
> From:
> rules-users-bounces at lists.jboss.org<mailto:rules-users-bounces at lists.j
> boss.org>
> [mailto:rules-users-bounces at lists.jboss.org<mailto:users-bounces at lists
> .jboss.org>]
> On Behalf Of Geoffrey De Smet
> Sent: Thursday, October 24, 2013 10:46 AM
> To: Rules Users List
> Subject: Re: [rules-users] Drools memory consumption
>
> Which version of drools?
> Memory consumption improved a lot in 5.2 or 5.3 IIRC.
> On 24-10-13 09:14, Elran Dvir wrote:
> Thanks for the quick response.
>
> I don't use entrypoints or windows in my rules.
> I am using temporal constraints.
> I can't retract the Log facts because I don't know whether other rules
> need them.
>
> Maybe It's because there is no expiration set for Log. But I am not
> sure if I can.
> (CorrelatedEvent has multiple expirations defined over the drl files)
> I wrote this question two days ago and no one responded:
>
> I have several drl files.
> Each drl file contains one rule.
> Each drl file is built to a knowledge package.
> Each drl file contains declaration of the same fact (referencing to a
> imported class defined outside of the drl). The fact's role is of
> course event.
>
> Can I define different expiration (@expires) for the fact in each drl?
> Is expiration relevant only to the same drl?
> Is the answer different if the fact is created in the rule's RHS
> (CorrelatedEvent) or if it inserted to the session's working memory
> outside of the drl (Log)?
>
> Thanks.
>
> From:
> rules-users-bounces at lists.jboss.org<mailto:rules-users-bounces at lists.j
> boss.org> [mailto:rules-users-bounces at lists.jboss.org] On Behalf Of
> Matteo Cusmai
> Sent: Thursday, October 24, 2013 9:32 AM
> To: Rules Users List
> Subject: Re: [rules-users] Drools memory consumption
>
>
> Are you using any entrypoint?
> Time or length based?
> It could be possible you have to retract manually the events.
> On 24 Oct 2013 08:23, "Elran Dvir"
> <elrand at checkpoint.com<mailto:elrand at checkpoint.com>> wrote:
> I am sending this message again because maybe the last wasn't sent
> because of the attached snapshots.
> I removed them now.
> Thanks.
>
> Hi all,
>
> I am using Drools Fusion. I am getting OutOfMemoryError rather fast.
> My JVM is running with -Xmx4g flag.
> I have rules defined in another (not Drools) language.
> Every rule is translated programmatically to a drl file. This is
> because the user can add and remove rules (in the other language) dynamically.
> The default configuration contains 125 rules.
>
> For example, one rule is supposed to identify a port scan event.
>
> The basic fact is connection log. For each combination of src (source
> IP) and dst (destination IP) , detect a port scan event, if over 60
> seconds there were at least 20 connection logs with different service and protocol.
>
> The event will stay closed for 10 minute - no event will be sent
> during this time for this combination of src and dst. The event the connection logs'
> ids (markers).
> (other rules are very similar in structure, but different in logic, of
> course)
>
> This is its programmatic drl file:
>
> package com.checkpoint.correlation.impl.drools.package30;
>
> import java.util.Date
> import java.util.HashMap
> import java.util.Set
> import com.checkpoint.correlation.impl.drools.Log
> import com.checkpoint.correlation.impl.drools.CorrelatedEvent
>
> global com.checkpoint.correlation.server.EventsHandler
> externalEventsHandler;
>
> import function
> com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInDayHour
> Range
> import function
> com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInIpRange
>
> function boolean filter(Log log) {
> return (!((log.fieldsMap.get("src")!= null &&
> isInIpRange(log.fieldsMap.get("src").toString(), "10.80.0.0",
> "10.80.255.255")) || (log.fieldsMap.get("src")!= null &&
> isInIpRange(log.fieldsMap.get("src").toString(), "124.0.0.0",
> "124.255.255.255")) || (log.fieldsMap.get("src")!= null &&
> isInIpRange(log.fieldsMap.get("src").toString(), "192.168.0.0",
> "192.168.255.255")) || (log.fieldsMap.get("src")!= null &&
> isInIpRange(log.fieldsMap.get("src").toString(), "195.158.7.0",
> "195.158.7.255")) || (log.fieldsMap.get("src")!= null &&
> isInIpRange(log.fieldsMap.get("src").toString(), "11.25.0.0",
> "11.25.255.255")) || (log.fieldsMap.get("src")!= null &&
> isInIpRange(log.fieldsMap.get("src").toString(), "128.157.0.0",
> "128.157.255.255")) || (log.fieldsMap.get("src")!= null &&
> isInIpRange(log.fieldsMap.get("src").toString(), "213.114.0.0",
> "213.114.255.255")))); }
>
> function String markersToString(Set markersSet) {
> int i = 0;
> String markersString = "";
> for (Object marker : markersSet) {
> if (i == 25) break;
> String markerStr = marker.toString();
> if (i > 0) markersString += "\n";
> markersString += markerStr;
> }
> return markersString;
> }
>
> function String calcSeverity(Log log) {
> return "High";
> }
>
> function String getUniqueId(Log log) {
> String uniqueId="";
> uniqueId += (log.fieldsMap.get("service") != null ?
> log.fieldsMap.get("service").toString() : "null");
> uniqueId += (log.fieldsMap.get("proto") != null ?
> log.fieldsMap.get("proto").toString() : "null");
> return uniqueId;
> }
>
>
> &n! bsp;
> declare Log
> @role(event)
> end
>
> declare CorrelatedEvent
> @role(event)
> @expires(600s)
> end
>
> rule "Port scan from external network"
> enabled true
> dialect "java"
> no-loop
> when
> $log : Log(eval(filter($log)))
> not CorrelatedEvent(getId() ==
> "{8AC52BA8-1EE8-4f18-9BB4-54492116501C}", groupByFieldsMap.get("src")
> == $log.fieldsMap.get("src"), groupByFieldsMap.get("dst") ==
> $log.fieldsMap.get("dst"))
> accumulate($accumulatedLog :
> Log(eval(filter($accumulatedLog)), this after[0s,60s] $log,
> fieldsMap.get("src") == $log.fieldsMap.get("src"),
> fieldsMap.get("dst") == $log.fieldsMap.get("dst"), $id : getUniqueId(this));
> $idSet : collectSet($id);
> $idSet.size > 19)
> accumulate($accumulatedLog :
> Log(eval(filter($accumulatedLog)), this after[0s,60s] $log,
> fieldsMap.get("src") == $log.fieldsMap.get("src"),
> fieldsMap.get("dst") == $log.fieldsMap.get("dst"), $idSet.contains(getUniqueId(this)), $marker :
> fieldsMap.get("marker"));
> $markerSet : collectSet($marker))
> then
> CorrelatedEvent $ce = new
> CorrelatedEvent("{8AC52BA8-1EE8-4f18-9BB4-54492116501C}");
> $ce.groupByFieldsMap.put("src", $log.fieldsMap.get("src"));
> $ce.groupByFieldsMap.put("dst", $log.fieldsMap.get("dst"));
> insert($ce);
> HashMap<String,Object> fieldsMap = new
> HashMap<String,Object>();
> fieldsMap.put("cu_rule_id",
> "{8AC52BA8-1EE8-4f18-9BB4-54492116501C}");
> fieldsMap.put("event_name", "Port scan from external
> network");
> fieldsMap.put("cu_rule_severity", calcSeverity($log));
> fieldsMap.put("cu_rule_category", "Scans");
> fieldsMap.put("cu_log_count", $markerSet.size());
> fieldsMap.put("time", new Date());
> fieldsMap.put("cu_markers_list",
> markersToString($markerSet));
> fieldsMap.put("src", $log.fieldsMap.get("src"));
> fieldsMap.put("src_machine_name",
> $log.fieldsMap.get("src_machine_name"));
> fieldsMap.put("src_user_name",
> $log.fieldsMap.get("src_user_name"));
> fieldsMap.put("dst", $log.fieldsMap.get("dst"));
> fieldsMap.put("dst_machine_name",
> $log.fieldsMap.get("dst_machine_name"));
> fieldsMap.put("dst_user_name",
> $log.fieldsMap.get("dst_user_name"));
> fieldsMap.put("service", $log.fieldsMap.get("service"));
> fieldsMap.put("proto", $log.fieldsMap.get("proto"));
> fieldsMap.put("product", $log.fieldsMap.get("product"));
> externalEventsHandler.handleEvent(fieldsMap);
> end
>
> I am sending logs in a rate of up to 200 logs/sec. After about 3
> minutes, my application starts to be unresponsive.
> I monitored the JVM with VisualVM. Two snapshots of VisualVM are attached.
> I found out that the class consuming most memory is FromNodeLeftTuple
> of drools (as can be seen in "instances.png").
>
>
> 1) Is my inserting rate is too high?
>
> 2) Is There a way I can make my rules more memory efficient?
>
> Thanks.
>
> Inserting logs:
> public void insertEvents(Collection<Map<String, Object>> logs) {
> for (Map<String, Object> map : logs) {
> Log log = new Log();
> Log.fieldsMap.putAll(map);
> session.insert(log);
> session.fireAllRules();
> }
> }
>
> Log class:
> public class Log
> {
> public HashMap<String, Object> fieldsMap = new HashMap<>(); }
>
> CorrelatedEvent class:
>
> public class CorrelatedEvent
> {
> public Map<String, Object> groupByFieldsMap;
>
> private String id;
>
> public CorrelatedEvent(String id)
> {
> groupByFieldsMap = new HashMap<>();
> this.id<http://this.id> = id;
> }
>
> public String getId()
> {
> return id;
> }
> }
>
>
>
> _______________________________________________
> rules-users mailing list
> rules-users at lists.jboss.org<mailto:rules-users at lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/rules-users
>
>
> Email secured by Check Point
>
>
>
>
> _______________________________________________
>
> rules-users mailing list
>
> rules-users at lists.jboss.org<mailto:rules-users at lists.jboss.org>
>
> https://lists.jboss.org/mailman/listinfo/rules-users
>
> _______________________________________________
> rules-users mailing list
> rules-users at lists.jboss.org<mailto:rules-users at lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/rules-users
>
>
>
> Email secured by Check Point
>
_______________________________________________
rules-users mailing list
rules-users at lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
Email secured by Check Point
More information about the rules-users
mailing list