[rules-users] Drools memory consumption
Elran Dvir
elrand at checkpoint.com
Thu Oct 31 10:44:49 EDT 2013
Wolfgang,
Thanks again for your responses.
I have changed the rule as you recommended me. The main change is using "over window:time" rather than "this after".
The heap size still gets to 4 GB.
The new drl looks like that (for the port scan event):
--------------------------------------------------------------------
package com.checkpoint.correlation.impl.drools.package30;
import java.util.Date
import java.util.HashMap
import java.util.HashSet
import java.util.Collection
import java.util.ArrayList
import com.checkpoint.correlation.impl.drools.Log
import com.checkpoint.correlation.impl.drools.CorrelatedEvent
global com.checkpoint.correlation.server.EventsHandler externalEventsHandler;
global com.checkpoint.correlation.impl.drools.future.Debugger debugger;
import function com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInDayHourRange
import function com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInIpRange
function boolean filter(Log log) {
return (!((log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "10.80.0.0", "10.80.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "124.0.0.0", "124.255.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "192.168.0.0", "192.168.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "195.158.7.0", "195.158.7.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "11.25.0.0", "11.25.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "128.157.0.0", "128.157.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "213.114.0.0", "213.114.255.255"))));
}
function Collection getMarkers(Collection matchedLogs) {
ArrayList<String> markers = new ArrayList<String>();
HashSet<String> idSet = new HashSet<String>();
for (Object matchedLogObj : matchedLogs) {
Log matchedLog = (Log) matchedLogObj;
String id = getUniqueId(matchedLog);
if (!idSet.contains(id)) {
idSet.add(id);
markers.add(matchedLog.fieldsMap.get("marker").toString());
if (markers.size() == 25) break;
}
}
return markers;
}
function String calcSeverity(Log log) {
return "High";
}
function String getUniqueId(Log log) {
String uniqueId="";
uniqueId += (log.fieldsMap.get("service") != null ? log.fieldsMap.get("service").toString() : "null");
uniqueId += (log.fieldsMap.get("proto") != null ? log.fieldsMap.get("proto").toString() : "null");
return uniqueId;
}
declare Log
@role(event)
end
declare CorrelatedEvent
@role(event)
@expires(600s)
end
rule "Port scan from external network"
enabled true
dialect "java"
no-loop
when
$log : Log(eval(filter($log)))
not CorrelatedEvent(getId() == "{8AC52BA8-1EE8-4f18-9BB4-54492116501C}", groupByFieldsMap.get("src") == $log.fieldsMap.get("src"), groupByFieldsMap.get("dst") == $log.fieldsMap.get("dst"))
accumulate($accumulatedLog : Log(eval(filter($accumulatedLog)), fieldsMap.get("src") == $log.fieldsMap.get("src"), fieldsMap.get("dst") == $log.fieldsMap.get("dst"), $id : getUniqueId(this)) over window:time(60s);
$matchedLogs: collectList($accumulatedLog), $idSet : collectSet($id);
$idSet.size > 19)
then
Collection markers = getMarkers($matchedLogs);
CorrelatedEvent $ce = new CorrelatedEvent("{8AC52BA8-1EE8-4f18-9BB4-54492116501C}");
$ce.groupByFieldsMap.put("src", $log.fieldsMap.get("src"));
$ce.groupByFieldsMap.put("dst", $log.fieldsMap.get("dst"));
insert($ce);
HashMap<String,Object> fieldsMap = new HashMap<String,Object>();
fieldsMap.put("cu_rule_id", "{8AC52BA8-1EE8-4f18-9BB4-54492116501C}");
fieldsMap.put("event_name", "Port scan from external network");
fieldsMap.put("cu_rule_severity", calcSeverity($log));
fieldsMap.put("cu_rule_category", "Scans");
fieldsMap.put("cu_log_count", markers.size());
fieldsMap.put("time", new Date());
fieldsMap.put("cu_markers_list", markers);
fieldsMap.put("src", $log.fieldsMap.get("src"));
fieldsMap.put("src_machine_name", $log.fieldsMap.get("src_machine_name"));
fieldsMap.put("src_user_name", $log.fieldsMap.get("src_user_name"));
fieldsMap.put("dst", $log.fieldsMap.get("dst"));
fieldsMap.put("dst_machine_name", $log.fieldsMap.get("dst_machine_name"));
fieldsMap.put("dst_user_name", $log.fieldsMap.get("dst_user_name"));
fieldsMap.put("service", $log.fieldsMap.get("service"));
fieldsMap.put("proto", $log.fieldsMap.get("proto"));
fieldsMap.put("product", $log.fieldsMap.get("product"));
externalEventsHandler.handleEvent(fieldsMap);
end
---------------------------------------------------------------------------------
I have tested it by sending logs in a rate of about 200 logs/sec with my 125 rules.
The class consuming most memory is still "FromNodeLeftTuple".
I have logged Drools activity. The compressed log file size is 750KB.
How can I send it to you?
Thank you very much.
-----Original Message-----
From: rules-users-bounces at lists.jboss.org [mailto:rules-users-bounces at lists.jboss.org] On Behalf Of Wolfgang Laun
Sent: Monday, October 28, 2013 7:52 PM
To: Rules Users List
Subject: Re: [rules-users] Drools memory consumption
On 28/10/2013, Elran Dvir <elrand at checkpoint.com> wrote:
> Thanks for the valuable feedback!
>
> Regarding remarks about the rule itself:
>> a) Each Log without a CorrelatedEvent creates an activation. A
>> CorrelatedEvent will not apear until there are 20 Logs with the same
>> src/dst combination. This means that there are up to 19 pending
>> accumulates until the first one reaches the threshold, inserts the
>> CorrelatedEvent and cancels the others.
> How can I make more efficient?
I think that window:time would be more appropriate. Or implement an equivalent functionality using an auxiliary container.
>
>> b) Additional inefficiency is produced by the separate accumulation
>> of the markers. This set does not participate in LHS constraints, and
>> hence could be easily computed on the RHS.
> I tried to remove accumulation of the markers.
> These are the changes:
>
> New function:
>
> function Set getMarkers(Set matchedLogs) {
> HashSet<String> markerSet = new HashSet<String>();
> HashSet<String> idSet = new HashSet<String>();
> for (Object matchedLogObj : matchedLogs) {
> Log matchedLog = (Log) matchedLogObj;
> String id = getUniqueId(matchedLog);
> if (!idSet.contains(id)) {
> idSet.add(id);
> markerSet.add(matchedLog.fieldsMap.get("marker").toString());
> if (markerSet.size() == 25) break;
> }
> }
> return markerSet;
> }
>
> In LHS:
> accumulate($accumulatedLog : Log(eval(filter($accumulatedLog)), this
> after[0s,60s] $log, fieldsMap.get("src") == $log.fieldsMap.get("src"),
> fieldsMap.get("dst") == $log.fieldsMap.get("dst"), $id :
> getUniqueId(this));
> $idSet : collectSet($id) , $matchedLogs:
> collectSet($accumulatedLog);
> $idSet.size > 19) In RHS:
>
> fieldsMap.put("cu_markers_list", getMarkers($matchedLogs));
>
> $matchedLogs return as empty set. Why? How can I make it work?
There's no reason why it should be empty if $idSet.size > 0.
-W
_______________________________________________
rules-users mailing list
rules-users at lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
Email secured by Check Point
More information about the rules-users
mailing list