1:22 a.m.
I am sending this message again because maybe the last wasn't sent because of the
attached snapshots.
I removed them now.
Thanks.
Hi all,
I am using Drools Fusion. I am getting OutOfMemoryError rather fast. My JVM is running
with -Xmx4g flag.
I have rules defined in another (not Drools) language.
Every rule is translated programmatically to a drl file. This is because the user can add
and remove rules (in the other language) dynamically.
The default configuration contains 125 rules.
For example, one rule is supposed to identify a port scan event.
The basic fact is connection log. For each combination of src (source IP) and dst
(destination IP) , detect a port scan event, if over 60 seconds there were at least 20
connection logs with different service and protocol.
The event will stay closed for 10 minute - no event will be sent during this time for this
combination of src and dst. The event the connection logs' ids (markers).
(other rules are very similar in structure, but different in logic, of course)
This is its programmatic drl file:
package com.checkpoint.correlation.impl.drools.package30;
import java.util.Date
import java.util.HashMap
import java.util.Set
import com.checkpoint.correlation.impl.drools.Log
import com.checkpoint.correlation.impl.drools.CorrelatedEvent
global com.checkpoint.correlation.server.EventsHandler externalEventsHandler;
import function
com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInDayHourRange
import function com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInIpRange
function boolean filter(Log log) {
return (!((log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(), "10.80.0.0",
"10.80.255.255")) || (log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(), "124.0.0.0",
"124.255.255.255")) || (log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(), "192.168.0.0",
"192.168.255.255")) || (log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(), "195.158.7.0",
"195.158.7.255")) || (log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(), "11.25.0.0",
"11.25.255.255")) || (log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(), "128.157.0.0",
"128.157.255.255")) || (log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(), "213.114.0.0",
"213.114.255.255"))));
}
function String markersToString(Set markersSet) {
int i = 0;
String markersString = "";
for (Object marker : markersSet) {
if (i == 25) break;
String markerStr = marker.toString();
if (i > 0) markersString += "\n";
markersString += markerStr;
}
return markersString;
}
function String calcSeverity(Log log) {
return "High";
}
function String getUniqueId(Log log) {
String uniqueId="";
uniqueId += (log.fieldsMap.get("service") != null ?
log.fieldsMap.get("service").toString() : "null");
uniqueId += (log.fieldsMap.get("proto") != null ?
log.fieldsMap.get("proto").toString() : "null");
return uniqueId;
}
declare Log
@role(event)
end
declare CorrelatedEvent
@role(event)
@expires(600s)
end
rule "Port scan from external network"
enabled true
dialect "java"
no-loop
when
$log : Log(eval(filter($log)))
not CorrelatedEvent(getId() ==
"{8AC52BA8-1EE8-4f18-9BB4-54492116501C}", groupByFieldsMap.get("src")
== $log.fieldsMap.get("src"), groupByFieldsMap.get("dst") ==
$log.fieldsMap.get("dst"))
accumulate($accumulatedLog : Log(eval(filter($accumulatedLog)), this
after[0s,60s] $log, fieldsMap.get("src") == $log.fieldsMap.get("src"),
fieldsMap.get("dst") == $log.fieldsMap.get("dst"), $id :
getUniqueId(this));
$idSet : collectSet($id);
$idSet.size > 19)
accumulate($accumulatedLog : Log(eval(filter($accumulatedLog)), this
after[0s,60s] $log, fieldsMap.get("src") == $log.fieldsMap.get("src"),
fieldsMap.get("dst") == $log.fieldsMap.get("dst"),
$idSet.contains(getUniqueId(this)), $marker : fieldsMap.get("marker"));
$markerSet : collectSet($marker))
then
CorrelatedEvent $ce = new
CorrelatedEvent("{8AC52BA8-1EE8-4f18-9BB4-54492116501C}");
$ce.groupByFieldsMap.put("src",
$log.fieldsMap.get("src"));
$ce.groupByFieldsMap.put("dst",
$log.fieldsMap.get("dst"));
insert($ce);
HashMap<String,Object> fieldsMap = new
HashMap<String,Object>();
fieldsMap.put("cu_rule_id",
"{8AC52BA8-1EE8-4f18-9BB4-54492116501C}");
fieldsMap.put("event_name", "Port scan from external
network");
fieldsMap.put("cu_rule_severity", calcSeverity($log));
fieldsMap.put("cu_rule_category", "Scans");
fieldsMap.put("cu_log_count", $markerSet.size());
fieldsMap.put("time", new Date());
fieldsMap.put("cu_markers_list", markersToString($markerSet));
fieldsMap.put("src", $log.fieldsMap.get("src"));
fieldsMap.put("src_machine_name",
$log.fieldsMap.get("src_machine_name"));
fieldsMap.put("src_user_name",
$log.fieldsMap.get("src_user_name"));
fieldsMap.put("dst", $log.fieldsMap.get("dst"));
fieldsMap.put("dst_machine_name",
$log.fieldsMap.get("dst_machine_name"));
fieldsMap.put("dst_user_name",
$log.fieldsMap.get("dst_user_name"));
fieldsMap.put("service",
$log.fieldsMap.get("service"));
fieldsMap.put("proto", $log.fieldsMap.get("proto"));
fieldsMap.put("product",
$log.fieldsMap.get("product"));
externalEventsHandler.handleEvent(fieldsMap);
end
I am sending logs in a rate of up to 200 logs/sec. After about 3 minutes, my application
starts to be unresponsive.
I monitored the JVM with VisualVM. Two snapshots of VisualVM are attached.
I found out that the class consuming most memory is FromNodeLeftTuple of drools (as can be
seen in "instances.png").
1) Is my inserting rate is too high?
2) Is There a way I can make my rules more memory efficient?
Thanks.
Inserting logs:
public void insertEvents(Collection<Map<String, Object>> logs)
{
for (Map<String, Object> map : logs) {
Log log = new Log();
Log.fieldsMap.putAll(map);
session.insert(log);
session.fireAllRules();
}
}
Log class:
public class Log
{
public HashMap<String, Object> fieldsMap = new HashMap<>();
}
CorrelatedEvent class:
public class CorrelatedEvent
{
public Map<String, Object> groupByFieldsMap;
private String id;
public CorrelatedEvent(String id)
{
groupByFieldsMap = new HashMap<>();
this.id = id;
}
public String getId()
{
return id;
}
}