I am sending this message again
because maybe the last wasn’t sent because of the
attached snapshots.
I removed them now.
Thanks.
Hi
all,
I
am using Drools Fusion. I am getting OutOfMemoryError
rather fast. My JVM is running with –Xmx4g flag.
I
have rules defined in another (not Drools) language.
Every
rule is translated programmatically to a drl file. This
is because the user can add and remove rules (in the
other language) dynamically.
The
default configuration contains 125 rules.
For
example, one rule is supposed to identify a port scan
event.
The basic fact is connection log. For each combination
of src (source IP) and dst (destination IP) , detect a
port scan event, if over 60 seconds there were at least
20 connection logs with different service and protocol.
The event will stay closed for 10 minute - no event
will be sent during this time for this combination of
src and dst. The event the connection logs’ ids
(markers).
(other
rules are very similar in structure, but different in
logic, of course)
This
is its programmatic drl file:
package
com.checkpoint.correlation.impl.drools.package30;
import
java.util.Date
import
java.util.HashMap
import
java.util.Set
import
com.checkpoint.correlation.impl.drools.Log
import
com.checkpoint.correlation.impl.drools.CorrelatedEvent
global
com.checkpoint.correlation.server.EventsHandler
externalEventsHandler;
import
function
com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInDayHourRange
import
function
com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInIpRange
function
boolean filter(Log log) {
return (!((log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(),
"10.80.0.0", "10.80.255.255")) ||
(log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(),
"124.0.0.0", "124.255.255.255")) ||
(log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(),
"192.168.0.0", "192.168.255.255")) ||
(log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(),
"195.158.7.0", "195.158.7.255")) ||
(log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(),
"11.25.0.0", "11.25.255.255")) ||
(log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(),
"128.157.0.0", "128.157.255.255")) ||
(log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(),
"213.114.0.0", "213.114.255.255"))));
}
function
String markersToString(Set markersSet) {
int i = 0;
String markersString = "";
for (Object marker : markersSet) {
if (i == 25) break;
String markerStr = marker.toString();
if (i > 0) markersString += "\n";
markersString += markerStr;
}
return markersString;
}
function
String calcSeverity(Log log) {
return "High";
}
function
String getUniqueId(Log log) {
String uniqueId="";
uniqueId += (log.fieldsMap.get("service") != null ?
log.fieldsMap.get("service").toString() : "null");
uniqueId += (log.fieldsMap.get("proto") != null ?
log.fieldsMap.get("proto").toString() : "null");
return uniqueId;
}
declare
Log
@role(event)
end
declare
CorrelatedEvent
@role(event)
@expires(600s)
end
rule
"Port scan from external network"
enabled
true
dialect
"java"
no-loop
when
$log : Log(eval(filter($log)))
not CorrelatedEvent(getId() ==
"{8AC52BA8-1EE8-4f18-9BB4-54492116501C}",
groupByFieldsMap.get("src") ==
$log.fieldsMap.get("src"), groupByFieldsMap.get("dst")
== $log.fieldsMap.get("dst"))
accumulate($accumulatedLog :
Log(eval(filter($accumulatedLog)), this after[0s,60s]
$log, fieldsMap.get("src") == $log.fieldsMap.get("src"),
fieldsMap.get("dst") == $log.fieldsMap.get("dst"), $id :
getUniqueId(this));
$idSet : collectSet($id);
$idSet.size > 19)
accumulate($accumulatedLog :
Log(eval(filter($accumulatedLog)), this after[0s,60s]
$log, fieldsMap.get("src") == $log.fieldsMap.get("src"),
fieldsMap.get("dst") == $log.fieldsMap.get("dst"),
$idSet.contains(getUniqueId(this)), $marker :
fieldsMap.get("marker"));
$markerSet : collectSet($marker))
then
CorrelatedEvent $ce = new
CorrelatedEvent("{8AC52BA8-1EE8-4f18-9BB4-54492116501C}");
$ce.groupByFieldsMap.put("src",
$log.fieldsMap.get("src"));
$ce.groupByFieldsMap.put("dst",
$log.fieldsMap.get("dst"));
insert($ce);
HashMap<String,Object> fieldsMap = new
HashMap<String,Object>();
fieldsMap.put("cu_rule_id",
"{8AC52BA8-1EE8-4f18-9BB4-54492116501C}");
fieldsMap.put("event_name", "Port scan from external
network");
fieldsMap.put("cu_rule_severity", calcSeverity($log));
fieldsMap.put("cu_rule_category", "Scans");
fieldsMap.put("cu_log_count", $markerSet.size());
fieldsMap.put("time", new Date());
fieldsMap.put("cu_markers_list",
markersToString($markerSet));
fieldsMap.put("src", $log.fieldsMap.get("src"));
fieldsMap.put("src_machine_name",
$log.fieldsMap.get("src_machine_name"));
fieldsMap.put("src_user_name",
$log.fieldsMap.get("src_user_name"));
fieldsMap.put("dst", $log.fieldsMap.get("dst"));
fieldsMap.put("dst_machine_name",
$log.fieldsMap.get("dst_machine_name"));
fieldsMap.put("dst_user_name",
$log.fieldsMap.get("dst_user_name"));
fieldsMap.put("service", $log.fieldsMap.get("service"));
fieldsMap.put("proto", $log.fieldsMap.get("proto"));
fieldsMap.put("product", $log.fieldsMap.get("product"));
externalEventsHandler.handleEvent(fieldsMap);
I
am sending logs in a rate of up to 200 logs/sec. After
about 3 minutes, my application starts to be
unresponsive.
I
monitored the JVM with VisualVM. Two snapshots of
VisualVM are attached.
I
found out that the class consuming most memory is
FromNodeLeftTuple of drools (as can be seen in
“instances.png”).
1) Is my
inserting rate is too high?
2) Is There
a way I can make my rules more memory efficient?
Thanks.
public
void insertEvents(Collection<Map<String,
Object>> logs)
{
for (Map<String, Object> map : logs) {
Log log = new Log();
Log.fieldsMap.putAll(map);
session.insert(log);
session.fireAllRules();
}
public
class Log
{
public HashMap<String, Object> fieldsMap = new
HashMap<>();
}
public
class CorrelatedEvent
{
public Map<String, Object> groupByFieldsMap;
private String id;
public CorrelatedEvent(String id)
{
groupByFieldsMap = new HashMap<>();
this.id = id;
}
public String getId()
{
return id;
}