Re: [rules-users] Drools memory consumption

Thursday, 24 October 2013

I haven't dug through your code in serious detail, but I did notice that you're
using insert instead of insertLogical.

That could lead to huge numbers of CorrelatedEvent instances in working memory, unless you
have rules elsewhere to retract them.

From a performance perspective, I would be very suspicious of your eval(filter(…))
condition. It looks as though it could easily be implemented in DRL.

And we all know that "eval" is only one vowel away from "evil". ;)

On 24 Oct 2013, at 15:03, Elran Dvir <elrand(a)checkpoint.com&gt; wrote:

...
 Hi All,

 Can someone please take a look at my drl file and say if and how I can improve its memory
consumption?
 I attach my case again.

 For example, one rule is supposed to identify a port scan event.
 The basic fact is connection log. For each combination of src (source IP) and dst
(destination IP) , detect a port scan event, if over 60 seconds there were at least 20
connection logs with different service and protocol.
 The event will stay closed for 10 minute - no event will be sent during this time for
this combination of  src and dst. The event the connection logs’ ids (markers).

 This is its drl file:

 package com.checkpoint.correlation.impl.drools.package30;

 import java.util.Date
 import java.util.HashMap
 import java.util.Set
 import com.checkpoint.correlation.impl.drools.Log
 import com.checkpoint.correlation.impl.drools.CorrelatedEvent

 global com.checkpoint.correlation.server.EventsHandler externalEventsHandler;

 import function
com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInDayHourRange
 import function com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInIpRange

 function boolean filter(Log log) {
                 return  (!((log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(), "10.80.0.0",
"10.80.255.255")) || (log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(), "124.0.0.0",
"124.255.255.255")) || (log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(), "192.168.0.0",
"192.168.255.255")) || (log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(), "195.158.7.0",
"195.158.7.255")) || (log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(), "11.25.0.0",
"11.25.255.255")) || (log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(), "128.157.0.0",
"128.157.255.255")) || (log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(), "213.114.0.0",
"213.114.255.255"))));
 }

 function String markersToString(Set markersSet) {
                 int i = 0;
                 String markersString = "";
                 for (Object marker : markersSet) {
                                 if (i == 25) break;
                                 String markerStr = marker.toString();
                                 if (i > 0) markersString += "\n";
                                 markersString += markerStr;
                 }
                 return markersString;
 }

 function String calcSeverity(Log log) {
                 return "High";
 }

 function String getUniqueId(Log log) {
                 String uniqueId="";
                 uniqueId += (log.fieldsMap.get("service") != null ?
log.fieldsMap.get("service").toString() : "null");
                 uniqueId += (log.fieldsMap.get("proto") != null ?
log.fieldsMap.get("proto").toString() : "null");
                 return uniqueId;
 }

 declare Log
                 @role(event)
 end

 declare CorrelatedEvent
                 @role(event)
                 @expires(600s)
 end

 rule "Port scan from external network"
 enabled true
 dialect "java"
 no-loop
 when
                 $log : Log(eval(filter($log)))
                 not CorrelatedEvent(getId() ==
"{8AC52BA8-1EE8-4f18-9BB4-54492116501C}", groupByFieldsMap.get("src")
== $log.fieldsMap.get("src"), groupByFieldsMap.get("dst") ==
$log.fieldsMap.get("dst"))
                 accumulate($accumulatedLog : Log(eval(filter($accumulatedLog)), this
after[0s,60s] $log, fieldsMap.get("src") == $log.fieldsMap.get("src"),
fieldsMap.get("dst") == $log.fieldsMap.get("dst"), $id :
getUniqueId(this));
                                 $idSet : collectSet($id);
                                 $idSet.size > 19)
                 accumulate($accumulatedLog : Log(eval(filter($accumulatedLog)), this
after[0s,60s] $log, fieldsMap.get("src") == $log.fieldsMap.get("src"),
fieldsMap.get("dst") == $log.fieldsMap.get("dst"),
$idSet.contains(getUniqueId(this)), $marker : fieldsMap.get("marker"));
                                 $markerSet : collectSet($marker))
                 then
                 CorrelatedEvent $ce = new
CorrelatedEvent("{8AC52BA8-1EE8-4f18-9BB4-54492116501C}");
                 $ce.groupByFieldsMap.put("src",
$log.fieldsMap.get("src"));
                 $ce.groupByFieldsMap.put("dst",
$log.fieldsMap.get("dst"));
                 insert($ce);
                 HashMap<String,Object> fieldsMap = new
HashMap<String,Object>();
                 fieldsMap.put("cu_rule_id",
"{8AC52BA8-1EE8-4f18-9BB4-54492116501C}");
                 fieldsMap.put("event_name", "Port scan from external
network");
                 fieldsMap.put("cu_rule_severity", calcSeverity($log));
                 fieldsMap.put("cu_rule_category", "Scans");
                 fieldsMap.put("cu_log_count", $markerSet.size());
                 fieldsMap.put("time", new Date());
                 fieldsMap.put("cu_markers_list", markersToString($markerSet));
                 fieldsMap.put("src", $log.fieldsMap.get("src"));
                 fieldsMap.put("src_machine_name",
$log.fieldsMap.get("src_machine_name"));
                 fieldsMap.put("src_user_name",
$log.fieldsMap.get("src_user_name"));
                 fieldsMap.put("dst", $log.fieldsMap.get("dst"));
                 fieldsMap.put("dst_machine_name",
$log.fieldsMap.get("dst_machine_name"));
                 fieldsMap.put("dst_user_name",
$log.fieldsMap.get("dst_user_name"));
                 fieldsMap.put("service",
$log.fieldsMap.get("service"));
                 fieldsMap.put("proto", $log.fieldsMap.get("proto"));
                 fieldsMap.put("product",
$log.fieldsMap.get("product"));
                 externalEventsHandler.handleEvent(fieldsMap);
 end  

 Thanks.

 From: Elran Dvir 
 Sent: Thursday, October 24, 2013 11:23 AM
 To: Rules Users List
 Subject: RE: [rules-users] Drools memory consumption

 I am using 5.5.0.Final.

 Thanks.

 From: rules-users-bounces(a)lists.jboss.org [mailto:rules-users-bounces@lists.jboss.org] On
Behalf Of Geoffrey De Smet
 Sent: Thursday, October 24, 2013 10:46 AM
 To: Rules Users List
 Subject: Re: [rules-users] Drools memory consumption

 Which version of drools?
 Memory consumption improved a lot in 5.2 or 5.3 IIRC.

 On 24-10-13 09:14, Elran Dvir wrote:
 Thanks for the quick response.

 I don’t use entrypoints or windows in my rules.
 I am using temporal constraints.
 I can’t retract the Log facts because I don’t know whether other rules need them.

 Maybe It’s because there is no expiration set for Log. But I am not sure if I can.
 (CorrelatedEvent has multiple expirations defined over the drl files)
 I wrote this question two days ago and no one responded:

 I have several drl files.
 Each drl file contains one rule.
 Each drl file is built to a knowledge package.
 Each drl file contains declaration of the same fact (referencing to a imported class
defined outside of the drl). The fact’s role is of course event.

 Can I define different expiration (@expires) for the fact in each drl?
 Is expiration relevant only to the same drl?
 Is the answer different if the fact is created in the rule’s RHS (CorrelatedEvent) or if
it inserted to the session’s working memory outside of the drl (Log)?

 Thanks.

 From: rules-users-bounces(a)lists.jboss.org [mailto:rules-users-bounces@lists.jboss.org] On
Behalf Of Matteo Cusmai
 Sent: Thursday, October 24, 2013 9:32 AM
 To: Rules Users List
 Subject: Re: [rules-users] Drools memory consumption

 Are you using any entrypoint?
 Time or length based?
 It could be possible you have to retract manually the events.

 On 24 Oct 2013 08:23, "Elran Dvir" <elrand(a)checkpoint.com&gt; wrote:
 I am sending this message again because maybe the last wasn’t sent because of the
attached snapshots.
 I removed them now.
 Thanks.

 Hi all,

 I am using Drools Fusion. I am getting OutOfMemoryError rather fast. My JVM is running
with –Xmx4g flag.
 I have rules defined in another (not Drools) language.
 Every rule is translated programmatically to a drl file. This is because the user can add
and remove rules (in the other language) dynamically.
 The default configuration contains 125 rules.

 For example, one rule is supposed to identify a port scan event.
 The basic fact is connection log. For each combination of src (source IP) and dst
(destination IP) , detect a port scan event, if over 60 seconds there were at least 20
connection logs with different service and protocol.

 The event will stay closed for 10 minute - no event will be sent during this time for
this combination of  src and dst. The event the connection logs’ ids (markers).

 (other rules are very similar in structure, but different in logic, of course)

 This is its programmatic drl file:

 package com.checkpoint.correlation.impl.drools.package30;

 import java.util.Date
 import java.util.HashMap
 import java.util.Set
 import com.checkpoint.correlation.impl.drools.Log
 import com.checkpoint.correlation.impl.drools.CorrelatedEvent

 global com.checkpoint.correlation.server.EventsHandler externalEventsHandler;

 import function
com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInDayHourRange
 import function com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInIpRange

 function boolean filter(Log log) {
                 return  (!((log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(), "10.80.0.0",
"10.80.255.255")) || (log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(), "124.0.0.0",
"124.255.255.255")) || (log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(), "192.168.0.0",
"192.168.255.255")) || (log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(), "195.158.7.0",
"195.158.7.255")) || (log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(), "11.25.0.0",
"11.25.255.255")) || (log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(), "128.157.0.0",
"128.157.255.255")) || (log.fieldsMap.get("src")!= null &&
isInIpRange(log.fieldsMap.get("src").toString(), "213.114.0.0",
"213.114.255.255"))));
 }

 function String markersToString(Set markersSet) {
                 int i = 0;
                 String markersString = "";
                 for (Object marker : markersSet) {
                                 if (i == 25) break;
                                 String markerStr = marker.toString();
                                 if (i > 0) markersString += "\n";
                                 markersString += markerStr;
                 }
                 return markersString;
 }

 function String calcSeverity(Log log) {
                 return "High";
 }

 function String getUniqueId(Log log) {
                 String uniqueId="";
                 uniqueId += (log.fieldsMap.get("service") != null ?
log.fieldsMap.get("service").toString() : "null");
                 uniqueId += (log.fieldsMap.get("proto") != null ?
log.fieldsMap.get("proto").toString() : "null");
                 return uniqueId;
 }

                                                              &n! bsp;                

 declare Log
                 @role(event)
 end

 declare CorrelatedEvent
                 @role(event)
                 @expires(600s)
 end

 rule "Port scan from external network"
 enabled true
 dialect "java"
 no-loop
 when
                 $log : Log(eval(filter($log)))
                 not CorrelatedEvent(getId() ==
"{8AC52BA8-1EE8-4f18-9BB4-54492116501C}", groupByFieldsMap.get("src")
== $log.fieldsMap.get("src"), groupByFieldsMap.get("dst") ==
$log.fieldsMap.get("dst"))
                 accumulate($accumulatedLog : Log(eval(filter($accumulatedLog)), this
after[0s,60s] $log, fieldsMap.get("src") == $log.fieldsMap.get("src"),
fieldsMap.get("dst") == $log.fieldsMap.get("dst"), $id :
getUniqueId(this));
                                 $idSet : collectSet($id);
                                 $idSet.size > 19)
                 accumulate($accumulatedLog : Log(eval(filter($accumulatedLog)), this
after[0s,60s] $log, fieldsMap.get("src") == $log.fieldsMap.get("src"),
fieldsMap.get("dst") == $log.fieldsMap.get("dst"),
$idSet.contains(getUniqueId(this)), $marker : fieldsMap.get("marker"));
                                 $markerSet : collectSet($marker))
                 then
                 CorrelatedEvent $ce = new
CorrelatedEvent("{8AC52BA8-1EE8-4f18-9BB4-54492116501C}");
                 $ce.groupByFieldsMap.put("src",
$log.fieldsMap.get("src"));
                 $ce.groupByFieldsMap.put("dst",
$log.fieldsMap.get("dst"));
                 insert($ce);
                 HashMap<String,Object> fieldsMap = new
HashMap<String,Object>();
                 fieldsMap.put("cu_rule_id",
"{8AC52BA8-1EE8-4f18-9BB4-54492116501C}");
                 fieldsMap.put("event_name", "Port scan from external
network");
                 fieldsMap.put("cu_rule_severity", calcSeverity($log));
                 fieldsMap.put("cu_rule_category", "Scans");
                 fieldsMap.put("cu_log_count", $markerSet.size());
                 fieldsMap.put("time", new Date());
                 fieldsMap.put("cu_markers_list", markersToString($markerSet));
                 fieldsMap.put("src", $log.fieldsMap.get("src"));
                 fieldsMap.put("src_machine_name",
$log.fieldsMap.get("src_machine_name"));
                 fieldsMap.put("src_user_name",
$log.fieldsMap.get("src_user_name"));
                 fieldsMap.put("dst", $log.fieldsMap.get("dst"));
                 fieldsMap.put("dst_machine_name",
$log.fieldsMap.get("dst_machine_name"));
                 fieldsMap.put("dst_user_name",
$log.fieldsMap.get("dst_user_name"));
                 fieldsMap.put("service",
$log.fieldsMap.get("service"));
                 fieldsMap.put("proto", $log.fieldsMap.get("proto"));
                 fieldsMap.put("product",
$log.fieldsMap.get("product"));
                 externalEventsHandler.handleEvent(fieldsMap);
 end  

 I am sending logs in a rate of up to 200 logs/sec. After about 3 minutes, my application
starts to be unresponsive.
 I monitored the JVM with VisualVM. Two snapshots of VisualVM are attached.
 I found out that the class consuming most memory is FromNodeLeftTuple of drools (as can
be seen in “instances.png”).

 1)      Is my inserting rate is too high?

 2)      Is There a way I can make my rules more memory efficient?

 Thanks.

 Inserting logs:
 public void insertEvents(Collection<Map<String, Object>> logs)
 {
                 for (Map<String, Object> map : logs) {
                                 Log log = new Log();
                                 Log.fieldsMap.putAll(map);
                                 session.insert(log);
                                 session.fireAllRules();
                 }
 }

 Log class:
 public class Log
 {
     public HashMap<String, Object> fieldsMap = new HashMap<>();
 }

 CorrelatedEvent class:

 public class CorrelatedEvent
 {
     public Map<String, Object> groupByFieldsMap;

     private String id;

     public CorrelatedEvent(String id)
     {
         groupByFieldsMap = new HashMap<>();
         this.id = id;
     }

     public String getId()
     {
         return id;
     }
 }

 _______________________________________________
 rules-users mailing list
 rules-users(a)lists.jboss.org
 https://lists.jboss.org/mailman/listinfo/rules-users

 Email secured by Check Point

 _______________________________________________
 rules-users mailing list
 rules-users(a)lists.jboss.org
 https://lists.jboss.org/mailman/listinfo/rules-users

 _______________________________________________
 rules-users mailing list
 rules-users(a)lists.jboss.org
 https://lists.jboss.org/mailman/listinfo/rules-users 

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Re: [rules-users] Drools memory consumption