<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:355547441;
mso-list-type:hybrid;
mso-list-template-ids:-1666445314 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi all,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am using Drools Fusion. I am getting OutOfMemoryError rather fast. My JVM is running with –Xmx4g flag.<o:p></o:p></p>
<p class="MsoNormal">I have rules defined in another (not Drools) language.<o:p></o:p></p>
<p class="MsoNormal">Every rule is translated programmatically to a drl file. This is because the user can add and remove rules (in the other language) dynamically.<o:p></o:p></p>
<p class="MsoNormal">The default configuration contains 125 rules.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">For example, one rule is supposed to identify a port scan event.<o:p></o:p></p>
<p class="MsoPlainText">The basic fact is connection log. For each combination of src (source IP) and dst (destination IP) , detect a port scan event, if over 60 seconds there were at least 20 connection logs with different service and protocol.<o:p></o:p></p>
<p class="MsoPlainText">The event will stay closed for 10 minute - no event will be sent during this time for this combination of src and dst. The event the connection logs’ ids (markers).<o:p></o:p></p>
<p class="MsoNormal">(other rules are very similar in structure, but different in logic, of course)
<o:p></o:p></p>
<div style="mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in"><o:p> </o:p></p>
<p class="MsoNormal" style="border:none;padding:0in">This is its programmatic drl file:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">package com.checkpoint.correlation.impl.drools.package30;<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">import java.util.Date<o:p></o:p></p>
<p class="MsoNormal">import java.util.HashMap<o:p></o:p></p>
<p class="MsoNormal">import java.util.Set<o:p></o:p></p>
<p class="MsoNormal">import com.checkpoint.correlation.impl.drools.Log<o:p></o:p></p>
<p class="MsoNormal">import com.checkpoint.correlation.impl.drools.CorrelatedEvent<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">global com.checkpoint.correlation.server.EventsHandler externalEventsHandler;<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">import function com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInDayHourRange<o:p></o:p></p>
<p class="MsoNormal">import function com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInIpRange<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">function boolean filter(Log log) { <o:p></o:p></p>
<p class="MsoNormal"> return (!((log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "10.80.0.0", "10.80.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "124.0.0.0",
"124.255.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "192.168.0.0", "192.168.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "195.158.7.0", "195.158.7.255"))
|| (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "11.25.0.0", "11.25.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "128.157.0.0", "128.157.255.255")) || (log.fieldsMap.get("src")!=
null && isInIpRange(log.fieldsMap.get("src").toString(), "213.114.0.0", "213.114.255.255"))));<o:p></o:p></p>
<p class="MsoNormal">}<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">function String markersToString(Set markersSet) { <o:p></o:p></p>
<p class="MsoNormal"> int i = 0;<o:p></o:p></p>
<p class="MsoNormal"> String markersString = "";<o:p></o:p></p>
<p class="MsoNormal"> for (Object marker : markersSet) { <o:p></o:p></p>
<p class="MsoNormal"> if (i == 25) break;<o:p></o:p></p>
<p class="MsoNormal"> String markerStr = marker.toString();<o:p></o:p></p>
<p class="MsoNormal"> if (i > 0) markersString += "\n";<o:p></o:p></p>
<p class="MsoNormal"> markersString += markerStr;<o:p></o:p></p>
<p class="MsoNormal"> }<o:p></o:p></p>
<p class="MsoNormal"> return markersString;<o:p></o:p></p>
<p class="MsoNormal">}<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">function String calcSeverity(Log log) { <o:p></o:p></p>
<p class="MsoNormal"> return "High";<o:p></o:p></p>
<p class="MsoNormal">}<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">function String getUniqueId(Log log) { <o:p></o:p></p>
<p class="MsoNormal"> String uniqueId="";<o:p></o:p></p>
<p class="MsoNormal"> uniqueId += (log.fieldsMap.get("service") != null ? log.fieldsMap.get("service").toString() : "null");<o:p></o:p></p>
<p class="MsoNormal"> uniqueId += (log.fieldsMap.get("proto") != null ? log.fieldsMap.get("proto").toString() : "null");<o:p></o:p></p>
<p class="MsoNormal"> return uniqueId;<o:p></o:p></p>
<p class="MsoNormal">}<o:p></o:p></p>
<p class="MsoNormal">
<o:p></o:p></p>
<p class="MsoNormal">declare Log<o:p></o:p></p>
<p class="MsoNormal"> @role(event)<o:p></o:p></p>
<p class="MsoNormal">end<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">declare CorrelatedEvent<o:p></o:p></p>
<p class="MsoNormal"> @role(event)<o:p></o:p></p>
<p class="MsoNormal"> @expires(600s)<o:p></o:p></p>
<p class="MsoNormal">end<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">rule "Port scan from external network"<o:p></o:p></p>
<p class="MsoNormal">enabled true<o:p></o:p></p>
<p class="MsoNormal">dialect "java"<o:p></o:p></p>
<p class="MsoNormal">no-loop<o:p></o:p></p>
<p class="MsoNormal">when<o:p></o:p></p>
<p class="MsoNormal"> $log : Log(eval(filter($log)))<o:p></o:p></p>
<p class="MsoNormal"> not CorrelatedEvent(getId() == "{8AC52BA8-1EE8-4f18-9BB4-54492116501C}", groupByFieldsMap.get("src") == $log.fieldsMap.get("src"), groupByFieldsMap.get("dst") == $log.fieldsMap.get("dst"))<o:p></o:p></p>
<p class="MsoNormal"> accumulate($accumulatedLog : Log(eval(filter($accumulatedLog)), this after[0s,60s] $log, fieldsMap.get("src") == $log.fieldsMap.get("src"), fieldsMap.get("dst") == $log.fieldsMap.get("dst"), $id : getUniqueId(this));<o:p></o:p></p>
<p class="MsoNormal"> $idSet : collectSet($id);<o:p></o:p></p>
<p class="MsoNormal"> $idSet.size > 19)<o:p></o:p></p>
<p class="MsoNormal"> accumulate($accumulatedLog : Log(eval(filter($accumulatedLog)), this after[0s,60s] $log, fieldsMap.get("src") == $log.fieldsMap.get("src"), fieldsMap.get("dst") == $log.fieldsMap.get("dst"), $idSet.contains(getUniqueId(this)),
$marker : fieldsMap.get("marker"));<o:p></o:p></p>
<p class="MsoNormal"> $markerSet : collectSet($marker))<o:p></o:p></p>
<p class="MsoNormal"> then<o:p></o:p></p>
<p class="MsoNormal"> CorrelatedEvent $ce = new CorrelatedEvent("{8AC52BA8-1EE8-4f18-9BB4-54492116501C}");<o:p></o:p></p>
<p class="MsoNormal"> $ce.groupByFieldsMap.put("src", $log.fieldsMap.get("src"));<o:p></o:p></p>
<p class="MsoNormal"> $ce.groupByFieldsMap.put("dst", $log.fieldsMap.get("dst"));<o:p></o:p></p>
<p class="MsoNormal"> insert($ce);<o:p></o:p></p>
<p class="MsoNormal"> HashMap<String,Object> fieldsMap = new HashMap<String,Object>();<o:p></o:p></p>
<p class="MsoNormal"> fieldsMap.put("cu_rule_id", "{8AC52BA8-1EE8-4f18-9BB4-54492116501C}");<o:p></o:p></p>
<p class="MsoNormal"> fieldsMap.put("event_name", "Port scan from external network");<o:p></o:p></p>
<p class="MsoNormal"> fieldsMap.put("cu_rule_severity", calcSeverity($log));<o:p></o:p></p>
<p class="MsoNormal"> fieldsMap.put("cu_rule_category", "Scans");<o:p></o:p></p>
<p class="MsoNormal"> fieldsMap.put("cu_log_count", $markerSet.size());<o:p></o:p></p>
<p class="MsoNormal"> fieldsMap.put("time", new Date());<o:p></o:p></p>
<p class="MsoNormal"> fieldsMap.put("cu_markers_list", markersToString($markerSet));<o:p></o:p></p>
<p class="MsoNormal"> fieldsMap.put("src", $log.fieldsMap.get("src"));<o:p></o:p></p>
<p class="MsoNormal"> fieldsMap.put("src_machine_name", $log.fieldsMap.get("src_machine_name"));<o:p></o:p></p>
<p class="MsoNormal"> fieldsMap.put("src_user_name", $log.fieldsMap.get("src_user_name"));<o:p></o:p></p>
<p class="MsoNormal"> fieldsMap.put("dst", $log.fieldsMap.get("dst"));<o:p></o:p></p>
<p class="MsoNormal"> fieldsMap.put("dst_machine_name", $log.fieldsMap.get("dst_machine_name"));<o:p></o:p></p>
<p class="MsoNormal"> fieldsMap.put("dst_user_name", $log.fieldsMap.get("dst_user_name"));<o:p></o:p></p>
<p class="MsoNormal"> fieldsMap.put("service", $log.fieldsMap.get("service"));<o:p></o:p></p>
<p class="MsoNormal"> fieldsMap.put("proto", $log.fieldsMap.get("proto"));<o:p></o:p></p>
<p class="MsoNormal"> fieldsMap.put("product", $log.fieldsMap.get("product"));<o:p></o:p></p>
<p class="MsoNormal"> externalEventsHandler.handleEvent(fieldsMap);<o:p></o:p></p>
<div style="mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in">end <o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am sending logs in a rate of up to 200 logs/sec. After about 3 minutes, my application starts to be unresponsive.
<o:p></o:p></p>
<p class="MsoNormal">I monitored the JVM with VisualVM. Two snapshots of VisualVM are attached.<o:p></o:p></p>
<p class="MsoNormal">I found out that the class consuming most memory is FromNodeLeftTuple of drools (as can be seen in “instances.png”).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">1)<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]><span dir="LTR"></span>Is my inserting rate is too high?<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">2)<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]><span dir="LTR"></span>Is There a way I can make my rules more memory efficient?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in">Inserting logs:<o:p></o:p></p>
</div>
<p class="MsoNormal">public void insertEvents(Collection<Map<String, Object>> logs)<o:p></o:p></p>
<p class="MsoNormal">{<o:p></o:p></p>
<p class="MsoNormal"> for (Map<String, Object> map : logs) {<o:p></o:p></p>
<p class="MsoNormal"> Log log = new Log();<o:p></o:p></p>
<p class="MsoNormal"> Log.fieldsMap.putAll(map);<o:p></o:p></p>
<p class="MsoNormal"> session.insert(log);<o:p></o:p></p>
<p class="MsoNormal"> session.fireAllRules();<o:p></o:p></p>
<p class="MsoNormal"> }<o:p></o:p></p>
<div style="mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in">}<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in">Log class:<o:p></o:p></p>
</div>
<p class="MsoNormal">public class Log<o:p></o:p></p>
<p class="MsoNormal">{<o:p></o:p></p>
<p class="MsoNormal"> public HashMap<String, Object> fieldsMap = new HashMap<>();<o:p></o:p></p>
<p class="MsoNormal">}<o:p></o:p></p>
<div style="mso-element:para-border-div;border-top:solid windowtext 1.0pt;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:none;padding:1.0pt 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in"><o:p> </o:p></p>
<p class="MsoNormal" style="border:none;padding:0in">CorrelatedEvent class:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">public class CorrelatedEvent<o:p></o:p></p>
<p class="MsoNormal">{<o:p></o:p></p>
<p class="MsoNormal"> public Map<String, Object> groupByFieldsMap;<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> private String id;<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> public CorrelatedEvent(String id)<o:p></o:p></p>
<p class="MsoNormal"> {<o:p></o:p></p>
<p class="MsoNormal"> groupByFieldsMap = new HashMap<>();<o:p></o:p></p>
<p class="MsoNormal"> this.id = id;<o:p></o:p></p>
<p class="MsoNormal"> }<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> public String getId()<o:p></o:p></p>
<p class="MsoNormal"> {<o:p></o:p></p>
<p class="MsoNormal"> return id;<o:p></o:p></p>
<p class="MsoNormal"> }<o:p></o:p></p>
<div style="mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in">}<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>