Drools memory consumption
by Elran Dvir
Hi all,
I am using Drools Fusion. I am getting OutOfMemoryError rather fast. My JVM is running with -Xmx4g flag.
I have rules defined in another (not Drools) language.
Every rule is translated programmatically to a drl file. This is because the user can add and remove rules (in the other language) dynamically.
The default configuration contains 125 rules.
For example, one rule is supposed to identify a port scan event.
The basic fact is connection log. For each combination of src (source IP) and dst (destination IP) , detect a port scan event, if over 60 seconds there were at least 20 connection logs with different service and protocol.
The event will stay closed for 10 minute - no event will be sent during this time for this combination of src and dst. The event the connection logs' ids (markers).
(other rules are very similar in structure, but different in logic, of course)
This is its programmatic drl file:
package com.checkpoint.correlation.impl.drools.package30;
import java.util.Date
import java.util.HashMap
import java.util.Set
import com.checkpoint.correlation.impl.drools.Log
import com.checkpoint.correlation.impl.drools.CorrelatedEvent
global com.checkpoint.correlation.server.EventsHandler externalEventsHandler;
import function com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInDayHourRange
import function com.checkpoint.correlation.impl.utils.UserDefinedFunctions.isInIpRange
function boolean filter(Log log) {
return (!((log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "10.80.0.0", "10.80.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "124.0.0.0", "124.255.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "192.168.0.0", "192.168.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "195.158.7.0", "195.158.7.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "11.25.0.0", "11.25.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "128.157.0.0", "128.157.255.255")) || (log.fieldsMap.get("src")!= null && isInIpRange(log.fieldsMap.get("src").toString(), "213.114.0.0", "213.114.255.255"))));
}
function String markersToString(Set markersSet) {
int i = 0;
String markersString = "";
for (Object marker : markersSet) {
if (i == 25) break;
String markerStr = marker.toString();
if (i > 0) markersString += "\n";
markersString += markerStr;
}
return markersString;
}
function String calcSeverity(Log log) {
return "High";
}
function String getUniqueId(Log log) {
String uniqueId="";
uniqueId += (log.fieldsMap.get("service") != null ? log.fieldsMap.get("service").toString() : "null");
uniqueId += (log.fieldsMap.get("proto") != null ? log.fieldsMap.get("proto").toString() : "null");
return uniqueId;
}
declare Log
@role(event)
end
declare CorrelatedEvent
@role(event)
@expires(600s)
end
rule "Port scan from external network"
enabled true
dialect "java"
no-loop
when
$log : Log(eval(filter($log)))
not CorrelatedEvent(getId() == "{8AC52BA8-1EE8-4f18-9BB4-54492116501C}", groupByFieldsMap.get("src") == $log.fieldsMap.get("src"), groupByFieldsMap.get("dst") == $log.fieldsMap.get("dst"))
accumulate($accumulatedLog : Log(eval(filter($accumulatedLog)), this after[0s,60s] $log, fieldsMap.get("src") == $log.fieldsMap.get("src"), fieldsMap.get("dst") == $log.fieldsMap.get("dst"), $id : getUniqueId(this));
$idSet : collectSet($id);
$idSet.size > 19)
accumulate($accumulatedLog : Log(eval(filter($accumulatedLog)), this after[0s,60s] $log, fieldsMap.get("src") == $log.fieldsMap.get("src"), fieldsMap.get("dst") == $log.fieldsMap.get("dst"), $idSet.contains(getUniqueId(this)), $marker : fieldsMap.get("marker"));
$markerSet : collectSet($marker))
then
CorrelatedEvent $ce = new CorrelatedEvent("{8AC52BA8-1EE8-4f18-9BB4-54492116501C}");
$ce.groupByFieldsMap.put("src", $log.fieldsMap.get("src"));
$ce.groupByFieldsMap.put("dst", $log.fieldsMap.get("dst"));
insert($ce);
HashMap<String,Object> fieldsMap = new HashMap<String,Object>();
fieldsMap.put("cu_rule_id", "{8AC52BA8-1EE8-4f18-9BB4-54492116501C}");
fieldsMap.put("event_name", "Port scan from external network");
fieldsMap.put("cu_rule_severity", calcSeverity($log));
fieldsMap.put("cu_rule_category", "Scans");
fieldsMap.put("cu_log_count", $markerSet.size());
fieldsMap.put("time", new Date());
fieldsMap.put("cu_markers_list", markersToString($markerSet));
fieldsMap.put("src", $log.fieldsMap.get("src"));
fieldsMap.put("src_machine_name", $log.fieldsMap.get("src_machine_name"));
fieldsMap.put("src_user_name", $log.fieldsMap.get("src_user_name"));
fieldsMap.put("dst", $log.fieldsMap.get("dst"));
fieldsMap.put("dst_machine_name", $log.fieldsMap.get("dst_machine_name"));
fieldsMap.put("dst_user_name", $log.fieldsMap.get("dst_user_name"));
fieldsMap.put("service", $log.fieldsMap.get("service"));
fieldsMap.put("proto", $log.fieldsMap.get("proto"));
fieldsMap.put("product", $log.fieldsMap.get("product"));
externalEventsHandler.handleEvent(fieldsMap);
end
I am sending logs in a rate of up to 200 logs/sec. After about 3 minutes, my application starts to be unresponsive.
I monitored the JVM with VisualVM. Two snapshots of VisualVM are attached.
I found out that the class consuming most memory is FromNodeLeftTuple of drools (as can be seen in "instances.png").
1) Is my inserting rate is too high?
2) Is There a way I can make my rules more memory efficient?
Thanks.
Inserting logs:
public void insertEvents(Collection<Map<String, Object>> logs)
{
for (Map<String, Object> map : logs) {
Log log = new Log();
Log.fieldsMap.putAll(map);
session.insert(log);
session.fireAllRules();
}
}
Log class:
public class Log
{
public HashMap<String, Object> fieldsMap = new HashMap<>();
}
CorrelatedEvent class:
public class CorrelatedEvent
{
public Map<String, Object> groupByFieldsMap;
private String id;
public CorrelatedEvent(String id)
{
groupByFieldsMap = new HashMap<>();
this.id = id;
}
public String getId()
{
return id;
}
}
10 years, 6 months
Help on writing a rule
by calcacuervo
Hi Guys. I have one quesiton on how could I implement my rule.
I have this rule:
when
t : PriceChange() over window:length(1)
signal : Signal( id == "1234" ) over window:length(1)
not OrderCreated( id == "1234" ) over window:time (1m)
then
insert(new OrderCreated("1234));
Basically, when there is some signal and some price change, create a new
order if there has no been any other order in the last minute.
But this sometimes is being fired twice:
12:31:01,121 INFO [org.drools.audit.WorkingMemoryConsoleLogger] (Camel
(camel-1) thread #4 - JmsConsumer[marketDataTopic]) ACTIVATION CREATED
rule:ertetr activationId:ertetr [393, 392, 0] declarations: t=PriceChange
[](392); signal=Signal@1413ddef(393)
12:31:01,160 INFO [org.drools.audit.WorkingMemoryConsoleLogger] (Camel
(camel-1) thread #4 - JmsConsumer[marketDataTopic]) ACTIVATION CREATED
rule:ertetr activationId:ertetr [393, 394, 0] declarations: t=PriceChange
[](394); signal=Signal@1413ddef(393)
I think that, when the 1m window have passed, it creates twice the
activations, as I have not called fire all rules yet. Does is make sense? Do
you have some idea on how could I implement this rule? "when there has been
some price change and some signal, and I did not create a new order in the
last minute, just create a new order".
Thanks in advance!
Demian
--
View this message in context: http://drools.46999.n3.nabble.com/Help-on-writing-a-rule-tp4026461.html
Sent from the Drools: User forum mailing list archive at Nabble.com.
10 years, 6 months
multiple expiration definitions for the same face type in different drl files
by Elran Dvir
Hi all,
I am examining Drools Fusion in stream mode.
This is my case:
I have several drl files.
Each drl file contains one rule.
Each drl file is built to a knowledge package.
Each drl file contains declaration of the same fact (referencing to a imported class defined outside of the drl). The fact's role is of course event.
Can I define different expiration (@expires) for the fact in each drl?
Is expiration relevant only to the same drl?
Is the answer different if the fact is created in the rule's RHS or if it inserted to the session's working memory outside of the drl?
Thanks.
10 years, 6 months
Drools Rules : Getting Started Tutorial Link Required
by Zahid Ahmed
Hi,
I am new to Drools Rules and using Drools-5.5.0.Final in my project. I need to know , how to setup JBPM runtime engine to perform rule evaluation for BusinessRule Nodes. Is there any separate server required for it or can it be done with in JBPM runtime. I need a tutorial on setup/configurations and a tutorial on rule writing, model uploading. I need to know that how to pass fact dynamically at runtime when a Business Rule Node is reached.
Thanks and Best Regards,
Zahid Ahmed
Senior Software Engineer
10 years, 6 months
Guvnor -> Best Practices for Changes in Production / DEV Environment
by Peach Wyss
Hy all
We're having the following Environments:
- DEV (dev site)
-> Guvnor WebApp: Used for defining the rules and exporting the xxx_rules.pkg during development.
-> XXX WebApp: Our WebApp which uses the xxx_rules.pkg for executing the rules, for testing during development.
-----------------------
- PROD (customer site)
-> XXX WebApp: WebApp in production.
- CHANGE (customer site)
-> Guvnor WebApp: For changing/adding rules by customer itself and exporting the changed xxx_rules.pkg by the customer
-> XXX WebApp: For Testing the changed xxx_rules.pkg by the customer
We're using Drools Guvnor (5.5.0 Final).
Now the question:
- How to deal with changes of the customer to the rules? We need those changes in our dev-guvnor too...
- So how to deal with the following scenario:
-> Customer changes a rule in Guvnor on the CHANGE Environment and tests the rule on his CHANGE Environment
-> After successful tests, he overwrites the xxx_rules.pkg in the PROD Environment
-> In the same time, the development team is adding new rules or changing existing rules on the DEV Environment (for example for Bugfixing or when adding new features to the WebApp)
-> After successful testing on DEV-Site, we want to deploy the new Version of our WebApp (and xxx:_rules.pkg) to the Customer (CHANGE and PROD). But when we just build the xxx_rules.pkg in our DEV-Guvnor and overwrite the xxx_rules.pkg from the customer, the changes from the Customer are lost. So how can we get the changes to the rules from the Customer to our DEV-Guvnor? The only way we now, is to export/import the whole repository. But with that, the changes (updated/created rules) from our DEV-Guvnor are overwritten...
How to deal with such a scenario? Is it possible to import/export single rules instead of the whole repository?
Thanks for your help,
Peter
10 years, 6 months
guvnor startup error
by sravan008
Hi I am using 5.5 drools war .when the time of deploying ,each time it
throwing below error .I request you to please let me know why I am getting
this error ,and please tell me how to resolve this .
INFO: Deploying configuration descriptor drools.xml
Oct 17, 2013 10:55:57 AM org.apache.catalina.loader.WebappClassLoader
validateJa
rFile
INFO: validateJarFile(C:\Documents and
Settings\Administrator\Desktop\apache-tom
cat-6.0.36\webapps\drools\WEB-INF\lib\jboss-servlet-api_3.0_spec-1.0.0.Final.jar
) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class:
javax/
servlet/Servlet.class
Oct 17, 2013 10:56:04 AM org.apache.catalina.session.StandardManager start
SEVERE: Exception loading sessions from persistent storage
java.lang.IllegalStateException: Singleton is not set
at
org.jboss.weld.bootstrap.api.helpers.IsolatedStaticSingletonProvider$
IsolatedStaticSingleton.get(IsolatedStaticSingletonProvider.java:52)
at org.jboss.weld.Container.instance(Container.java:54)
at
org.jboss.weld.manager.BeanManagerImpl.readResolve(BeanManagerImpl.ja
va:904)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:10
91)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1
786)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:19
--
View this message in context: http://drools.46999.n3.nabble.com/guvnor-startup-error-tp4026413.html
Sent from the Drools: User forum mailing list archive at Nabble.com.
10 years, 6 months
Integration between Drools and Infinispan?
by Keith West
Wanted to see if anyone has given any thought to (or implemented) an integration between Drools (Expert, Fusion, or jBPM) and Infinispan - in particular to have the working memory/facts or workflow processes be housed in an Infinispan data grid? if so, I'd like to find out more about this.
Thanks,
Keith West
10 years, 6 months
