[rules-users] Implementation of my use case - what am I doing wrong?
Elran Dvir
elrand at checkpoint.com
Wed Oct 9 11:33:23 EDT 2013
Hi all,
I didn't get mail messages whole past week, so I didn't know if someone answered my questions.
If someone did answered me, it will much appreciated if he or she can send the answer again.
Thanks.
-----Original Message-----
From: Elran Dvir
Sent: Monday, September 30, 2013 4:40 PM
To: 'Rules Users List'
Subject: FW: [rules-users] Implementation of my use case - what am I doing wrong?
I wrote my question in the past, but now I want to ask about another strange behavior I have encountered.
I will rewrite all relevant data.
I am trying to identify a port scan event:
Basic event is connection log. For each combination of source_ip and destination_ip, detect a port scan event, if over 5 seconds there were more than 2 connection logs with different ports .
The event will stay open for 10 seconds and an update will be sent for any new port detected. Every update will contain the count of connection logs combining it and their id ("marker").
This is my drl file:
----------------------------------------------------------------------------------------------------
package test;
import correlation.impl.drools.Log
import correlation.impl.drools.CorrelatedEvent
global correlation.server.EventsHandler externalEventsHandler;
declare Log
@role( event)
end
declare CorrelatedEvent
//@propertyReactive
@role( event)
@timestamp( getTimestamp())
@expires( 10s )
@duration( getDuration() )
end
// this rule will create a "Port Scan" event if none exist for this group-by values rule "Create Port Scan Event"
dialect "java" //dialect can be either mvel or java for the RHS (only java dialect support generic containers)
no-loop //this means - do not revaluate rules if the events were altered in memory
when
$log : Log()
accumulate( Log( this after[0s,5s] $log, fieldsMap.get("src") == $log.fieldsMap.get("src") , fieldsMap.get("dst") == $log.fieldsMap.get("dst"), $port : fieldsMap.get("port"));
$portSet : collectSet($port);
$portSet.size > 2 )
accumulate( Log( this after[0s,5s] $log, fieldsMap.get("src") == $log.fieldsMap.get("src") , fieldsMap.get("dst") == $log.fieldsMap.get("dst") , $portSet.contains(fieldsMap.get("port")) , $marker : fieldsMap.get("marker"));
$markerSet : collectSet($marker))
not CorrelatedEvent(getName() == "portScan" , fieldsMap.get("src") == $log.fieldsMap.get("src") , fieldsMap.get("dst") == $log.fieldsMap.get("dst")) then
CorrelatedEvent $ce = new CorrelatedEvent();
$ce.setName("portScan");
$ce.setEventsHandler(externalEventsHandler);
$ce.setDurationInSec(10);
$ce.fieldsMap.put("src", $log.fieldsMap.get("src"));
$ce.fieldsMap.put("dst", $log.fieldsMap.get("dst"));
$ce.endUpdate($markerSet, $portSet);
insert($ce);
end
rule "Create Port Scan Event - update"
dialect "java" //dialect can be either mvel or java for the RHS (only java dialect support generic containers)
no-loop
when
$ce: CorrelatedEvent(getName() == "portScan", $portSet: getUniqueSet())
$log: Log(fieldsMap.get("src") == $ce.fieldsMap.get("src") , fieldsMap.get("dst") == $ce.fieldsMap.get("dst") , !$portSet.contains(fieldsMap.get("port")), this after $ce.getStartTime() , this before $ce.getEndTime()) then
modify( $ce ) {endUpdate($log.fieldsMap.get("marker").toString(), $log.fieldsMap.get("port"))} end
------------------------------------------------------------------------------------------------------------------------------------
I test it like this:
I insert a connection log and fire the rules every second. I have 25 logs with the same "src" and "dst", but each has different (serial) "port" and "marker".
So after 12-13 logs, I expect to identify a new event with another consecutive 3 logs.
In each rule's RHS, I print the rule fired and the ports accumulated so far.
With existing implementation, I see the following output at 14th second:
rule fired: Create Port Scan Event - update
portSet: [1, 7, 11, 8, 5, 6, 3, 10, 4, 9, 12, 2]
rule fired: Create Port Scan Event
portSet: [13, 11, 12]
As we can see, the first rule processes logs already processed by the second rule (11,12).
After examining the first rule, I thought I understood the reason for this behavior.
I changed the order of conditions in the LHS of the first rule by moving "not CorrelatedEvent..." to be the second condition.
But It still behaves the same.
Why is that? How can I prevent the first rule from processing logs already processed by the second rule?
Thanks a lot!
---------------------------------------------------------------
Log class:
----------------------------------------------------------------
public class Log {
public HashMap<String, Object> fieldsMap = new HashMap<>(); }
-------------------------------------------------------------------------------------------
CorrelatedEvent class:
---------------------------------------------------------------------------------------------
public class CorrelatedEvent
{
public Map<String, Object> fieldsMap;
public int duration; //in ms
public long timestamp;
private String name;
private Set<String> markersSet;
private Set uniqueSet;
private long logsCount;
private Calendar startTime;
private Calendar endTime;
private EventsHandler eventsHandler;
public CorrelatedEvent()
{
startTime = Calendar.getInstance();
timestamp = startTime.getTime().getTime();
endTime = Calendar.getInstance();
endTime.setTime(startTime.getTime());
duration = 0;
fieldsMap = new HashMap<>();
markersSet = Collections.newSetFromMap(new ConcurrentHashMap<String, Boolean>());
uniqueSet = Collections.newSetFromMap(new ConcurrentHashMap<Object, Boolean>());
logsCount = 0;
}
public Set getUniqueSet()
{
return uniqueSet;
}
public void endUpdate(Set<String> markersSet, Set uniqueSet)
{
this.markersSet.addAll(markersSet);
this.uniqueSet.addAll(uniqueSet);
handleUpdate();
}
public void endUpdate(String marker, Object uniqueValue)
{
markersSet.add(marker);
uniqueSet.add(uniqueValue);
handleUpdate();
}
private void handleUpdate()
{
if (markersSet.size() != uniqueSet.size())
return;
if (markersSet.size() > logsCount) {
logsCount = this.markersSet.size();
if (eventsHandler == null)
return;
Map<String, Object> clonedFieldsMap = getClonedFieldsMap();
clonedFieldsMap.put("markers", markersSet.toString());
clonedFieldsMap.put("count", logsCount);
eventsHandler.handleEvent(clonedFieldsMap);
}
}
private Map<String, Object> getClonedFieldsMap()
{
Map<String, Object> clonedFieldsMap = new HashMap<>();
clonedFieldsMap.putAll(fieldsMap);
return clonedFieldsMap;
}
public Date getStartTime()
{
return startTime.getTime();
}
public void setDurationInSec(int duration)
{
this.duration = duration * 1000;
endTime.setTime(startTime.getTime());
endTime.add(Calendar.SECOND, duration);
}
public Date getEndTime()
{
return endTime.getTime();
}
public long getTimestamp()
{
return timestamp;
}
public int getDuration()
{
return duration;
}
public String getName()
{
return name;
}
public void setName(String name)
{
this.name = name;
fieldsMap.put("name", name);
}
public EventsHandler getEventsHandler()
{
return eventsHandler;
}
public void setEventsHandler(EventsHandler eventsHandler)
{
this.eventsHandler = eventsHandler;
}
}
------------------------------------------------------------------------------------------------
-----Original Message-----
From: Elran Dvir
Sent: Sunday, September 29, 2013 11:01 AM
To: Rules Users List
Subject: RE: [rules-users] Implementation of my use case - what am I doing wrong?
Hi,
Thanks for the response.
I am not sure I fully understand your answer.
After all the rewriting of the rules, I decided to try "(this meets $ce || this during $ce || this metby $ce)" again.
Now the rule is fired, but an exception is thrown (not all times the rule is fired, just once between the third and fourth event):
Exception in thread "Thread-1" java.lang.RuntimeException: java.lang.NoSuchMethodException: com.checkpoint.correlation.impl.drools.Log.meets0()
at org.drools.rule.constraint.ConditionAnalyzer.analyzeNode(ConditionAnalyzer.java:239)
at org.drools.rule.constraint.ConditionAnalyzer.analyzeSingleCondition(ConditionAnalyzer.java:133)
at org.drools.rule.constraint.ConditionAnalyzer.analyzeCondition(ConditionAnalyzer.java:99)
at org.drools.rule.constraint.ConditionAnalyzer.analyzeCombinedCondition(ConditionAnalyzer.java:140)
at org.drools.rule.constraint.ConditionAnalyzer.analyzeCondition(ConditionAnalyzer.java:96)
at org.drools.rule.constraint.ConditionAnalyzer.analyzeCombinedCondition(ConditionAnalyzer.java:140)
at org.drools.rule.constraint.ConditionAnalyzer.analyzeCondition(ConditionAnalyzer.java:96)
at org.drools.rule.constraint.ConditionAnalyzer.analyzeCombinedCondition(ConditionAnalyzer.java:141)
at org.drools.rule.constraint.ConditionAnalyzer.analyzeCondition(ConditionAnalyzer.java:96)
at org.drools.rule.constraint.ConditionAnalyzer.analyzeCombinedCondition(ConditionAnalyzer.java:141)
at org.drools.rule.constraint.ConditionAnalyzer.analyzeCondition(ConditionAnalyzer.java:96)
at org.drools.rule.constraint.ConditionAnalyzer.analyzeCombinedCondition(ConditionAnalyzer.java:141)
at org.drools.rule.constraint.ConditionAnalyzer.analyzeCondition(ConditionAnalyzer.java:96)
at org.drools.rule.constraint.ConditionAnalyzer.analyzeCondition(ConditionAnalyzer.java:70)
at org.drools.rule.constraint.MvelConditionEvaluator.getAnalyzedCondition(MvelConditionEvaluator.java:83)
at org.drools.rule.constraint.MvelConstraint.executeJitting(MvelConstraint.java:270)
at org.drools.rule.constraint.MvelConstraint.access$200(MvelConstraint.java:51)
at org.drools.rule.constraint.MvelConstraint$ConditionJitter.run(MvelConstraint.java:250)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.NoSuchMethodException: com.checkpoint.correlation.impl.drools.Log.meets0()
at java.lang.Class.getMethod(Class.java:1622)
at org.drools.rule.constraint.ConditionAnalyzer.analyzeNode(ConditionAnalyzer.java:230)
... 20 more
The complete latest drl file is followed.
Duration for Log is 0, duration for CorrelatedEvent is 10s.
Did you have a chance to look at my (original) fourth question? This is the issue that bothers me the most.
If necessary, I'll write everything again.
Thanks a lot.
This is my drl file:
----------------------------------------------------------------------------------------------------
package test;
import correlation.impl.drools.Log
import correlation.impl.drools.CorrelatedEvent
global correlation.server.EventsHandler externalEventsHandler;
declare Log
@role( event)
end
declare CorrelatedEvent
//@propertyReactive
@role( event)
@timestamp( getTimestamp())
@expires( 10s )
@duration( getDuration() )
end
// this rule will create a "Port Scan" event if none exist for this group-by values rule "Create Port Scan Event"
dialect "java" //dialect can be either mvel or java for the RHS (only java dialect support generic containers)
no-loop //this means - do not revaluate rules if the events were altered in memory
when
$log : Log()
accumulate( Log( this after[0s,5s] $log, fieldsMap.get("src") == $log.fieldsMap.get("src") , fieldsMap.get("dst") == $log.fieldsMap.get("dst"), $port : fieldsMap.get("port"));
$portSet : collectSet($port);
$portSet.size > 2 )
accumulate( Log( this after[0s,5s] $log, fieldsMap.get("src") == $log.fieldsMap.get("src") , fieldsMap.get("dst") == $log.fieldsMap.get("dst") , $portSet.contains(fieldsMap.get("port")) , $marker : fieldsMap.get("marker"));
$markerSet : collectSet($marker))
not CorrelatedEvent(getName() == "portScan" , fieldsMap.get("src") == $log.fieldsMap.get("src") , fieldsMap.get("dst") == $log.fieldsMap.get("dst")) then
CorrelatedEvent $ce = new CorrelatedEvent();
$ce.setName("portScan");
$ce.setEventsHandler(externalEventsHandler);
$ce.setDurationInSec(10);
$ce.fieldsMap.put("src", $log.fieldsMap.get("src"));
$ce.fieldsMap.put("dst", $log.fieldsMap.get("dst"));
$ce.endUpdate($markerSet, $portSet);
insert($ce);
end
rule "Create Port Scan Event - update"
dialect "java" //dialect can be either mvel or java for the RHS (only java dialect support generic containers)
no-loop
when
$ce: CorrelatedEvent(getName() == "portScan", $portSet: getUniqueSet())
$log: Log(fieldsMap.get("src") == $ce.fieldsMap.get("src") , fieldsMap.get("dst") == $ce.fieldsMap.get("dst") , !$portSet.contains(fieldsMap.get("port")), (this meets $ce || this during $ce || this metby $ce)) then
modify( $ce ) {endUpdate($log.fieldsMap.get("marker").toString(), $log.fieldsMap.get("port"))} end
------------------------------------------------------------------------------------------------------------------------------------
-----Original Message-----
From: rules-users-bounces at lists.jboss.org [mailto:rules-users-bounces at lists.jboss.org] On Behalf Of Wolfgang Laun
Sent: Sunday, September 22, 2013 7:06 PM
To: Rules Users List
Subject: Re: [rules-users] Implementation of my use case - what am I doing wrong?
On 22/09/2013, Elran Dvir <elrand at checkpoint.com> wrote:
> Thanks for the response.
>
>>Computationally, during the evaluation of the temporal operators.
>>Why would you want to do this? (My advice to replace the temporal
>>operators by the equivalent expressions was meant as a debugging aid,
>>to show you where the problem with this constraint is.)
> So if changing the condition to use startTimeStamp and endTimestamp
> caused a compilation error, How can I debug my problem? What is the
> equivalent of "(this meets $ce || this during $ce || this metby $ce)"
> without my own startTimeStamp and endTimestamp fields?
>
So I'm quoting my reply from an earlier mail <quote>
> Elran Dvir <elrand at checkpoint.com> wrote:
>
> 2) When I tested it with matching data, rule "Create Port Scan Event -
> update" was never fired. When I replaced "(this meets $ce || this
> during $ce
> || this metby $ce)" with "this after $ce.getStartTime() , this before
> $ce.getEndTime()" everything worked fine.
> Why?
Just take the constraints and replace the temporal operator by its definition in the "Fusion" manual and use a little elementary math:
A meets || A during B || A metby B
becomes
abs( B.startTimestamp - A.endTimestamp ) == 0 ||
B.startTimestamp < A.startTimestamp && A.endTimestamp < B.endTimestamp || abs( A.startTimestamp - B.endTimestamp ) == 0 becomes
...
</quote>
Did you overlook the "use a little elementary math"? YOu'll have to provide the values for start and, given the duration (zero,
presumably) the end time. If B has a duration of 0, there's no way the second term can ever be true. And if A has a duration of 0 too, A and B must have the same timestamp in order for the other terms to be true.
Note that "debugging" was meant to be done offline ;-)
> > - Why do you use this complex declare?
> > @timestamp( timestamp ) @duration( duration )
> > is sufficient.
>
> I want to protect the fields from being set outside their designated
> functions .I simplified it a bit in my correction.
I don't know of any danger of your field being set outside their designated functions by simply writing the field name between the parentheses.
-W
_______________________________________________
rules-users mailing list
rules-users at lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
Email secured by Check Point
More information about the rules-users
mailing list