Re: [rules-users] Implementation of my use case - what am I doing wrong?

Wednesday, 9 October 2013

Hi all,

I didn't get mail messages whole past week, so I didn't know if someone answered
my questions.
If someone did answered me, it will much appreciated if he or she can send the answer
again.

Thanks.

-----Original Message-----
From: Elran Dvir 
Sent: Monday, September 30, 2013 4:40 PM
To: 'Rules Users List'
Subject: FW: [rules-users] Implementation of my use case - what am I doing wrong?

I wrote my question in the past, but now I want to ask about another strange behavior I
have encountered.
I will rewrite all relevant data.

I am trying to identify a port scan event: 
Basic event is connection log. For each combination of source_ip and destination_ip,
detect a port scan event, if over 5 seconds there were more than 2 connection logs with
different ports .
The event will stay open for 10 seconds and an update will be sent for any new port
detected. Every update will contain the count of connection logs combining it and their id
("marker").

This is my drl file:
----------------------------------------------------------------------------------------------------
package test;

import correlation.impl.drools.Log
import correlation.impl.drools.CorrelatedEvent

global correlation.server.EventsHandler externalEventsHandler;

declare Log
      @role( event)
end

declare CorrelatedEvent
        //@propertyReactive
        @role( event)
        @timestamp( getTimestamp())
        @expires( 10s )
        @duration( getDuration() )
end

// this rule will create a "Port Scan" event if none exist for this group-by
values rule "Create Port Scan Event"
dialect "java"    //dialect can be either mvel or java for the RHS (only java
dialect support generic containers)
no-loop                //this means - do not revaluate rules if the events were altered in
memory
when
                $log : Log() 
                accumulate( Log( this after[0s,5s] $log, fieldsMap.get("src") ==
$log.fieldsMap.get("src") , fieldsMap.get("dst") ==
$log.fieldsMap.get("dst"), $port : fieldsMap.get("port"));
                               	$portSet : collectSet($port);
                               	$portSet.size > 2 )
	   accumulate( Log( this after[0s,5s] $log, fieldsMap.get("src") ==
$log.fieldsMap.get("src") , fieldsMap.get("dst") ==
$log.fieldsMap.get("dst") , $portSet.contains(fieldsMap.get("port")) ,
$marker : fieldsMap.get("marker"));
                              	 $markerSet : collectSet($marker))
                not CorrelatedEvent(getName() == "portScan" ,
fieldsMap.get("src") == $log.fieldsMap.get("src") ,
fieldsMap.get("dst") == $log.fieldsMap.get("dst"))  then
  CorrelatedEvent $ce = new CorrelatedEvent();
  $ce.setName("portScan");
  $ce.setEventsHandler(externalEventsHandler);
  $ce.setDurationInSec(10);
  $ce.fieldsMap.put("src", $log.fieldsMap.get("src"));
  $ce.fieldsMap.put("dst", $log.fieldsMap.get("dst"));
  $ce.endUpdate($markerSet, $portSet);

  insert($ce);
end

rule "Create Port Scan Event - update"
dialect "java"    //dialect can be either mvel or java for the RHS (only java
dialect support generic containers)
no-loop
when
                $ce: CorrelatedEvent(getName() == "portScan", $portSet:
getUniqueSet())
                $log: Log(fieldsMap.get("src") ==
$ce.fieldsMap.get("src") , fieldsMap.get("dst") ==
$ce.fieldsMap.get("dst") , !$portSet.contains(fieldsMap.get("port")),
this after $ce.getStartTime() , this before $ce.getEndTime())  then
  modify( $ce ) {endUpdate($log.fieldsMap.get("marker").toString(),
$log.fieldsMap.get("port"))} end
------------------------------------------------------------------------------------------------------------------------------------
I test it like this:
I insert a connection log and fire the rules every second. I have 25 logs with the same
"src" and "dst", but each has different (serial) "port" and
"marker".
So after 12-13 logs, I expect to identify a new event with another consecutive 3 logs.
In each rule's RHS, I print the rule fired and the ports accumulated so far. 
With existing implementation, I see the following output at 14th second:

	rule fired: Create Port Scan Event - update
	portSet: [1, 7, 11, 8, 5, 6, 3, 10, 4, 9, 12, 2]

	rule fired: Create Port Scan Event
	portSet: [13, 11, 12]

As we can see, the first rule processes logs already processed by the second rule
(11,12).
After examining the first rule, I thought I understood the reason for this behavior. 
I  changed the order of conditions in the LHS of the first rule by moving "not
CorrelatedEvent..." to be the second condition.
But It still behaves the same. 

Why is that? How can I prevent the first rule from processing logs already processed by
the second rule? 

Thanks a lot!
---------------------------------------------------------------

Log class:
----------------------------------------------------------------
public class Log {
    public HashMap<String, Object> fieldsMap = new HashMap<>(); }
-------------------------------------------------------------------------------------------

CorrelatedEvent class:
---------------------------------------------------------------------------------------------
public class CorrelatedEvent
{
    public Map<String, Object> fieldsMap;
    public int duration; //in ms
    public long timestamp;

    private String name;
    private Set<String> markersSet;
    private Set uniqueSet;
    private long logsCount;
    private Calendar startTime;
    private Calendar endTime;
    private EventsHandler eventsHandler;

    public CorrelatedEvent()
    {
        startTime = Calendar.getInstance();
        timestamp = startTime.getTime().getTime();
        endTime = Calendar.getInstance();
        endTime.setTime(startTime.getTime());
        duration = 0;
        fieldsMap = new HashMap<>();

        markersSet = Collections.newSetFromMap(new ConcurrentHashMap<String,
Boolean>());
        uniqueSet = Collections.newSetFromMap(new ConcurrentHashMap<Object,
Boolean>());
        logsCount = 0;
    }

	public Set getUniqueSet()
    {
        return uniqueSet;
    }

    public void endUpdate(Set<String> markersSet, Set uniqueSet)
    {
        this.markersSet.addAll(markersSet);
        this.uniqueSet.addAll(uniqueSet);
        handleUpdate();
    }

    public void endUpdate(String marker, Object uniqueValue)
    {
        markersSet.add(marker);
        uniqueSet.add(uniqueValue);
        handleUpdate();
    }

    private void handleUpdate()
    {
        if (markersSet.size() != uniqueSet.size())
            return;
        if (markersSet.size() > logsCount) {
            logsCount = this.markersSet.size();
            if (eventsHandler == null)
                return;
            Map<String, Object> clonedFieldsMap = getClonedFieldsMap();
            clonedFieldsMap.put("markers", markersSet.toString()); 
            clonedFieldsMap.put("count", logsCount);
            eventsHandler.handleEvent(clonedFieldsMap);
        }
    }

    private Map<String, Object> getClonedFieldsMap()
    {
        Map<String, Object> clonedFieldsMap = new HashMap<>();
        clonedFieldsMap.putAll(fieldsMap);

        return clonedFieldsMap;
    }

    public Date getStartTime()
    {
        return startTime.getTime();
    }

    public void setDurationInSec(int duration)
    {
        this.duration = duration * 1000;
        endTime.setTime(startTime.getTime());
        endTime.add(Calendar.SECOND, duration);
    }

    public Date getEndTime()
    {
        return endTime.getTime();
    }

    public long getTimestamp()
    {
        return timestamp;
    }

    public int getDuration()
    {
        return duration;
    }

    public String getName()
    {
        return name;
    }

    public void setName(String name)
    {
        this.name = name;
        fieldsMap.put("name", name);
    }

    public EventsHandler getEventsHandler()
    {
        return eventsHandler;
    }

    public void setEventsHandler(EventsHandler eventsHandler)
    {
        this.eventsHandler = eventsHandler;
    }
}
------------------------------------------------------------------------------------------------

-----Original Message-----
From: Elran Dvir
Sent: Sunday, September 29, 2013 11:01 AM
To: Rules Users List
Subject: RE: [rules-users] Implementation of my use case - what am I doing wrong?

Hi,

Thanks for the response.
I am not sure I fully understand your answer.
After all the rewriting of the rules, I decided to try "(this meets $ce || this
during $ce || this metby $ce)" again.
Now the rule is fired, but an exception is thrown (not all times the rule is fired, just
once between the third and fourth event):

Exception in thread "Thread-1" java.lang.RuntimeException:
java.lang.NoSuchMethodException: com.checkpoint.correlation.impl.drools.Log.meets0()
	at org.drools.rule.constraint.ConditionAnalyzer.analyzeNode(ConditionAnalyzer.java:239)
	at
org.drools.rule.constraint.ConditionAnalyzer.analyzeSingleCondition(ConditionAnalyzer.java:133)
	at
org.drools.rule.constraint.ConditionAnalyzer.analyzeCondition(ConditionAnalyzer.java:99)
	at
org.drools.rule.constraint.ConditionAnalyzer.analyzeCombinedCondition(ConditionAnalyzer.java:140)
	at
org.drools.rule.constraint.ConditionAnalyzer.analyzeCondition(ConditionAnalyzer.java:96)
	at
org.drools.rule.constraint.ConditionAnalyzer.analyzeCombinedCondition(ConditionAnalyzer.java:140)
	at
org.drools.rule.constraint.ConditionAnalyzer.analyzeCondition(ConditionAnalyzer.java:96)
	at
org.drools.rule.constraint.ConditionAnalyzer.analyzeCombinedCondition(ConditionAnalyzer.java:141)
	at
org.drools.rule.constraint.ConditionAnalyzer.analyzeCondition(ConditionAnalyzer.java:96)
	at
org.drools.rule.constraint.ConditionAnalyzer.analyzeCombinedCondition(ConditionAnalyzer.java:141)
	at
org.drools.rule.constraint.ConditionAnalyzer.analyzeCondition(ConditionAnalyzer.java:96)
	at
org.drools.rule.constraint.ConditionAnalyzer.analyzeCombinedCondition(ConditionAnalyzer.java:141)
	at
org.drools.rule.constraint.ConditionAnalyzer.analyzeCondition(ConditionAnalyzer.java:96)
	at
org.drools.rule.constraint.ConditionAnalyzer.analyzeCondition(ConditionAnalyzer.java:70)
	at
org.drools.rule.constraint.MvelConditionEvaluator.getAnalyzedCondition(MvelConditionEvaluator.java:83)
	at org.drools.rule.constraint.MvelConstraint.executeJitting(MvelConstraint.java:270)
	at org.drools.rule.constraint.MvelConstraint.access$200(MvelConstraint.java:51)
	at
org.drools.rule.constraint.MvelConstraint$ConditionJitter.run(MvelConstraint.java:250)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.NoSuchMethodException:
com.checkpoint.correlation.impl.drools.Log.meets0()
	at java.lang.Class.getMethod(Class.java:1622)
	at org.drools.rule.constraint.ConditionAnalyzer.analyzeNode(ConditionAnalyzer.java:230)
	... 20 more

The complete latest drl file is followed.
Duration for Log is 0, duration for CorrelatedEvent is 10s.

Did you have a chance to look at my (original) fourth question? This is the issue that
bothers me the most.
If necessary, I'll write everything again.

Thanks a lot. 

This is my drl file:
----------------------------------------------------------------------------------------------------
package test;

import correlation.impl.drools.Log
import correlation.impl.drools.CorrelatedEvent

global correlation.server.EventsHandler externalEventsHandler;

declare Log
      @role( event)
end

declare CorrelatedEvent
        //@propertyReactive
        @role( event)
        @timestamp( getTimestamp())
        @expires( 10s )
        @duration( getDuration() )
end

// this rule will create a "Port Scan" event if none exist for this group-by
values rule "Create Port Scan Event"
dialect "java"    //dialect can be either mvel or java for the RHS (only java
dialect support generic containers)
no-loop                //this means - do not revaluate rules if the events were altered in
memory
when
                $log : Log() 
                accumulate( Log( this after[0s,5s] $log, fieldsMap.get("src") ==
$log.fieldsMap.get("src") , fieldsMap.get("dst") ==
$log.fieldsMap.get("dst"), $port : fieldsMap.get("port"));
                               	$portSet : collectSet($port);
                               	$portSet.size > 2 )
	   accumulate( Log( this after[0s,5s] $log, fieldsMap.get("src") ==
$log.fieldsMap.get("src") , fieldsMap.get("dst") ==
$log.fieldsMap.get("dst") , $portSet.contains(fieldsMap.get("port")) ,
$marker : fieldsMap.get("marker"));
                              	 $markerSet : collectSet($marker))
                not CorrelatedEvent(getName() == "portScan" ,
fieldsMap.get("src") == $log.fieldsMap.get("src") ,
fieldsMap.get("dst") == $log.fieldsMap.get("dst"))  then
  CorrelatedEvent $ce = new CorrelatedEvent();
  $ce.setName("portScan");
  $ce.setEventsHandler(externalEventsHandler);
  $ce.setDurationInSec(10);
  $ce.fieldsMap.put("src", $log.fieldsMap.get("src"));
  $ce.fieldsMap.put("dst", $log.fieldsMap.get("dst"));
  $ce.endUpdate($markerSet, $portSet);

  insert($ce);
end

rule "Create Port Scan Event - update"
dialect "java"    //dialect can be either mvel or java for the RHS (only java
dialect support generic containers)
no-loop
when
                $ce: CorrelatedEvent(getName() == "portScan", $portSet:
getUniqueSet())
                $log: Log(fieldsMap.get("src") ==
$ce.fieldsMap.get("src") , fieldsMap.get("dst") ==
$ce.fieldsMap.get("dst") , !$portSet.contains(fieldsMap.get("port")),
(this meets $ce || this during $ce || this metby $ce))  then
  modify( $ce ) {endUpdate($log.fieldsMap.get("marker").toString(),
$log.fieldsMap.get("port"))} end
------------------------------------------------------------------------------------------------------------------------------------

-----Original Message-----
From: rules-users-bounces(a)lists.jboss.org [mailto:rules-users-bounces@lists.jboss.org] On
Behalf Of Wolfgang Laun
Sent: Sunday, September 22, 2013 7:06 PM
To: Rules Users List
Subject: Re: [rules-users] Implementation of my use case - what am I doing wrong?

On 22/09/2013, Elran Dvir <elrand(a)checkpoint.com&gt; wrote:
...
 Thanks for the response.

>Computationally,  during the evaluation of the temporal operators.
>Why would you want to do this? (My advice to replace the temporal 
>operators  by the equivalent expressions was meant as a debugging aid, 
>to show you  where the problem with this constraint is.)
 So if changing the condition to use startTimeStamp and endTimestamp 
 caused a compilation error, How can I debug my problem? What is the 
 equivalent of "(this meets $ce || this during $ce || this metby $ce)"
 without my own startTimeStamp and endTimestamp fields?

So I'm quoting my reply from an earlier mail <quote>
...
 Elran Dvir <elrand(a)checkpoint.com&gt; wrote:

 2)       When I tested it with matching data, rule "Create Port Scan Event -
 update" was never fired. When I replaced "(this meets $ce || this 
 during $ce
 || this metby $ce)" with "this after $ce.getStartTime() , this before
 $ce.getEndTime()" everything worked fine.
 Why? 
Just take the constraints and replace the temporal operator by its definition in the
"Fusion" manual and use a little elementary math:

    A meets  || A during B || A metby B
becomes
   abs( B.startTimestamp - A.endTimestamp ) == 0 ||
   B.startTimestamp < A.startTimestamp && A.endTimestamp < B.endTimestamp ||
abs( A.startTimestamp - B.endTimestamp ) == 0 becomes
  ...
</quote>

Did you overlook the "use a little elementary math"? YOu'll have to provide
the values for start and, given the duration (zero,
presumably) the end time. If B has a duration of 0, there's no way the second term can
ever be true. And if A has a duration of 0 too, A and B must have the same timestamp in
order for the other terms to be true.

Note that "debugging" was meant to be done offline ;-)

...
   > - Why do you use this complex declare?
    >   @timestamp( timestamp ) @duration( duration )
   >   is sufficient.

 I want to protect the fields from being set outside their designated 
 functions .I simplified it a bit in my correction. 
I don't know of any danger of your field being set outside their designated functions
by simply writing the field name between the parentheses.

-W
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users

Email secured by Check Point

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Re: [rules-users] Implementation of my use case - what am I doing wrong?