[rules-users] FW: Implementation of my use case - what am I doing wrong?

I wrote my question in the past, but now I want to ask about another strange behavior I have encountered. I will rewrite all relevant data. I am trying to identify a port scan event: Basic event is connection log. For each combination of source_ip and destination_ip, detect a port scan event, if over 5 seconds there were more than 2 connection logs with different ports . The event will stay open for 10 seconds and an update will be sent for any new port detected. Every update will contain the count of connection logs combining it and their id ("marker"). This is my drl file: ---------------------------------------------------------------------------------------------------- package test; import correlation.impl.drools.Log import correlation.impl.drools.CorrelatedEvent global correlation.server.EventsHandler externalEventsHandler; declare Log @role( event) end declare CorrelatedEvent //@propertyReactive @role( event) @timestamp( getTimestamp()) @expires( 10s ) @duration( getDuration() ) end // this rule will create a "Port Scan" event if none exist for this group-by values rule "Create Port Scan Event" dialect "java" //dialect can be either mvel or java for the RHS (only java dialect support generic containers) no-loop //this means - do not revaluate rules if the events were altered in memory when $log : Log() accumulate( Log( this after[0s,5s] $log, fieldsMap.get("src") == $log.fieldsMap.get("src") , fieldsMap.get("dst") == $log.fieldsMap.get("dst"), $port : fieldsMap.get("port")); $portSet : collectSet($port); $portSet.size > 2 ) accumulate( Log( this after[0s,5s] $log, fieldsMap.get("src") == $log.fieldsMap.get("src") , fieldsMap.get("dst") == $log.fieldsMap.get("dst") , $portSet.contains(fieldsMap.get("port")) , $marker : fieldsMap.get("marker")); $markerSet : collectSet($marker)) not CorrelatedEvent(getName() == "portScan" , fieldsMap.get("src") == $log.fieldsMap.get("src") , fieldsMap.get("dst") == $log.fieldsMap.get("dst")) then CorrelatedEvent $ce = new CorrelatedEvent(); $ce.setName("portScan"); $ce.setEventsHandler(externalEventsHandler); $ce.setDurationInSec(10); $ce.fieldsMap.put("src", $log.fieldsMap.get("src")); $ce.fieldsMap.put("dst", $log.fieldsMap.get("dst")); $ce.endUpdate($markerSet, $portSet); insert($ce); end rule "Create Port Scan Event - update" dialect "java" //dialect can be either mvel or java for the RHS (only java dialect support generic containers) no-loop when $ce: CorrelatedEvent(getName() == "portScan", $portSet: getUniqueSet()) $log: Log(fieldsMap.get("src") == $ce.fieldsMap.get("src") , fieldsMap.get("dst") == $ce.fieldsMap.get("dst") , !$portSet.contains(fieldsMap.get("port")), this after $ce.getStartTime() , this before $ce.getEndTime()) then modify( $ce ) {endUpdate($log.fieldsMap.get("marker").toString(), $log.fieldsMap.get("port"))} end ------------------------------------------------------------------------------------------------------------------------------------ I test it like this: I insert a connection log and fire the rules every second. I have 25 logs with the same "src" and "dst", but each has different (serial) "port" and "marker". So after 12-13 logs, I expect to identify a new event with another consecutive 3 logs. In each rule's RHS, I print the rule fired and the ports accumulated so far. With existing implementation, I see the following output at 14th second: rule fired: Create Port Scan Event - update portSet: [1, 7, 11, 8, 5, 6, 3, 10, 4, 9, 12, 2] rule fired: Create Port Scan Event portSet: [13, 11, 12] As we can see, the first rule processes logs already processed by the second rule (11,12). After examining the first rule, I thought I understood the reason for this behavior. I changed the order of conditions in the LHS of the first rule by moving "not CorrelatedEvent..." to be the second condition. But It still behaves the same. Why is that? How can I prevent the first rule from processing logs already processed by the second rule? Thanks a lot! --------------------------------------------------------------- Log class: ---------------------------------------------------------------- public class Log { public HashMap<String, Object> fieldsMap = new HashMap<>(); } ------------------------------------------------------------------------------------------- CorrelatedEvent class: --------------------------------------------------------------------------------------------- public class CorrelatedEvent { public Map<String, Object> fieldsMap; public int duration; //in ms public long timestamp; private String name; private Set<String> markersSet; private Set uniqueSet; private long logsCount; private Calendar startTime; private Calendar endTime; private EventsHandler eventsHandler; public CorrelatedEvent() { startTime = Calendar.getInstance(); timestamp = startTime.getTime().getTime(); endTime = Calendar.getInstance(); endTime.setTime(startTime.getTime()); duration = 0; fieldsMap = new HashMap<>(); markersSet = Collections.newSetFromMap(new ConcurrentHashMap<String, Boolean>()); uniqueSet = Collections.newSetFromMap(new ConcurrentHashMap<Object, Boolean>()); logsCount = 0; } public Set getUniqueSet() { return uniqueSet; } public void endUpdate(Set<String> markersSet, Set uniqueSet) { this.markersSet.addAll(markersSet); this.uniqueSet.addAll(uniqueSet); handleUpdate(); } public void endUpdate(String marker, Object uniqueValue) { markersSet.add(marker); uniqueSet.add(uniqueValue); handleUpdate(); } private void handleUpdate() { if (markersSet.size() != uniqueSet.size()) return; if (markersSet.size() > logsCount) { logsCount = this.markersSet.size(); if (eventsHandler == null) return; Map<String, Object> clonedFieldsMap = getClonedFieldsMap(); clonedFieldsMap.put("markers", markersSet.toString()); clonedFieldsMap.put("count", logsCount); eventsHandler.handleEvent(clonedFieldsMap); } } private Map<String, Object> getClonedFieldsMap() { Map<String, Object> clonedFieldsMap = new HashMap<>(); clonedFieldsMap.putAll(fieldsMap); return clonedFieldsMap; } public Date getStartTime() { return startTime.getTime(); } public void setDurationInSec(int duration) { this.duration = duration * 1000; endTime.setTime(startTime.getTime()); endTime.add(Calendar.SECOND, duration); } public Date getEndTime() { return endTime.getTime(); } public long getTimestamp() { return timestamp; } public int getDuration() { return duration; } public String getName() { return name; } public void setName(String name) { this.name = name; fieldsMap.put("name", name); } public EventsHandler getEventsHandler() { return eventsHandler; } public void setEventsHandler(EventsHandler eventsHandler) { this.eventsHandler = eventsHandler; } } ------------------------------------------------------------------------------------------------ -----Original Message----- From: Elran Dvir Sent: Sunday, September 29, 2013 11:01 AM To: Rules Users List Subject: RE: [rules-users] Implementation of my use case - what am I doing wrong? Hi, Thanks for the response. I am not sure I fully understand your answer. After all the rewriting of the rules, I decided to try "(this meets $ce || this during $ce || this metby $ce)" again. Now the rule is fired, but an exception is thrown (not all times the rule is fired, just once between the third and fourth event): Exception in thread "Thread-1" java.lang.RuntimeException: java.lang.NoSuchMethodException: com.checkpoint.correlation.impl.drools.Log.meets0() at org.drools.rule.constraint.ConditionAnalyzer.analyzeNode(ConditionAnalyzer.java:239) at org.drools.rule.constraint.ConditionAnalyzer.analyzeSingleCondition(ConditionAnalyzer.java:133) at org.drools.rule.constraint.ConditionAnalyzer.analyzeCondition(ConditionAnalyzer.java:99) at org.drools.rule.constraint.ConditionAnalyzer.analyzeCombinedCondition(ConditionAnalyzer.java:140) at org.drools.rule.constraint.ConditionAnalyzer.analyzeCondition(ConditionAnalyzer.java:96) at org.drools.rule.constraint.ConditionAnalyzer.analyzeCombinedCondition(ConditionAnalyzer.java:140) at org.drools.rule.constraint.ConditionAnalyzer.analyzeCondition(ConditionAnalyzer.java:96) at org.drools.rule.constraint.ConditionAnalyzer.analyzeCombinedCondition(ConditionAnalyzer.java:141) at org.drools.rule.constraint.ConditionAnalyzer.analyzeCondition(ConditionAnalyzer.java:96) at org.drools.rule.constraint.ConditionAnalyzer.analyzeCombinedCondition(ConditionAnalyzer.java:141) at org.drools.rule.constraint.ConditionAnalyzer.analyzeCondition(ConditionAnalyzer.java:96) at org.drools.rule.constraint.ConditionAnalyzer.analyzeCombinedCondition(ConditionAnalyzer.java:141) at org.drools.rule.constraint.ConditionAnalyzer.analyzeCondition(ConditionAnalyzer.java:96) at org.drools.rule.constraint.ConditionAnalyzer.analyzeCondition(ConditionAnalyzer.java:70) at org.drools.rule.constraint.MvelConditionEvaluator.getAnalyzedCondition(MvelConditionEvaluator.java:83) at org.drools.rule.constraint.MvelConstraint.executeJitting(MvelConstraint.java:270) at org.drools.rule.constraint.MvelConstraint.access$200(MvelConstraint.java:51) at org.drools.rule.constraint.MvelConstraint$ConditionJitter.run(MvelConstraint.java:250) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.NoSuchMethodException: com.checkpoint.correlation.impl.drools.Log.meets0() at java.lang.Class.getMethod(Class.java:1622) at org.drools.rule.constraint.ConditionAnalyzer.analyzeNode(ConditionAnalyzer.java:230) ... 20 more The complete latest drl file is followed. Duration for Log is 0, duration for CorrelatedEvent is 10s. Did you have a chance to look at my (original) fourth question? This is the issue that bothers me the most. If necessary, I'll write everything again. Thanks a lot. This is my drl file: ---------------------------------------------------------------------------------------------------- package test; import correlation.impl.drools.Log import correlation.impl.drools.CorrelatedEvent global correlation.server.EventsHandler externalEventsHandler; declare Log @role( event) end declare CorrelatedEvent //@propertyReactive @role( event) @timestamp( getTimestamp()) @expires( 10s ) @duration( getDuration() ) end // this rule will create a "Port Scan" event if none exist for this group-by values rule "Create Port Scan Event" dialect "java" //dialect can be either mvel or java for the RHS (only java dialect support generic containers) no-loop //this means - do not revaluate rules if the events were altered in memory when $log : Log() accumulate( Log( this after[0s,5s] $log, fieldsMap.get("src") == $log.fieldsMap.get("src") , fieldsMap.get("dst") == $log.fieldsMap.get("dst"), $port : fieldsMap.get("port")); $portSet : collectSet($port); $portSet.size > 2 ) accumulate( Log( this after[0s,5s] $log, fieldsMap.get("src") == $log.fieldsMap.get("src") , fieldsMap.get("dst") == $log.fieldsMap.get("dst") , $portSet.contains(fieldsMap.get("port")) , $marker : fieldsMap.get("marker")); $markerSet : collectSet($marker)) not CorrelatedEvent(getName() == "portScan" , fieldsMap.get("src") == $log.fieldsMap.get("src") , fieldsMap.get("dst") == $log.fieldsMap.get("dst")) then CorrelatedEvent $ce = new CorrelatedEvent(); $ce.setName("portScan"); $ce.setEventsHandler(externalEventsHandler); $ce.setDurationInSec(10); $ce.fieldsMap.put("src", $log.fieldsMap.get("src")); $ce.fieldsMap.put("dst", $log.fieldsMap.get("dst")); $ce.endUpdate($markerSet, $portSet); insert($ce); end rule "Create Port Scan Event - update" dialect "java" //dialect can be either mvel or java for the RHS (only java dialect support generic containers) no-loop when $ce: CorrelatedEvent(getName() == "portScan", $portSet: getUniqueSet()) $log: Log(fieldsMap.get("src") == $ce.fieldsMap.get("src") , fieldsMap.get("dst") == $ce.fieldsMap.get("dst") , !$portSet.contains(fieldsMap.get("port")), (this meets $ce || this during $ce || this metby $ce)) then modify( $ce ) {endUpdate($log.fieldsMap.get("marker").toString(), $log.fieldsMap.get("port"))} end ------------------------------------------------------------------------------------------------------------------------------------ -----Original Message----- From: rules-users-bounces(a)lists.jboss.org [mailto:rules-users-bounces@lists.jboss.org] On Behalf Of Wolfgang Laun Sent: Sunday, September 22, 2013 7:06 PM To: Rules Users List Subject: Re: [rules-users] Implementation of my use case - what am I doing wrong? On 22/09/2013, Elran Dvir <elrand(a)checkpoint.com&gt; wrote: So I'm quoting my reply from an earlier mail <quote> Just take the constraints and replace the temporal operator by its definition in the "Fusion" manual and use a little elementary math: A meets || A during B || A metby B becomes abs( B.startTimestamp - A.endTimestamp ) == 0 || B.startTimestamp < A.startTimestamp && A.endTimestamp < B.endTimestamp || abs( A.startTimestamp - B.endTimestamp ) == 0 becomes ... </quote> Did you overlook the "use a little elementary math"? YOu'll have to provide the values for start and, given the duration (zero, presumably) the end time. If B has a duration of 0, there's no way the second term can ever be true. And if A has a duration of 0 too, A and B must have the same timestamp in order for the other terms to be true. Note that "debugging" was meant to be done offline ;-) I don't know of any danger of your field being set outside their designated functions by simply writing the field name between the parentheses. -W _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users Email secured by Check Point