The organizers of RuleML 2009 have announced that one of their<br>topics for the Challenge <a href="http://www.defeasible.org/ruleml2009/challenge">http://www.defeasible.org/ruleml2009/challenge</a><br>of this year's event is going to be "benchmark for evaluation of rule engines". <br>
<br>Folks interested in benchmarks are cordially invited to provide <br>requirements, outlines, ideas, etc., for benchmarks to be sent in for<br>the Challenge, and, of course, to submit actual benchmarks to the<br>RuleML Symposium.<br>
<br>Spearhead RBS implementations such as Drools provide an<br>excellent arena for real-world applications of rules. Personally, I think<br>that benchmarks should not only address purely FOL-lish<br>pattern combinations but also assess how well the interaction with the<br>
embedding environment (such as predicate evaluation, <br>the execution of RHS consequences with agenda updates, etc.,)<br>is handled. <br><br>Regards<br>Wolfgang Laun<br> RuleML 2009 Program Committee<br><br><br><div class="gmail_quote">
On Fri, Mar 27, 2009 at 11:09 PM, Steve Núñez <span dir="ltr"><<a href="mailto:brms@illation.com.au">brms@illation.com.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Mark,<br>
<br>
Agreed that Manners needs improvement. In it's current form, it's nearly<br>
useless as a comparative benchmark. You might want to check with Charles<br>
Young, if he's not on the list, who did a very through analysis of Manners a<br>
while back and may have some ideas.<br>
<br>
Whilst on the topic, I am interested in any other benchmarking ideas that<br>
folks may have. We're in the process of putting together (hopefully)<br>
comprehensive set of benchmarks for performance testing.<br>
<br>
Cheers,<br>
- Steve<br>
<div><div></div><div class="h5"><br>
On 28/03/09 5:06 AM, "Mark Proctor" <<a href="mailto:mproctor@codehaus.org">mproctor@codehaus.org</a>> wrote:<br>
<br>
> I was wondering if anyone fancied having a go at improving Miss Manners<br>
> to make it harder and less easy to cheat. The problem with manners at<br>
> the moment is that it computes a large cross product, of which only one<br>
> rule fires and the other activations are cancelled. What many engines do<br>
> now is abuse the test by not calculating the full cross product and thus<br>
> not doing all the work.<br>
><br>
> Mannsers is explained here:<br>
> <a href="https://hudson.jboss.org/hudson/job/drools/lastSuccessfulBuild/artifact/trunk/" target="_blank">https://hudson.jboss.org/hudson/job/drools/lastSuccessfulBuild/artifact/trunk/</a><br>
> target/docs/drools-expert/html/ch09.html#d0e7455<br>
><br>
> So I was thinking that first the amounts of data needs to be increased<br>
> from say 128 guests to 512 guests. Then the problem needs to be made<br>
> harder, and the full conflict set needs to be forced to be evalated. So<br>
> maybe the first assign_seating rule is as normal where it just finds M/F<br>
> pairs with same hobbies, but additionally we should have a scoring<br>
> process so that those matched in the first phase then each must have<br>
> some compatability score calculated against them and then the one with<br>
> the best score is picked. Maybe people have other ways to improve the<br>
> complexity of the test, both in adding more rules and more complex rules<br>
> and more data.<br>
><br>
> Mark<br>
><br>
> _______________________________________________<br>
> rules-dev mailing list<br>
> <a href="mailto:rules-dev@lists.jboss.org">rules-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/rules-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/rules-dev</a><br>
<br>
</div></div>--<br>
<br>
Level 40<br>
140 William Street<br>
Melbourne, VIC 3000<br>
Australia<br>
<br>
Phone: +61 3 9607 8287<br>
Mobile: +61 4 0096 4240<br>
Fax: +61 3 9607 8282<br>
<a href="http://illation.com.au" target="_blank">http://illation.com.au</a><br>
<div><div></div><div class="h5"><br>
<br>
_______________________________________________<br>
rules-dev mailing list<br>
<a href="mailto:rules-dev@lists.jboss.org">rules-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/rules-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/rules-dev</a><br>
</div></div></blockquote></div><br>