The organizers of RuleML 2009 have announced that one of their
topics for the Challenge http://www.defeasible.org/ruleml2009/challenge
of this year's event is going to be "benchmark for evaluation of rule engines".

Folks interested in benchmarks are cordially invited to provide
requirements, outlines, ideas, etc., for benchmarks to be sent in for
the Challenge, and, of course, to submit actual benchmarks to the
RuleML Symposium.

Spearhead RBS implementations such as Drools provide an
excellent arena for real-world applications of rules. Personally, I think
that benchmarks should not only address purely FOL-lish
pattern combinations but also assess how well the interaction with the
embedding environment (such as predicate evaluation,
the execution of RHS consequences with agenda updates, etc.,)
is handled.

Regards
Wolfgang Laun
RuleML 2009 Program Committee


On Fri, Mar 27, 2009 at 11:09 PM, Steve Núñez <brms@illation.com.au> wrote:
Mark,

Agreed that Manners needs improvement. In it's current form, it's nearly
useless as a comparative benchmark. You might want to check with Charles
Young, if he's not on the list, who did a very through analysis of Manners a
while back and may have some ideas.

Whilst on the topic, I am interested in any other benchmarking ideas that
folks may have. We're in the process of putting together (hopefully)
comprehensive set of benchmarks for performance testing.

Cheers,
   - Steve

On 28/03/09 5:06 AM, "Mark Proctor" <mproctor@codehaus.org> wrote:

> I was wondering if anyone fancied having a go at improving Miss Manners
> to make it harder and less easy to cheat. The problem with manners at
> the moment is that it computes a large cross product, of which only one
> rule fires and the other activations are cancelled. What many engines do
> now is abuse the test by not calculating the full cross product and thus
> not doing all the work.
>
> Mannsers is explained here:
> https://hudson.jboss.org/hudson/job/drools/lastSuccessfulBuild/artifact/trunk/
> target/docs/drools-expert/html/ch09.html#d0e7455
>
> So I was thinking that first the amounts of data needs to be increased
> from say 128 guests to  512 guests. Then the problem needs to be made
> harder, and the full conflict set needs to be forced to be evalated. So
> maybe the first assign_seating rule is as normal where it just finds M/F
> pairs with same hobbies, but additionally we should have a scoring
> process so that those matched in the first phase then each must have
> some compatability score calculated against them and then the one with
> the best score is picked. Maybe people have other ways to improve the
> complexity of the test, both in adding more rules and more complex rules
> and more data.
>
> Mark
>
> _______________________________________________
> rules-dev mailing list
> rules-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-dev

--

Level 40
140 William Street
Melbourne, VIC 3000
Australia

Phone:  +61 3 9607 8287
Mobile: +61 4 0096 4240
Fax:    +61 3 9607 8282
http://illation.com.au


_______________________________________________
rules-dev mailing list
rules-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-dev