<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Greetings:<div><br></div><div>Well, I really wish more people would read what Steve and I have been blogging about benchmarks for the past few years. While one proposes improving Miss Manners the other proposes dropping it altogether. My personal feelings "used" to be that we could improve Miss Manners so that it could be used. Now, I feel that is any "improvement" will come through massive amounts of data because the rules are WAY too simplistic - and then all you will be testing is the performance of the system itself and NOT the rulebase.</div><div><br></div><div>Also, if you trace the firing of the rules, you will find that once started, for all practical purposes, only one rule keeps firing over and over. Miss Manners was designed for one purpose; to stress-test the Agenda Table by recursively putting all of the guests on the table over and over and over again. Greg Barton - as stated earlier somewhere - repeated this test using straight-up Java and did it just as quickly.</div><div><br></div><div>Regardless, <span class="Apple-style-span" style="color: rgb(51, 51, 51); font-family: Georgia; font-size: 16px; line-height: 24px; "> a<font class="Apple-style-span" face="'Times New Roman'" size="5"><span class="Apple-style-span" style="font-size: 18px;"> rulebase benchmark should be composed of several tests:<br><br>Forward Chaining<br>Backward Chaining<br>Non-Monotonicity (The hallmark of a rulebase)<br>Complex Rules<br>Rules with a high level of Specificity<br>Lots of (maybe 100 or more) "simple" rules that chain between themselves<br>Stress the conflict resolution strategy<br>Stress pattern matching<br><br>Just having an overwhelming amount of data is not sufficient for a rulebase benchmark - that would be more in line with a test of the database efficiency and/or the available memory. Further, it has been "proven" over time that compiling rules into Java code or into C++ code (something that vendors call "sequential rules") is much faster than using the inference engine. True, and it should be. After all, most inference engines are based in Java or C++ code and the rules are merely an extension, another layer of abstraction, if you will. But sequential rules do not have the flexibility of the engine and, in most cases, have to be "manually" arranged so that they fire in the correct order. An inference engine, being non-monotonic, does not have that restriction. Simply put, most rulebased systems cannot pass muster on the simple WaltzDB-16 benchmark. We now have a WaltzDB-200 test should they want to try something more massive. </span></font></span></div><div><font class="Apple-style-span" color="#333333"><span class="Apple-style-span" style="line-height: 24px;"><span class="Apple-style-span" style="font-family: Georgia; font-size: 16px; "><font class="Apple-style-span" face="'Times New Roman'" size="5"><span class="Apple-style-span" style="font-size: 18px;"><br>New Benchmarks: Perhaps we should try some of the NP-hard problems - that would eliminate most of the "also ran" tools. Also, perhaps we should be checking on the "flexibility" of a rulebase by processing on multiple platforms (not just Windows) as well as checking performance and scalability on multiple processors; perhaps 4, 8 or 16 (or more) CPU machines. An 8/16 CPU Mac is now available at a reasonable price as is the i7 Intel (basically 4/8 cores) CPU. But these are 64-bit CPUs and some rule engines are not supported for 64-bit platforms. Sad, but true. Some won't even run on Unix but only on LInux. Again, sad, but true.<br><br>So, any ideas? I'm thinking that someone, somewhere has a <b><span class="Apple-style-span" style="text-decoration: underline;">better</span></b> suggestion than a massive decision table, 64-Queens, Sudoku or a revision of the Minnnesota database benchmark. Hopefully... For now, I strongly suggest that we resolve to use the WaltzDB-200 benchmark (which should satisfy all parties for this year) and develop something much better for 2010.</span></font></span></span></font></div><div><font class="Apple-style-span" color="#333333" face="Georgia" size="4"><span class="Apple-style-span" style="line-height: 24px; font-size: 16px;"><br></span></font></div><div><div apple-content-edited="true"> <span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 18px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 18px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 18px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 18px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>jco</div><div>"NOW do you believe?" </div><div>(Morpheus to Trinity in "The Matrix")</div><div>Buy USA FIRST ! </div><div><br></div><div><div><font class="Apple-style-span" face="'Bodoni SvtyTwo SC ITC TT'"><font class="Apple-style-span" color="#FF0000"><a href="http://www.kbsc.com">http://www.kbsc.com</a> [Home base for AI connections]</font></font></div><div><span class="Apple-style-span" style="color: rgb(255, 0, 0); font-family: 'Bodoni SvtyTwo SC ITC TT'; "><a href="http://www.OctoberRulesFest.org">http://www.OctoberRulesFest.org</a> [Home for AI conferences]</span></div><div><font class="Apple-style-span" face="Garamond"><font class="Apple-style-span" color="#0000FF"><a href="http://JavaRules.blogspot.com">http://JavaRules.blogspot.com</a> [Java-Oriented Rulebased Systems]</font></font></div><div><font class="Apple-style-span" face="Garamond"><font class="Apple-style-span" color="#0000FF"><a href="http://ORF2009.blogspot.com">http://ORF2009.blogspot.com</a> [October Rules Fest]</font></font></div><div><font class="Apple-style-span" face="Garamond"><font class="Apple-style-span" color="#0000FF"><a href="http://exscg.blogspot.com/">http://exscg.blogspot.com/</a> [Expert Systems Consulting Group]</font></font></div><div><font class="Apple-style-span" color="#0000FF" face="Garamond"><br></font></div></div></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"> </div><br><div><div>On Mar 28, 2009, at 1:21 AM, Wolfgang Laun wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">The organizers of RuleML 2009 have announced that one of their<br>topics for the Challenge <a href="http://www.defeasible.org/ruleml2009/challenge">http://www.defeasible.org/ruleml2009/challenge</a><br>of this year's event is going to be "benchmark for evaluation of rule engines". <br> <br>Folks interested in benchmarks are cordially invited to provide <br>requirements, outlines, ideas, etc., for benchmarks to be sent in for<br>the Challenge, and, of course, to submit actual benchmarks to the<br>RuleML Symposium.<br> <br>Spearhead RBS implementations such as Drools provide an<br>excellent arena for real-world applications of rules. Personally, I think<br>that benchmarks should not only address purely FOL-lish<br>pattern combinations but also assess how well the interaction with the<br> embedding environment (such as predicate evaluation, <br>the execution of RHS consequences with agenda updates, etc.,)<br>is handled. <br><br>Regards<br>Wolfgang Laun<br> RuleML 2009 Program Committee<br><br><br><div class="gmail_quote"> On Fri, Mar 27, 2009 at 11:09 PM, Steve Núñez <span dir="ltr"><<a href="mailto:brms@illation.com.au">brms@illation.com.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> Mark,<br> <br> Agreed that Manners needs improvement. In it's current form, it's nearly<br> useless as a comparative benchmark. You might want to check with Charles<br> Young, if he's not on the list, who did a very through analysis of Manners a<br> while back and may have some ideas.<br> <br> Whilst on the topic, I am interested in any other benchmarking ideas that<br> folks may have. We're in the process of putting together (hopefully)<br> comprehensive set of benchmarks for performance testing.<br> <br> Cheers,<br> - Steve<br> <div><div></div><div class="h5"><br> On 28/03/09 5:06 AM, "Mark Proctor" <<a href="mailto:mproctor@codehaus.org">mproctor@codehaus.org</a>> wrote:<br> <br> > I was wondering if anyone fancied having a go at improving Miss Manners<br> > to make it harder and less easy to cheat. The problem with manners at<br> > the moment is that it computes a large cross product, of which only one<br> > rule fires and the other activations are cancelled. What many engines do<br> > now is abuse the test by not calculating the full cross product and thus<br> > not doing all the work.<br> ><br> > Mannsers is explained here:<br> > <a href="https://hudson.jboss.org/hudson/job/drools/lastSuccessfulBuild/artifact/trunk/" target="_blank">https://hudson.jboss.org/hudson/job/drools/lastSuccessfulBuild/artifact/trunk/</a><br> > target/docs/drools-expert/html/ch09.html#d0e7455<br> ><br> > So I was thinking that first the amounts of data needs to be increased<br> > from say 128 guests to 512 guests. Then the problem needs to be made<br> > harder, and the full conflict set needs to be forced to be evalated. So<br> > maybe the first assign_seating rule is as normal where it just finds M/F<br> > pairs with same hobbies, but additionally we should have a scoring<br> > process so that those matched in the first phase then each must have<br> > some compatability score calculated against them and then the one with<br> > the best score is picked. Maybe people have other ways to improve the<br> > complexity of the test, both in adding more rules and more complex rules<br> > and more data.<br> ><br> > Mark<br> ><br> > _______________________________________________<br> > rules-dev mailing list<br> > <a href="mailto:rules-dev@lists.jboss.org">rules-dev@lists.jboss.org</a><br> > <a href="https://lists.jboss.org/mailman/listinfo/rules-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/rules-dev</a><br> <br> </div></div>--<br> <br> Level 40<br> 140 William Street<br> Melbourne, VIC 3000<br> Australia<br> <br> Phone: +61 3 9607 8287<br> Mobile: +61 4 0096 4240<br> Fax: +61 3 9607 8282<br> <a href="http://illation.com.au" target="_blank">http://illation.com.au</a><br> <div><div></div><div class="h5"><br> <br> _______________________________________________<br> rules-dev mailing list<br> <a href="mailto:rules-dev@lists.jboss.org">rules-dev@lists.jboss.org</a><br> <a href="https://lists.jboss.org/mailman/listinfo/rules-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/rules-dev</a><br> </div></div></blockquote></div><br> _______________________________________________<br>rules-dev mailing list<br><a href="mailto:rules-dev@lists.jboss.org">rules-dev@lists.jboss.org</a><br>https://lists.jboss.org/mailman/listinfo/rules-dev<br></blockquote></div><br></div></body></html>