[rules-dev] plugging in a new resource type for the Knowledge Builder stuff

Wed Jan 13 16:12:34 EST 2010

I have implemented a UserResource type to enable custom DRL rule generation
using Drools.

Implementation details:
   drools-api: org.drools.builder.ResoureType - Addition of static User
   ResourceType (this adds it to the registry for use in PackageBuilder).
   drools-compiler: org.drools.compiler.PackageBuilder - Addition of User
   ResourceType handling in addKnowledgeResource method
   drools-api: org.drools.UserResource added
   drools-api: org.drools.UserResourceProvider added
   drools-api: org.drools.UserResourceFactory added
   drools-core: org.drools.UserResourceImpl added (Default Implementation -
   Probably not needed)
   drools-core: org.drools.UserResourceProviderImpl added (Default
   Implementation - Probably not needed)

The UserResourceFactoryProvider extends the ProviderLocator as per the
recommendations for the KnowledgeBase in the API Documentation. This class
has a newUserResource method that uses a private getUserResourceProvider to
return a UserResource. The getUserResourceProvider makes use of the
ProviderLocator newProvider method to locate the resource provider. This
enables me to specify the org.drools.UserResourceProvider in the
drools.services.conf properties file.

The UserResourceProvider as per the KnowledgeBaseProvider model returns a
new implementation via its newUserResource method. This method is
implemented in concrete implementations to provide the custom resource
implementation. The UserResource interface currently only has a single
method loadFromInputStream which returns an ArrayList<BufferedReader> - the
custom generated Drl file readers for use in the new User Resource Package
Builder code.

I have implemented a CustomUserResourceProviderImpl which implements the
UserResourceProvider interface in my test project. Which returns an
instance of the test projects UserResourceImpl implementation class. The
custom UserResourceImpl returns my list of Drl files and is called  in the
PackageBuilder code.

The PackageBuilder code:

} else if ( ResourceType.USER.equals( type ) ) {
    ((InternalResource) resource).setResourceType( type );
    UserResource userResource = UserResourceFactory.newUserResource();
    ArrayList<BufferedReader> drlFiles =
      userResource.loadFromInputStream( resource.getInputStream());
    for(BufferedReader drlFileReader : drlFiles) {
      addPackageFromDrl(drlFileReader);
    }
}...

This enables a single or multiple custom generated Drl files to be added to
the KnowledgeBase. There is also a requirement for multiple directories in
some of our projects, which this solution solves. The whole process of
custom generation is kicked off via the change set addition of the user
resource:

<resource type="USER" source=
"file:src/main/resources/spreadsheet/SampleDecisionTreeOnePage.xls"/>

This resource must be of type USER, but can have any file as the source.
This will then use the CustomUserResourceProviderImpl to provide the custom
Drl rule generation class (which returns the Drl files for use in
PackageBuilder).

I do have one issue at the moment in that the drools.services.conf file
only seems to be picked up when in the project root directory (I am running
the test project using mvn jetty:run) and would like this to be in the
resources directory - is this something related to the ProviderLocator (as
I had a scan throught the source code for that class and the
ChainedProperties class).

Does this seem reasonable as a solution and fit in with the Drools team
vision? What is the next step in the process?

Regards,

Peter Sellars
Senior Developer
Solnet Solutions Limited
Level 7, Brookfields House
19 Victoria Street, Auckland 1010, New Zealand
PO Box 6619, Auckland 1141, New Zealand
DDI: +64 9 977 5808  Mobile: +64 21 809 009
Main: +64 9 977 5800 Fax: +64 9 977 5801

www.solnetsolutions.co.nz
A Solnet Group Company

  From:       Michael Neale <michael.neale at gmail.com>

  To:         Peter Sellars <Peter.Sellars at solnetsolutions.co.nz>

  Cc:         Damon Horrell <Damon.Horrell at solnetsolutions.co.nz>

  Date:       07/01/2010 05:57 p.m.

  Subject:    Re: Tohu Dynamic Rule Implementation

yes, a TOHU type would be fine, as long as the code was refactored
such that you could register new types from outside the core.

On Thu, Jan 7, 2010 at 2:27 PM, Peter Sellars
<Peter.Sellars at solnetsolutions.co.nz> wrote:
> I had started looking at this Michael, so thanks for the information.
>
> I had begun with a "TOHU" ResourceType which I will change to be generic
> USER_TYPE as has been suggested. Had been looking at the areas you
> suggested, so will crack on with an implementation. Had the
PackageBuilder
> implementation added and am at the stage where I was deciding how to get
> the rules from the Spreadsheet.
>
> Will now look at how this can be done for a generic USER_TYPE.
>
> Regards,
>
> Peter Sellars
> Senior Developer
> Solnet Solutions Limited
> Level 7, Brookfields House
> 19 Victoria Street, Auckland 1010, New Zealand
> PO Box 6619, Auckland 1141, New Zealand
> DDI: +64 9 977 5808  Mobile: +64 21 809 009
> Main: +64 9 977 5800 Fax: +64 9 977 5801
>
> www.solnetsolutions.co.nz
> A Solnet Group Company
>
>
>
>
>  From:       Michael Neale <michael.neale at gmail.com>
>
>  To:         Peter Sellars <Peter.Sellars at solnetsolutions.co.nz>
>
>  Cc:         Damon Horrell <Damon.Horrell at solnetsolutions.co.nz>
>
>  Date:       07/01/2010 04:18 p.m.
>
>  Subject:    Re: Tohu Dynamic Rule Implementation
>
>
>
>
>
>
> Hey guys - it seems the idea of a generic resource type that delegates
> the processing to user provided class is welcome.
>
> So Peter if you wanted to, you could look at a patch to the
> drools-core to support something like this, we can probably make sure
> it gets in:
>
> So my suggestion is to take a look at drools-core source code,
> specifically: KnowledgeAgentImpl - you can see a lot of the work is
> delegated to PackageBuilder. Look in PackageBuilder, method:
> addKnowledgeResource - which takes a ResourceType etc - its really an
> ugly if statement based on the type. You can see down the bottom it
> has a catch-all else, which then tries to find a builder in the
> registry (also, take a look above, at how XLS is handled - its pretty
> simple).
>
> So there are a few options, one is to come up with a generic user type
> - and then somehow register, or somehow change the code to allow a
> ResourceType of arbirary name-string, and then register a builder
> etc...
>
>
> Good luck,
>
> Michael.
>
>
> On Wed, Jan 6, 2010 at 2:04 PM, Michael Neale <michael.neale at gmail.com>
> wrote:
>> well the resource type is (I think) just constants, not an enum, so
>> the idea is that it could be pluggable (which I am trying to find out)
>> so wouldnt' have to be generic - more that you register a resource
>> handler for your type of rule resource, and the KA delegates to it
>> when it sees it - not sure if that is the design aim, or if it is done
>> yet (but certainly sounds a good idea).
>>
>>
>> On Wed, Jan 6, 2010 at 2:02 PM, Peter Sellars
>> <Peter.Sellars at solnetsolutions.co.nz> wrote:
>>> Thanks Michael.
>>>
>>> I will jump on to the list. The ResourceType as you stated was one of
> the
>>> early implementations I had in mind. I was unsure if this would be met
> with
>>> approval as the other resource types appear to be Drools specific.
Maybe
>>> something along the lines of "GENERIC_IMPL" with a way to implement any
>>> kind of rule generation technique would meet with more approval.
>>>
>>> Any thoughts/issues with that Damon?
>>>
>>> Regards,
>>>
>>> Peter Sellars
>>> Senior Developer
>>> Solnet Solutions Limited
>>> Level 7, Brookfields House
>>> 19 Victoria Street, Auckland 1010, New Zealand
>>> PO Box 6619, Auckland 1141, New Zealand
>>> DDI: +64 9 977 5808  Mobile: +64 21 809 009
>>> Main: +64 9 977 5800 Fax: +64 9 977 5801
>>>
>>> www.solnetsolutions.co.nz
>>> A Solnet Group Company
>>>
>>>
>>>
>>>
>>>  From:       Michael Neale <michael.neale at gmail.com>
>>>
>>>  To:         Peter Sellars <Peter.Sellars at solnetsolutions.co.nz>
>>>
>>>  Cc:         Damon Horrell <Damon.Horrell at solnetsolutions.co.nz>
>>>
>>>  Date:       06/01/2010 03:55 p.m.
>>>
>>>  Subject:    Re: Tohu Dynamic Rule Implementation
>>>
>>>
>>>
>>>
>>>
>>>
>>> I have asked on the dev mailing list how we could plug in a new
>>> "ResourceType" - so in theory the knowledge agent, once a resource
>>> type is registered, will know if you say type="TOHU_XLS" or something
>>> what to do (whence it will delegate to the tohu loader - which would
>>> implement some standard interface so the core doesn't need to know
>>> about the tohu specifics).
>>>
>>> if you like, jump on the dev list (its low traffic) and hopefully will
>>> find out there.
>>>
>>>
>>>
>>> On Wed, Jan 6, 2010 at 1:26 PM, Peter Sellars
>>> <Peter.Sellars at solnetsolutions.co.nz> wrote:
>>>> The Decision Table Provider implementation uses the
SpreadsheetCompiler
>>>> compile method to generate rules. What I had aimed to do was to create
> a
>>>> provider implementation that uses the TohuSpreadsheet Loader to
> generate
>>>> rules.
>>>>
>>>> How does the decision table provider work?
>>>> - My understanding is that the provider need only implement the
>>>> loadFromInputStream method of the DecisionTableProvider interface,
> which
>>>> has a DecisionTableConfiguration (contains worksheet name etc). It
>>> returns
>>>> the DRL for use by the engine. - which does raise the question, can it
>>>> return multiple .drl files as are required by Tohu? (I shall have to
> dig
>>> a
>>>> little deeper - I believe it will not, so this may be a dead end!)
>>>>
>>>> Does it provide a generic way of generating rules from xls?  .. Or is
> it
>>>> more specific/constrained than that?
>>>> - The default implementation compiles xls or csv compilation via a
> switch
>>>> statement based on the configuraiton input type. This compilation step
>>> was
>>>> where the TohuSpreadsheet loading and rule generation could be
>>> implemented.
>>>>
>>>> Does the process involve generating .drl files in some temporary
> location
>>>> or is that step bypassed and it just goes straight to compiled rules?
>>>> - From reading the source code the .drl file is bundled into a rule
>>> package
>>>> in the PackageBuilder once the xls has been compiled. (Again it
appears
>>> to
>>>> be based on a single .drl file implementation).
>>>>
>>>> Having looked at this level it I am having second thoughts about this
>>> being
>>>> a feasible way to do what I need. As a Tohu Spreadsheet generates a
>>> number
>>>> of .drl files (based on the number of pages etc), whilst the
>>>> DecisionTableProvider is built based on a single .drl file generated
> for
>>>> each xls file. This code is in the PackageBuilder class method
>>>> addKnowledgeResource.
>>>>
>>>> DecisionTableConfiguration dtableConfiguration =
>>>> (DecisionTableConfiguration) configuration;
>>>> String string = DecisionTableFactory.loadFromInputStream
>>>> ( resource.getInputStream(),
>>>>
>>>> dtableConfiguration );
>>>> addPackageFromDrl( new StringReader( string ) );
>>>>
>>>> I will look at creating a reference implementation for review over the
>>> next
>>>> few days - and would appreciate the continued feedback/input on
>>>> ideas/suggestions.
>>>>
>>>> Regards,
>>>>
>>>> Peter Sellars
>>>> Senior Developer
>>>> Solnet Solutions Limited
>>>> Level 7, Brookfields House
>>>> 19 Victoria Street, Auckland 1010, New Zealand
>>>> PO Box 6619, Auckland 1141, New Zealand
>>>> DDI: +64 9 977 5808  Mobile: +64 21 809 009
>>>> Main: +64 9 977 5800 Fax: +64 9 977 5801
>>>>
>>>> www.solnetsolutions.co.nz
>>>> A Solnet Group Company
>>>>
>>>>
>>>>
>>>>
>>>>  From:       Damon Horrell/Employees/SolNet Solutions
>>>>
>>>>  To:         Peter Sellars/Employees/SolNet Solutions at SSL
>>>>
>>>>  Cc:         Michael Neale <michael.neale at gmail.com>
>>>>
>>>>  Date:       06/01/2010 01:49 p.m.
>>>>
>>>>  Subject:    Re: Tohu Dynamic Rule Implementation
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Yes you are correct.  Drools Execution Server will become a component
> of
>>>> Drools.  It's only checked in under Tohu as a temporary measure so we
>>> have
>>>> somewhere convenient to share the code.  Michael should be able to
give
>>> an
>>>> ETA on when it will be moved into Drools proper.  Michael was working
> on
>>>> this prior to Tohu starting but I don't know if anyone is actively
> using
>>> it
>>>> outside of Tohu.
>>>>
>>>> The changes you propose sound reasonable to me but I'm not really
>>> familiar
>>>> with this code so it will need Michael's input.
>>>>
>>>> How does the decision table provider work?  Does it provide a generic
> way
>>>> of generating rules from xls?  Or is it more specific/constrained than
>>>> that?  Does the process involve generating .drl files in some
temporary
>>>> location or is that step bypassed and it just goes straight to
compiled
>>>> rules?  I'm a bit out of the loop on this one.  I guess it doesn't
> matter
>>>> though as long as it works end-to-end.
>>>>
>>>> Damon
>>>>
>>>> -----Peter Sellars/Employees/SolNet Solutions wrote: -----
>>>>
>>>> To: Damon Horrell/Employees/SolNet Solutions at SSL
>>>> From: Peter Sellars/Employees/SolNet Solutions
>>>> Date: 01/06/2010 01:35PM
>>>> cc: Michael Neale <michael.neale at gmail.com>
>>>> Subject: Re: Tohu Dynamic Rule Implementation
>>>>
>>>> Thanks Damon.
>>>>
>>>> Can you clarify the role of the drools execution server that we have
as
>>>> part of the Tohu test project. I notice that it is something that
> should
>>>> ultimately be incorporated into the Drools code base, but is currently
>>> only
>>>> used by Tohu (or is that assumption wrong?).
>>>>
>>>> At the moment I am looking at a way to implement a
>>>> TohuDecisionTableProviderImpl based on the code in blue below. I will
> set
>>>> the table provider to use this implementation which will compile a
>>>> TohuSpreadsheet. I believe this should enable the TohuSpreadsheet to
> work
>>>> with the current Drools resource change framework to update the rules
>>> when
>>>> the spreadsheet changes - as I am aiming to do.
>>>>
>>>> class AgentListener extends ServletContextListener {
>>>>  def  contextInitialized( ev : ServletContextEvent) = {
>>>>    ev.getServletContext.log("Starting knowledge agent service...")
>>>>    ResourceFactory.getResourceChangeNotifierService.start
>>>>    ResourceFactory.getResourceChangeScannerService.start
>>>>    ev.getServletContext.log("Checking Decision Table Provider...")
>>>>    DecisionTableFactory.setDecisionTableProvider(new
>>>> DecisionTableProviderImpl())
>>>>    ev.getServletContext.log("Decision Table Provider is " +
>>>> DecisionTableFactory.getDecisionTableProvider())
>>>>    classOf[KnowledgeAgent].getResourceAsStream("/scanner.properties")
>>>> match {
>>>>        case null => ev.getServletContext.log("Using default scanner
>>>> settings...")
>>>>        case ins: InputStream => {
>>>>          val  props = new Properties
>>>>          props.load(ins)
>>>>          val   conf  =
>>>>
>>>
>
ResourceFactory.getResourceChangeScannerService.newResourceChangeScannerConfiguration

>
>>>
>>>> (props);
>>>>          ResourceFactory.getResourceChangeScannerService.configure
>>> ( conf )
>>>>          ev.getServletContext.log("Using scanner.properties")
>>>>        }
>>>>      }
>>>>  }
>>>>  def  contextDestroyed( ev : ServletContextEvent) = {
>>>>    ev.getServletContext.log("Stopping knowledge agent service")
>>>>    ResourceFactory.getResourceChangeNotifierService.stop
>>>>    ResourceFactory.getResourceChangeScannerService.stop
>>>>  }
>>>> }
>>>>
>>>> Once I have implemented this class and tested it I would appreciate
> some
>>>> feedback from Michael on how he would propose making this hook
>>> configurable
>>>> rather than 'hard-coded' as in my current investigative
implementation.
> I
>>>> believe this could be the right place for what I require - based on my
>>>> understanding of how to KnowledgeAgent is meant to be used.
>>>>
>>>> Regards,
>>>>
>>>> Peter Sellars
>>>> Senior Developer
>>>> Solnet Solutions Limited
>>>> Level 7, Brookfields House
>>>> 19 Victoria Street, Auckland 1010, New Zealand
>>>> PO Box 6619, Auckland 1141, New Zealand
>>>> DDI: +64 9 977 5808  Mobile: +64 21 809 009
>>>> Main: +64 9 977 5800 Fax: +64 9 977 5801
>>>>
>>>> www.solnetsolutions.co.nz

>>>> A Solnet Group Company
>>>>
>>>>
>>>> Damon Horrell---06/01/2010 11:35:59 a.m.---Hi Peter,  I think Michael
>>> will
>>>> be the best person to talk to about this as it's pushing the limits
>>>>
>>>>
>>>>  From:      Damon Horrell/Employees/SolNet Solutions
>>>>
>>>>
>>>>  To:        Peter Sellars/Employees/SolNet Solutions at SSL
>>>>
>>>>
>>>>  Cc:        Michael Neale <michael.neale at gmail.com>
>>>>
>>>>
>>>>  Date:      06/01/2010 11:35 a.m.
>>>>
>>>>
>>>>  Subject:   Re: Tohu Dynamic Rule Implementation
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Hi Peter,
>>>>
>>>> I think Michael will be the best person to talk to about this as it's
>>>> pushing the limits of what I know about how drools loads the rules.  I
>>>> remember looking at a class ages ago that handled loading of the
> various
>>>> types of files .drl, .pkg, etc, but you're right we can't put
>>> Tohu-specific
>>>> code there.
>>>>
>>>> I think the ideal would be if there is or could be a drools hook so
> that
>>>> custom types can be handled, so then you can hook in a handler
for .xls
>>> or
>>>> whatever you want to call the files and generate the rules from them
on
>>> the
>>>> fly.
>>>>
>>>> Damon
>>>>
>>>> -----Peter Sellars/Employees/SolNet Solutions wrote: -----
>>>>
>>>> To: Damon Horrell/Employees/SolNet Solutions at SSL
>>>> From: Peter Sellars/Employees/SolNet Solutions
>>>> Date: 01/06/2010 09:51AM
>>>> Subject: Tohu Dynamic Rule Implementation
>>>>
>>>> Hi Damon,
>>>>
>>>> How are you? Hope you had a great Christmas and New Year. Long time,
no
>>> see
>>>> - have you got bored of Auckland :)
>>>>
>>>> I am currently investigating a specific area related to Tohu, and was
>>>> wondering if you could help me some information.
>>>>
>>>> The actual problem I am trying to solve is the dynamic loading of Tohu
>>>> spreadsheet rules. Currently a PeriodicRuleLoader is being used to
> reload
>>>> the rules (created by David Plumpton) and is started using a separate
>>> Maven
>>>> exec:java process. I am trying to remove this separate process but
> retain
>>>> the dynamic loading capabilities it provides.
>>>>
>>>> I have been carrying out some investigation and so far have come up
> with
>>> a
>>>> few areas to look at, but none have really been successful. Here is a
>>> list
>>>> of some of the areas I looked at:
>>>>            Adding a "TOHU" type to the resource types - allowing use
in
>>>>            the change set (discounted as meant adding Tohu specific
> code
>>>>            to core Drools code)
>>>>            Extending the KnowledgeAgent (discounted as I didn't see
how
>>> it
>>>>            could help me do what I needed)
>>>>            Using the change set and providing a DecisionTableProvider
>>>>            using a TohuDescisionTableProviderImpl via the
>>>>            DecisionTableFactory setDecisionTableProvider method
(unable
>>> to
>>>>            find a hook to do so? - any clues on where I may be able to
> do
>>>>            this?)
>>>>
>>>> From what I understand due to the specific nature of the
> TohuSpreadsheet
>>>> compilation and rules creation we can't use the Drools dynamic loading
>>>> framework. Do you know if it is possible/feasible for Tohu projects to
>>>> create and manage dynamic rule loading by building and maintaining the
>>>> rules via code instead of configuration as it does now?
>>>>
>>>> I have implemented some logging in the Tohu project
> ExecutionServerHelper
>>>> class that I never see. Is this class now redundant? I had considered
>>> using
>>>> this as the hook for the Tohu specific DecisionTableProvider.
>>>>
>>>> Any information, insight, wisdom, flashes of brilliance would be much
>>>> appreciated.
>>>>
>>>> Regards,
>>>>
>>>> Peter Sellars
>>>> Senior Developer
>>>> Solnet Solutions Limited
>>>> Level 7, Brookfields House
>>>> 19 Victoria Street, Auckland 1010, New Zealand
>>>> PO Box 6619, Auckland 1141, New Zealand
>>>> DDI: +64 9 977 5808  Mobile: +64 21 809 009
>>>> Main: +64 9 977 5800 Fax: +64 9 977 5801
>>>>
>>>> www.solnetsolutions.co.nz
>>>> A Solnet Group Company
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Attention:
>>>> This email may contain information intended for the sole use of
>>>> the original recipient. Please respect this when sharing or
>>>> disclosing this email's contents with any third party. If you
>>>> believe you have received this email in error, please delete it
>>>> and notify the sender or postmaster at solnetsolutions.co.nz as
>>>> soon as possible. The content of this email does not necessarily
>>>> reflect the views of Solnet Solutions Ltd.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Michael D Neale
>>> home: www.michaelneale.net
>>> blog: michaelneale.blogspot.com
>>>
>>>
>>>
>>>
>>>
>>> Attention:
>>> This email may contain information intended for the sole use of
>>> the original recipient. Please respect this when sharing or
>>> disclosing this email's contents with any third party. If you
>>> believe you have received this email in error, please delete it
>>> and notify the sender or postmaster at solnetsolutions.co.nz as
>>> soon as possible. The content of this email does not necessarily
>>> reflect the views of Solnet Solutions Ltd.
>>>
>>>
>>
>>
>>
>> --
>> Michael D Neale
>> home: www.michaelneale.net
>> blog: michaelneale.blogspot.com
>>
>
>
>
> --
> Michael D Neale
> home: www.michaelneale.net
> blog: michaelneale.blogspot.com
>
>
>
>
>
> Attention:
> This email may contain information intended for the sole use of
> the original recipient. Please respect this when sharing or
> disclosing this email's contents with any third party. If you
> believe you have received this email in error, please delete it
> and notify the sender or postmaster at solnetsolutions.co.nz as
> soon as possible. The content of this email does not necessarily
> reflect the views of Solnet Solutions Ltd.
>
>

--
Michael D Neale
home: www.michaelneale.net
blog: michaelneale.blogspot.com

Attention:
This email may contain information intended for the sole use of
the original recipient. Please respect this when sharing or
disclosing this email's contents with any third party. If you
believe you have received this email in error, please delete it
and notify the sender or postmaster at solnetsolutions.co.nz as
soon as possible. The content of this email does not necessarily
reflect the views of Solnet Solutions Ltd.