[jboss-as7-dev] Modularity is the spawn of Lucifer and a stinking donkey

[Steve Ebersole]

Hey all, just to tie this up with a bow I wanted to send my take away 
from the discussions we had on IRC and make sure everyone had the same 
take away.  Jason and David, wanted to thank you guys again for your 
time and input.  It really helped out.  I also went ahead and 
copied/pasted the transcript so we had record of discussion.

The basic idea is to control this via a specific deployment descriptor 
(jboss-hibernate.xml or somesuch).  It would, in the general case, 
provide 2 pieces of information:
1) Which Hibernate version are we talking about.  This is a very 
high-level concept of version, mainly 3 or 4.
2) Which "modules" do you want included.  OGM?  Envers?  Etc.  It won't 
name modules per-se in terms of AS modules.  But more what Hibernate 
integrations do you want included.  We will also allow for custom AS 
modules to be named here, since they could also be "Hibernate 
integrations" (provide service loader locatable impls of Integrator).  
Additionally, the inclusion of a module here will export the classes 
from the corresponding AS module into the app classloader.

To account for app deployments that contain Hibernate jars, the 
suggestion was to allow a flag in this deployment descriptor to disable 
the inclusion of AS provided Hibernate modules/jars.  This could also 
be leveraged to, for example, bundle an alternate JPA provider.

There is still some gray area around what to do when the app deployment 
contains Hibernate jars but has no deployment descriptor.  Its kind of 
a 50/50 proposition and seems like there is not a good solution.


On Mon 21 May 2012 02:25:47 PM CDT, Scott Marlow wrote:
> On 05/10/2012 03:04 PM, Jason T. Greene wrote:
>> On 5/10/12 11:21 AM, David M. Lloyd wrote:
>>> On 05/10/2012 10:24 AM, Scott Marlow wrote:
>>>> On 05/09/2012 02:01 PM, David M. Lloyd wrote:
>>>>> OK I admit I LOL'ed.
>>>>>
>>>>> On 05/09/2012 11:50 AM, Emmanuel Bernard wrote:
>>>>>> Now that I have your attention, I'd like to discuss issues we are
>>>>>> experiencing when trying to modularize the Hibernate portfolio and
>>>>>> make it work in AS 7.1.
>>>>>>
>>>>>> ## Disclaimer
>>>>>>
>>>>>> I perfectly understand all the coolness about modularity (speed,
>>>>>> easier dependency isolation etc). I have also carefully read :
>>>>>>
>>>>>> - https://community.jboss.org/wiki/ModuleCompatibleClassloadingGuide
>>>>>> - https://community.jboss.org/wiki/ModularSerialization
>>>>>>
>>>>>> But these tend to avoid the more complex cases of portable libraries
>>>>>> that ought to run even outside AS 7 but have a wide variety of class
>>>>>> and resource loading needs.
>>>>>> I am not a complete modularity bozo but I am definitely not familiar
>>>>>> with JBoss Modules nor similar solution.
>>>>>>
>>>>>> ## Requirements / Landscape
>>>>>>
>>>>>> Hibernate ORM uses the notion of service registry and integrator
>>>>>> object that help during the integration or customization of the
>>>>>> engine behavior by third-party frameworks.
>>>>>> Enlistment of Integrators is done via the service locator pattern (a
>>>>>> service file in META-INF/services/ that is looked up and contain the
>>>>>> implementation class(es) at stake.
>>>>>>
>>>>>> Hibernate Envers is one of those customizer that depends on
>>>>>> Hibernate
>>>>>> ORM. Note that the core of Hibernate ORM does not depend on
>>>>>> Hibernate
>>>>>> Envers. The service locator file is contained in Hibernate Envers
>>>>>> JAR.
>>>>>> Hibernate OGM likewise, heavily customizes ORM and depends on
>>>>>> Hibernate ORM classes - the reverse is not true. The service locator
>>>>>> file is contained in Hibernate OGM JAR.
>>>>>> Hibernate Search optionally depend on Hibernate ORM and JPA. The
>>>>>> core
>>>>>> of Hibernate Search is independent but an Hibernate Search ORM
>>>>>> module
>>>>>> has an integrator implementation. On top of that, Hibernate Search
>>>>>> optionally depend on some JPA classes and behaves differently if
>>>>>> they
>>>>>> are there - we look them up in the classpath by reflection.
>>>>>>
>>>>>> On top of that, these projects do load resources (config files,
>>>>>> classes):
>>>>>>
>>>>>> - from what Jason calls a Deployment classloader (the user
>>>>>> application classes and resources really) - entities, custom
>>>>>> analyzer
>>>>>> implementations, resources files etc. A user could even write a
>>>>>> custom Integrator and use the service locator pattern from his
>>>>>> application.
>>>>>> - from direct dependencies (Lucene is a declared dependency of
>>>>>> Hibernate Search)
>>>>>> - from dependencies of the deployment: for example an app developer
>>>>>> adds the phonetic analyzer as a dependency of his application and
>>>>>> ask
>>>>>> Hibernate Search to use it
>>>>>> - from modules that use these projects. Modeshape and Capedwarf are
>>>>>> being modularized and are making use of Hibernate Search as a
>>>>>> module.
>>>>>> Properly loading the necessary classes located in Modeshape or
>>>>>> Capedwarf's module but from Hibernate Search's engine proves to be
>>>>>> very hard in our current approach.
>>>>>>
>>>>>> All of these projects should be able to run outside JBoss AS 7, so a
>>>>>> modular friendly solution should somehow be abstracted and generic
>>>>>> enough.
>>>>>>
>>>>>> ## What solution?
>>>>>>
>>>>>> More and more projects are being modularized including ones with
>>>>>> complex resource loading dependencies like the ones I have
>>>>>> described.
>>>>>> AFAIK Infinispan is even in a worse situation as clustering and late
>>>>>> class binding is at stake but let's put this one aside.
>>>>>> I'd love to get a reference design outcome from this thread that
>>>>>> could be copied across all these projects and future ones like Bean
>>>>>> Validation.
>>>>>>
>>>>>> Today, we mostly use the good old and simple TCCL model which works
>>>>>> fine if the jars are directly embedded in the app but fail the
>>>>>> minute
>>>>>> we start to move these dependencies into modules. Sanne, Strong,
>>>>>> Scott Marlow and I are using dangerous amount of Advil to try and
>>>>>> make everything work as expected. Some help would be awesome.
>>>>>>
>>>>>> To sum up:
>>>>>>
>>>>>> - can the Hibernate portfolio be supported within JBoss Module and
>>>>>> how?
>>>>>> - what kind of ClassloaderService contract should we use within
>>>>>> these
>>>>>> projects to be modular friendly (JBoss Modules and others)?
>>>>>> - would such contract be generic enough to be extrapolated to
>>>>>> JSRs in
>>>>>> need of modular friendliness?
>>>>>> - how to solve the chicken and egg issue of the bootstrapping: if we
>>>>>> need to pass a ClassloaderService impl?
>>>>> How do we do that best in a modular environment without forcing the
>>>>> application developer to implement such godforsaken
>>>>> ClassloaderService
>>>>> contract or even worse pass directly to us the right classloader for
>>>>> each call.
>>>>>
>>>>> I'll just start at the beginning and you can skip over the
>>>>> background if
>>>>> you like.
>>>>>
>>>>> The key starting concept is that a class' (or package's) identity is
>>>>> not
>>>>> just its name but also its class loader. This is the underlying
>>>>> (existing) truth that modularity brings to the fore. Corollary to
>>>>> this
>>>>> are the fact that a single VM may have more than one class or package
>>>>> with the same name, as well as the fact that not all classes/packages
>>>>> are always directly visible from a given class loader.
>>>>>
>>>>> This problem (as you've seen) manifests itself primarily when you're
>>>>> locating a class or a resource by name. You basically have two
>>>>> options.
>>>>> You can search *relative* to a class loader (most commonly, TCCL,
>>>>> though using a single explicit class loader or the caller's class
>>>>> loader
>>>>> also fall into this category). Or, you can use the *absolute*
>>>>> identity
>>>>> of a class.
>>>>>
>>>>> Using relative resolution is often a perfectly adequate solution when
>>>>> you're loading a single class or resource; in fact for some cases
>>>>> (like
>>>>> ServiceLoader for example) it's a perfect fit in conjunction with
>>>>> TCCL
>>>>> (in its capacity as an identifier for the "current" application). You
>>>>> want the user to be able to specify their implementation of
>>>>> something,
>>>>> and you want it to be reasonably transparent; ServiceLoader+TCCL does
>>>>> this fairly well.
>>>>>
>>>>> ServiceLoader also does well from the perspective of APIs with a
>>>>> static,
>>>>> fixed number of implementations. In this case, it is appropriate
>>>>> for a
>>>>> framework to have ServiceLoader use the class loader of the framework
>>>>> itself. The framework would then be sure to import the
>>>>> implementations
>>>>> in question (including their service descriptors); in our modular
>>>>> environment, which we call a "service import". Note that this often
>>>>> means there is a circular dependency between API and implementation:
>>>>> that's OK!
>>>>
>>>> We currently use this for envers but that doesn't seem as desirable
>>>> for
>>>> other members of the Hibernate portfolio that may be on a separate
>>>> lifecycle. For example, the Hibernate OGM is a persistence provider
>>>> that
>>>> depends on Hibernate ORM. If we have Hibernate ORM depend on OGM, that
>>>> limits the number of OGM versions that can be in use on AS7.
>>>
>>> Cases where implementations are pluggable could be handled using
>>> TCCL or
>>> by specifying a class loader, or by explicit registration at run time.
>>> As I said above, you would only use static dependencies if the
>>> implementations are more or less fixed.
>>>
>>>> Would it be possible, to add a MSC enhancement, that allows an inverse
>>>> dependency service loader dependency to be expressed? Such that it
>>>> would
>>>> be enough to only have OGM depend on ORM (with an inverse service
>>>> dependency specified). I'm thinking that the OGM module would need to
>>>> exchange the service dependency information with the ORM module and
>>>> clear it, when OGM goes away.
>>>
>>> MSC has absolutely nothing to do with any of this. Module loading is
>>> static and is not affected by services in any way.
>>>
>>> In your case you probably want a registration system. But be aware that
>>> if you have more than one OGM implementation, you're not going to be
>>> able to use the ORM API solely to load them anyway (how would you tell
>>> it which one you want?); instead you'd need some mechanism to specify,
>>> so why not use TCCL or explicit class loader specification, as
>>> appropriate?
>>
>> In case it's helpful. The way he handle JAXP is that we have a container
>> default and we also support TCCL.
>>
>> The container default is defined by a special module:
>> https://github.com/jbossas/jboss-as/blob/master/build/src/main/resources/modules/javax/xml/jaxp-provider/main/module.xml
>>
>>
>>
>>
>> The rules are
>>
>> 1. Use service loader on TCCL (users deployment first), if an impl is
>> found use that
>> 2. If not result is find use the default implementation, loaded from the
>> classloader of the "jaxp-provider" default module.
>> 3. If the default jaxp-provider module doesn't contain it, we use the
>> JVM default, whatever it may be.
>>
>> The way we implement this relies on installing a false, delegating JAXP
>> provider. Although in our own code we should define SPIs of some sort.
>>
>> So for example, Hibernate OGM could provide a ProviderLocator interface,
>> which AS would then implement. Our implementation would follow similar
>> logic as above, although maybe you want slightly different rules. OGM
>> would ship with a predefined ProviderLocator which would simply search
>> TCCL.
>>
>
> David/Jason,
>
> It might be helpful if we discussed the Hibernate ORM case on IRC
> (either in #Hibernate-dev or #jboss-as7 (or a private room perhaps).
> Do you guys have any time this week for that?
>
> Scott

--
steve at hibernate.org
http://hibernate.org
-------------- next part --------------
[14:54] <Nihility> sebersole: smarlow im around just lately waste a great deal of my time on conference calls :)
[14:55] <Nihility> sebersole: smarlow i dont see how anything can be done with modules to address the issue, unless you are able to make it psychic
[14:55] <Nihility> :)
[14:56] --> spagop has joined this channel (~spagop at ppp-93-104-15-160.dynamic.mnet-online.de).
[14:56] <sebersole> Nihility: well one thing that keeps rolling around in my head is the notion of exporting for the sake of integration
[14:57] *** rcernich is now known as rcernich_away.
[14:57] <sebersole> since really, all this is spi-level stuff
[14:57] <dmlloyd> sebersole: I'm available now if you want to talk
[14:57] <dmlloyd> smarlow: ^^
[14:57] <smarlow> dmlloyd: thanks :)
[14:58] <dmlloyd> that email chain was somewhat vague in terms of actual requirements
[14:58] <dmlloyd> so I threw out a bunch of possible solutions to common situations
[14:58] <dmlloyd> but the book is not exhausted by any means
[14:58] <sebersole> i am really thinking that it is best to to not have each jar/module handle this itself
[14:58] <sebersole> dmlloyd: i saw some of those
[14:58] <sebersole> i just joined that mailing list though
[14:59] <sebersole> like 5 minutes ago
[14:59] <dmlloyd> the best approach for a given situation really depends on what it is you need to accomplish and why
[14:59] <sebersole> dmlloyd: sure
[14:59] <sebersole> so lets take the first one
[15:00] <sebersole> Integrator, which is the service loader thing emmanuel asked about specifically
[15:00] <sebersole> so Integrator is a contract defined in hibernate orm jars
[15:00] <sebersole> other jars would contain impls
[15:00] <sebersole> and have their serevice loader locator file
[15:01] <sebersole> META-INF/services/org.hibernate...
[15:01] <sebersole> the original suggestion was a bi-directional dep
[15:01] <dmlloyd> are the implementations typically scoped to a be system-wide, per-application, or what?
[15:01] <sebersole> which i think is awful (imho)
[15:02] --> barreiro has joined this channel (~barreiro at 2.83.128.38).
[15:02] <sebersole> well thats a good question that i can't sum up in a single word :)
[15:02] <sebersole> its per application's choice of orm and integrator combo
[15:02] <sebersole> so per-application maybe
[15:03] <sebersole> not sure how that fits in your terms
[15:03] <sebersole> an example would be hibernate ogm
[15:03] <dmlloyd> a complicating factor is that a module system is a general-purpose system which you could run any app on, but *our* module distribution is really just an AS distribution
[15:03] <sebersole> which has one of these Integrators
[15:03] <dmlloyd> so things you'd do in a general sense you'd maybe do differently for that reason
[15:03] <sebersole> well i think the problems are general
[15:04] <sebersole> i dont think they are specific to "this particular" module sustem
[15:04] <dmlloyd> yeah but the solutions can be more pragmatic rather than more idealistic
[15:04] --> mbabacek has joined this channel (~karm at redhat/jboss/mbabacek).
[15:04] *** ChanServ gives mbabacek permission to talk.
[15:04] <sebersole> though you obviously have more exp
[15:04] <dmlloyd> right, I get what you're saying
[15:04] <sebersole> like i was saying above..
[15:04] <dmlloyd> the elephant in the room is that there isn't a general-purpose implementation mechanism/registry
[15:04] <sebersole> one general pattern i seem to see is that of a set of "intergation exports"
[15:05] <sebersole> as oppsed to api exports
[15:05] <sebersole> this is exactly the reason i decided to split hibernate into api/spi/internal
[15:05] <sebersole> in terms of packaging
[15:05] <-- ttarrant has left this server (Ping timeout: 265 seconds).
[15:05] <sebersole> to help facilitate this
[15:06] <sebersole> the spis need to be exposed for communication between "modules"
[15:06] <-- mdobozy has left this server (Quit: Leaving.).
[15:06] <sebersole> but they should not be exported to the app itself
[15:06] <sebersole> if that makes sense
[15:06] <dmlloyd> sure
[15:07] <sebersole> the real problem i have with defining dep on both sides...
[15:07] <sebersole> is custom modules, or customizing modules
[15:07] <sebersole> now we have users mucking with AS provided modules
[15:07] <dmlloyd> right
[15:07] <Nihility> quick question, do you ever want users to be able to deploy say a different ogm or ORM?
[15:08] <Nihility> but yet work with the containers opposite
[15:08] <sebersole> Nihility: smarlow does :)
[15:08] --> rhusar has joined this channel (~rhusar at ip-89-103-39-182.net.upcbroadband.cz).
[15:08] <sebersole> i personally dont :)
[15:08] <dmlloyd> well, what we have today is based on the fundamental assumption that we currently only have two practical/performant options for finding implementations: the API class loader and an implicit (TCCL) or explicit classloader parameter
[15:09] <dmlloyd> so, everything has grown from there
[15:09] <-- jcacek_wfh has left this server (Quit: Ex-Chat).
[15:09] <Nihility> theres two ways you can go about that which have a large impact
[15:09] --> oskutka has joined this channel (~ony at 212-96-177-49.cust.selfnet.cz).
[15:09] --> jjaggars has joined this channel (~jjaggars at 12.237.225.202).
[15:09] <Nihility> either you consider all of the hibernate stuff to be one fixed group
[15:09] <sebersole> Nihility: if i were doing that, i'd personally rather have an explicit module file
[15:09] <Nihility> (e.g. orm version A with ogm version B only)
[15:09] <sebersole> and it would name orm version x and ogm version y
[15:10] <sebersole> Nihility: yeah there is a problem witrh that
[15:10] <sebersole> in that they do not follow the same release cycle
[15:10] <Nihility> and then you allow overriding the set
[15:10] <Nihility> but not the components
[15:10] <sebersole> so versioning that gets complicated
[15:10] <sebersole> sure, i get what you are saying
[15:10] <Nihility> or you let both be replaced
[15:10] <Nihility> and let the user figure out the right combination
[15:10] <Nihility> they are "on there own" as it were
[15:11] <Nihility> and we provide the recommended combination by default
[15:11] <sebersole> Nihility: like I said, if i were doing that, i'd personally rather have an explicit module file
[15:11] <sebersole> as part of the app
[15:11] <sebersole> naming the explicit deps
[15:11] <sebersole> orm version x, ogm version y
[15:11] <sebersole> but
[15:12] <sebersole> that still doesnt solve the underlying problem
[15:12] <Nihility> the other question is do you want to support in-deployment components
[15:12] <Nihility> like i deploy hibernate orm x.y.z
[15:12] <sebersole> in that orm still could not see ogm classloader
[15:12] <Nihility> and i expect it to wire up with ogm
[15:12] <Nihility> which is provided by the system
[15:12] <sebersole> Nihility: the problem is that that is not the nature
[15:13] <sebersole> the nature of these is that you have orm
[15:13] <sebersole> and ogm, via this Integrator, reconfigures orm
[15:13] <sebersole> but
[15:13] <sebersole> plenty of people use orm w/o ogm
[15:14] <sebersole> in fact thats the 80% use case
[15:14] <sebersole> 80%+
[15:14] <dmlloyd> it's not clear what the arity is here - is any of the behavior you're talking about global, or is it per instance of ORM (if there is such a thing)?
[15:14] *** qmx is now known as qmx|away.
[15:15] <sebersole> dmlloyd: for orm+ogm in particular?
[15:15] <dmlloyd> yeah
[15:16] <Nihility> yeah basically we should describe this from a users point of view. What should they be able to do (ideally) in your opinion
[15:16] <smarlow> currently, app deployments could have their own instance of the ORM classes or use ORM via a module
[15:16] <sebersole> well there its a combo
[15:16] <dmlloyd> (also is there anything they can do now that they shouldn't be able to?)
[15:16] <smarlow> by instance, I mean classloader
[15:16] <Nihility> dont worry so much about the how, just if all magic worked how could they use this stuff
[15:16] <sebersole> an app could (1) use orm, (2) use orm+ogm
[15:16] <sebersole> Nihility: yes, trying :)
[15:17] <sebersole> so in that orm+ogm combo
[15:17] <sebersole> they might (a) use the orm and ogm modules
[15:17] <sebersole> (b) use the orm module and supply ogm themselves
[15:17] <sebersole> (c) supply both prm and ogm themselves
[15:17] <sebersole> (d) supply custom module(s)
[15:18] <sebersole> at least thats how i understand the possibilities in AS
[15:18] --> rbalent has joined this channel (~rbalent at ip-62-245-108-79.net.upcbroadband.cz).
[15:18] <-- rbalent has left this server (Changing host).
[15:18] --> rbalent has joined this channel (~rbalent at redhat/jboss/rbalent).
[15:18] *** ChanServ gives rbalent permission to talk.
[15:18] <jbossbot> git [jboss-as] push master 8c9cb5d.. James Perkins Updated/added translations
[15:18] <jbossbot> git [jboss-as] push master URL: http://github.com/jbossas/jboss-as/compare/f55e762...8c9cb5d
[15:19] <sebersole> tbh, i personally think the app should name that it wants those modules as deps
[15:19] <jbossbot> git [jboss-as] push 7.1 8d4712f.. James Perkins Updated/added translations
[15:19] <jbossbot> git [jboss-as] push 7.1 URL: http://github.com/jbossas/jboss-as/compare/9d3beda...8d4712f
[15:19] <sebersole> orm is somehow discovered to be needed
[15:19] <sebersole> iiuv
[15:19] <sebersole> iiuc
[15:19] <sebersole> but thats not really possible with ogm
[15:20] --> ALR has joined this channel (~alr at redhat/jboss/ALR).
[15:20] *** ChanServ gives ALR permission to talk.
[15:20] <sebersole> its a completely "behind the scenes" deal
[15:20] <-- vjuranek has left this server (Ping timeout: 240 seconds).
[15:20] --> vjuranek has joined this channel (~vjuranek at ip-89-102-30-62.net.upcbroadband.cz).
[15:20] <sebersole> but we will certainly be doing some of what Nihility suggested
[15:21] <sebersole> like envers really should be part of the hibernate orm module
[15:21] <sebersole> it has this same Integrator deal
[15:21] <sebersole> but it will be part of that module classloader, so that will be fine
[15:21] --> pmuir has joined this channel (~pmuir at redhat/jboss/pmuir).
[15:21] *** ChanServ gives pmuir permission to talk.
[15:22] <sebersole> envers is released in cycle with orm, so that makes perfect sense to me
[15:22] <sebersole> (that its part of the same module i mean)
[15:23] <sebersole> dmlloyd: another suggestion i was told you made was to have ogm in this example register its classloader with orm
[15:23] <sebersole> which i think is ok in the general sense
[15:23] <sebersole> but
[15:23] <sebersole> i think that should be done by the module itself somehow
[15:24] <dmlloyd> it kinda sounds like (in the AS case at least) each deployment should be configurable with the exact combination of what pieces it wants
[15:24] <sebersole> not siure if the module allows such "hooks"
[15:24] <sebersole> dmlloyd: modules have just a single export correct?
[15:25] <dmlloyd> yes and no
[15:26] <dmlloyd> if you have a class in a module (or any other class loader), it can only ever be linked once
[15:27] <dmlloyd> so this means that by extension, to give an example, an API implementation module can only ever link against one API module
[15:27] <sebersole> ok
[15:27] <dmlloyd> if you want two versions of the API module, you need two versions of the implementation module
[15:27] <dmlloyd> now this has nothing to do with modules, it's just the way class loading works
[15:27] <dmlloyd> now *that said*
[15:28] <sebersole> sure
[15:28] <dmlloyd> there's no reason that you cant have N implementations for a given API
[15:29] <sebersole> dmlloyd: well more what i mean is whether a module could export its api and spi separately
[15:29] <sebersole> i am going off osgi names 
[15:29] <dmlloyd> well for modules in particular, they always export everything; the filtering happens at import
[15:29] <sebersole> sorry if that does not match up exactly with as
[15:29] <-- kenfinnigan has left this server (Quit: Leaving.).
[15:29] <dmlloyd> so you could introduce an intermediate module for example which filters down the full set of another module, if you wanted to
[15:30] <dmlloyd> but normally I'd say if you have three JARs they should be three modules
[15:30] <sebersole> well unfortunately we ended up having to go the other direction
[15:30] <sebersole> to work around these problems
[15:30] <dmlloyd> you have one JAR?
[15:30] <sebersole> the hibernate orm module contains multiple jars
[15:31] <dmlloyd> that's unfortunate, and actually shouldn't be necessary since you can do everything with three modules that you can do with one
[15:31] <sebersole> hibernate orm is a project
[15:31] <sebersole> it produces 6-7 jars
[15:31] <dmlloyd> s/three/N/
[15:31] <sebersole> sure i knew what ya meant :)
[15:32] <dmlloyd> basically it boils down to, fewer modules = fewer options
[15:32] <sebersole> more options not always == good :)
[15:32] <-- pferraro has left this server (Quit: pferraro).
[15:32] <sebersole> often more options -> more confusion
[15:33] <dmlloyd> I mean for you, not for users
[15:33] <sebersole> whcih is where i feel we are
[15:33] <sebersole> well
[15:33] <sebersole> so lets explore that...
[15:33] <dmlloyd> well I think the module organization is a separate question from how to interface this stuff to users, it's just that so far, messing with modules is just how it's been done
[15:33] <sebersole> so today hibernate-envers is a separate module
[15:34] <sebersole> and we have exactly this service loader problem there as well
[15:34] <sebersole> we actually planned on moving it into the main orm module to work around that
[15:35] <smarlow> we use the 1-1 bidirectional dependency currently or envers
[15:35] <smarlow> we use the 1-1 bidirectional dependency currently *for* envers
[15:35] <dmlloyd> fundamentally, using service loader by itself breaks down a bit when you've got multiple combinations of APIs and impls
[15:35] <sebersole> which i wanted to do away with
[15:35] <dmlloyd> it works well for e.g. jboss marshalling where you only have one or two implementations *ever*
[15:36] <sebersole> dmlloyd: i disagree wrt service loader
[15:36] <sebersole> i think it works perfect for how hibernate uses it as well
[15:36] <dmlloyd> at the lowest level, if you want to be able to stitch together arbitrary pieces you have to be able to tell it what class loaders to get what pieces from
[15:36] <dmlloyd> (for a given "instance" of the framework)
[15:36] <-- ncross has left this server (Quit: Konversation terminated!).
[15:37] <sebersole> you have to understand that Integrator is used to reconfigure parts of orm, prior to it filly starting up
[15:37] <sebersole> fully
[15:37] <sebersole> they are not used during runtime
[15:38] <sebersole> your service loader example is a example of "supply us with some custom override behavior"
[15:38] <sebersole> here, we truly might have multiple integrations
[15:38] <dmlloyd> right well I should say "using service loader with the calling class loader"
[15:38] <dmlloyd> should have said
[15:39] <sebersole> well i dont do that either ;)
[15:39] <sebersole> i have an extremely verbose bit of code trying to use multiple classloaders
[15:39] --> ncross has joined this channel (~rnc at redhat/jboss/ncross).
[15:39] *** ChanServ gives ncross permission to talk.
[15:39] <sebersole> we built in the capability in hibernate for environment to pass in classloading behavior
[15:40] <sebersole> generally that breaks down into it passing in multiple classloaders
[15:40] <sebersole> so i have this capability already
[15:40] <sebersole> it works great
[15:41] <sebersole> the problem is lack of visibility in as7 right now between these things
[15:41] <sebersole> the problem is lack of visibility between these things right now when deployed to as7
[15:42] <sebersole> that sounds better and more accurate :)
[15:42] <dmlloyd> can you give a specific example?
[15:42] <sebersole> i have been :)
[15:42] <sebersole> orm and ogm
[15:42] <dmlloyd> okay those are two JARs?
[15:42] <dmlloyd> I'm just trying to understand the internal relationships
[15:42] <sebersole> for the sake of this discussion. lets say yes :)
[15:42] <dmlloyd> I assume that the ORM has an SPI portion which OGM implements directly?
[15:42] <sebersole> there are mainly 2 jars involved
[15:43] <sebersole> hibernate-core
[15:43] <sebersole> hibernate-ogm
[15:43] <sebersole> hibernate-core is a jar from orm
[15:43] <dmlloyd> a core JAR it would seem :)
[15:43] <sebersole> it contains this Integrator interface
[15:43] <sebersole> lol
[15:43] <sebersole> THE core jar
[15:43] <sebersole> the ruling jar
[15:44] <sebersole> anyway
[15:44] <sebersole> hibernate-core knows nada about hibernate-ogm
[15:44] <sebersole> its not dependent on it at all
[15:44] <sebersole> hibernate-ogm however, has dep on hibernate-core
[15:45] <sebersole> because it implements a number of spis that are defined in hibernate-core
[15:45] <sebersole> however
[15:45] <sebersole> hibernate-core needs to be able to "see" hibernate-ogm
[15:45] <sebersole> to find its service locator
[15:46] <dmlloyd> but only at run time right? not link time?
[15:46] <sebersole> correct
[15:46] <dmlloyd> and there could normally be other implementations?
[15:47] <sebersole> well it also needs to be able to "see" it so it can load classes from it
[15:47] <sebersole> dmlloyd: yep
[15:47] <dmlloyd> okay, sounds like core should *not* have any kind of dep on ogm
[15:47] <sebersole> correct
[15:47] <dmlloyd> (postulating 2 modules for the moment)
[15:47] <sebersole> dmlloyd: right or wrong, i always take this back to build depsa
[15:48] <sebersole> in terms of visualizing
[15:48] <dmlloyd> the module deps should in fact be similar to build deps
[15:48] <sebersole> yep
[15:48] <sebersole> i agree
[15:48] <sebersole> so i am not far off there then... yaay!
[15:48] <dmlloyd> it is true that we *do* use circular deps for certain things but nobody should think that they are *required* to for all similar situations
[15:48] <dmlloyd> in this case it wouldn't be appropriate at all
[15:49] <dmlloyd> but it does give rise to certain questions
[15:49] <sebersole> i agree that circular deps have their place
[15:49] <sebersole> infact we use that in the hibernate orm build! ;)
[15:49] <dmlloyd> the obvious one being, how do you load classes from a certain implementation
[15:49] <sebersole> yep
[15:49] <dmlloyd> and I can tell you how *not* to do it: do *not* rely on the classloader of core, nor TCCL in this case
[15:50] <dmlloyd> (to load classes from OGM)
[15:50] <sebersole> we dont
[15:50] <sebersole> to date we have been using bi-dir deps
[15:50] <sebersole> but it sounds like we all agree those should be removed
[15:50] <dmlloyd> yeah definitely
[15:50] <sebersole> replaced with what though is the question :)
[15:51] <dmlloyd> yeah and that really depends on how your SPI looks
[15:51] <sebersole> https://github.com/hibernate/hibernate-orm/blob/master/hibernate-core/src/main/java/org/hibernate/integrator/spi/Integrator.java
[15:51] <sebersole> thats the spi, pretty simple
[15:52] <sebersole> ignore the second method
[15:53] <sebersole> the basic idea is that it allows callbacks to reshape how the Hibernate SessionFactory will get built
[15:53] <dmlloyd> okay, so this does happen in the context of a deployment then?
[15:54] --> milestone has joined this channel (~milestone at ip-109-84-0-17.web.vodafone.de).
[15:54] <sebersole> in context of building a SessionFactory
[15:54] <sebersole> which *might* be deployment
[15:54] <sebersole> aka, in the case of container managed EMF
[15:55] <sebersole> or do you just mean in terms of scope?
[15:55] <-- whitingjr has left this channel.
[15:55] <sebersole> it will be scoped to that deployemnt that triggered the SF building
[15:56] <dmlloyd> okay so we use SF for internal stuff sometimes, and deployments other times then?
[15:56] <dmlloyd> so which implementations to select depend on various factors it seems
[15:57] <-- rmartinelli has left this server (Quit: Leaving).
[15:57] <sebersole> dmlloyd: an app could build a SessionFactory themselves
[15:58] <sebersole> in the case of JPA, it could manage the EMF itself
[15:58] <sebersole> EMF roughly equates to SessionFactiry
[15:58] <sebersole> SessionFactory
[16:00] <sebersole> dmlloyd: you mean which implementations to select in terms of Integrator?
[16:00] <sebersole> if so then yeah
[16:00] <sebersole> its really app specific
[16:00] <sebersole> going back to build dep visualization
[16:00] <-- vjuranek has left this server (Quit: Konversation terminated!).
[16:00] <sebersole> its really up to them to name which depns they want
[16:01] *** pilhuhn is now known as pil-zZzzzZZzzzz.
[16:04] <dmlloyd> from a deployment perspective then, what we really need is an easy to use and concise means to spell it all out
[16:04] <sebersole> yep
[16:05] <dmlloyd> (for internal stuff we can just use whatever low-level APIs, doesn't matter as much)
[16:05] <sebersole> org.hibernate:orm:{version}?
[16:05] <sebersole> are modules versioned explicitly?
[16:05] <sebersole> or just via name?
[16:05] <-- adietisheim has left this server (Quit: Leaving.).
[16:05] <dmlloyd> no, they're versioned by category or "slot"
[16:06] <dmlloyd> I think we have a 3 and 4 category for hibernate (right smarlow?)
[16:06] <dmlloyd> the categories would typically be mutually exclusive though
[16:06] <sebersole> oh, ok
[16:06] <dmlloyd> i.e. you wouldn't mix 3.x and 4.x
[16:06] <smarlow> dmlloyd:  currently yes, we have org.hibernate:main for 4.x and org.hibernate:3 for 3.x
[16:06] <sebersole> maybe main should be orm? ;)
[16:07] <dmlloyd> but I think that users probably shouldn't concern themselves with the module aspect, wouldn't you agree?
[16:07] <sebersole> smarlow: but dmlloyd was recommending splitting that module up into one per jar
[16:07] <sebersole> dmlloyd: just not sure of another way to do it
[16:07] <dmlloyd> we can rename modules and maintain an alias for the old name (just saying, that's something that's possible if you wanted to do it)
[16:07] <dmlloyd> well I mean we could have a DD or something
[16:08] <sebersole> sure, but specifying wqhat?
[16:08] *** balunasj is now known as balunasj|away.
[16:08] <dmlloyd> something that says "I want hibernate 4.x.  I want this OGM thing plus that thing plus that thing.  Make it work"
[16:08] <dmlloyd> whatever options the user has
[16:08] <sebersole> but thats modules isnt it ;)
[16:08] <dmlloyd> that's how it's implementated
[16:08] <dmlloyd> implemented*
[16:08] <dmlloyd> but not how it's typically specified
[16:08] <sebersole> well you know that way better than me
[16:09] <smarlow> currently, they could specify a PU property to indicate which provider module should be used.
[16:09] <sebersole> and actuall smarlow has something sorta kinda similar today
[16:09] <dmlloyd> I mean if what we're talking about is just giving them a choice between 3 and 4 then that's pretty simple
[16:09] <sebersole> dmlloyd: sure
[16:09] <sebersole> but its more than that
[16:09] <dmlloyd> but if they need to be able to plug in impls, or their own accessory modules or whatever then that needs to be spelled out
[16:09] <sebersole> it a choice between 3/4 + what integratiosn they wat
[16:09] <-- galderz has left this server (Quit: Leaving).
[16:10] <smarlow> its eclipselink/openjpa/datanucleus/other providers also
[16:10] <dmlloyd> hibernate integrations?
[16:10] <sebersole> no, he means alternate jpa providers
[16:10] <sebersole> oh, you asking me
[16:10] <dmlloyd> okay
[16:10] <sebersole> well, yeah, ogm
[16:10] <dmlloyd> we're not at JPA providers yet :)
[16:10] <sebersole> envers
[16:10] <smarlow> okay :)
[16:11] <-- wolfc has left this server (Quit: Leaving.).
[16:11] <sebersole> so its not just "i want hibernate 4"
[16:11] <sebersole> its, "i want hibernate 4 and ogm"
[16:11] <sebersole> or
[16:11] <sebersole> "i want hibernate 4 and ogm and envers"
[16:11] <sebersole> "i want hibernate 4 and ogm and envers and bean validation"
[16:11] <dmlloyd> would things like envers or ogm have a 3.x vs 4.x variant?
[16:12] <sebersole> yes
[16:12] <sebersole> though
[16:12] <smarlow> bean validation is different, we pass that in via properties
[16:12] <sebersole> we dont have to support them i dont think
[16:12] <sebersole> smarlow: no
[16:12] <dmlloyd> okay so they'd say what major version of "stuff" they want, and then what other optional bits
[16:12] <sebersole> you pass in the factory
[16:12] <sebersole> the integrator still needs to run
[16:13] <smarlow> sebersole: okay, if you mean choosing a different version, your right
[16:13] <dmlloyd> and then we'd just put it into the oven and give them their SF
[16:13] <dmlloyd> now we *could* do that with module names *but* the user would then have to have knowledge of what our module names are, and also of the various other flags they'd have to set on the module in order to make it so that the TCCL could be used to load all the pieces up
[16:14] <-- milestone has left this server (Remote host closed the connection).
[16:14] <dmlloyd> alternatively we could just set up a DD with a custom documented XSD that spells it all out in plain english (which also makes it possible to avoid "leaking" impls on to the deployment class path, if that's important to anyone)
[16:14] <sebersole> dmlloyd: so the alt you think is (1) name the category (2) name the intg pieces?
[16:15] <sebersole> so something like (1) i want hibernate 4 (2) I want orm, envers, ogm
[16:16] <sebersole> as far as leaking, yeah the Integrator does have a "shutdown" piece if you notices
[16:16] <sebersole> noticed
[16:16] <sebersole> that should be called on undeploy, shutdown etc
[16:17] <-- jfd has left this server (Ping timeout: 245 seconds).
[16:18] <sebersole> dmlloyd: so is this DD handled explicitly by the hibernate deployer?
[16:18] <sebersole> assuming there is still a hibernate deployer :)
[16:19] --> jfd has joined this channel (jdoyle at nat/redhat/x-oajxfaslcvqnxoad).
[16:19] <-- jfd has left this server (Changing host).
[16:19] --> jfd has joined this channel (jdoyle at redhat/jboss/jfd).
[16:19] *** ChanServ gives jfd permission to talk.
[16:20] <smarlow> sebersole:  no, we don't have a hibernate deployer 
[16:21] <dmlloyd> not presently
[16:21] <smarlow> the as6 hibenate deployer is still there though :)
[16:21] <dmlloyd> but yeah sebersole that seems to me like the best way to do it from the user's perspective
[16:21] <alesj> sebersole: i haven't rewriten it yet … this time :-)
[16:21] <dmlloyd> the idea would be to make it something that is future-compatible so that we don't keep changing it on every release
[16:22] <dmlloyd> we do this by avoiding specific member, class and module names and instead favor general descriptions and flags that make sense to the user
[16:22] <sebersole> dmlloyd: what about a custom module that contains an Integrator
[16:22] <dmlloyd> then saying "we will no longer support XYZ in AS 9" == deprecating a specific XML element or similar
[16:23] <dmlloyd> well yeah in that case it would be allowed
[16:23] <dmlloyd> I just mean for our stuff
[16:23] <dmlloyd> our DDs should refrain from exposing internal implementation details whenever possible
[16:23] <dmlloyd> 5/6 was majorly guilty of this
[16:23] <sebersole> i just mean what it looks like in DD, would it name the custom module in that case?
[16:23] <dmlloyd> yeah you'd probably say "give me envers, ogm, and this module "foo.bar""
[16:23] <sebersole> dmlloyd: hibernate still does that today :(
[16:24] <jbossbot> new jira [AS7-4858] jconsole.sh does not start on Mac OSX [Open (Unresolved) Feature Request, Major, Jason Greene] https://issues.jboss.org/browse/AS7-4858
[16:24] <sebersole> lot of impl class names in config settings
[16:25] *** rcernich_away is now known as rcernich.
[16:26] <dmlloyd> speaking from a place of total ignorance, if I were using OGM in my application, would that imply that I need access to additional API, or is it all just behind-the-scenes/configuration stuff?
[16:26] <dmlloyd> (just picking out OGM as an example, but any integrator really)
[16:27] <ctomc> alesj: ping
[16:28] <alesj> ctomc: wazup?
[16:28] <smarlow> OGM is a JPA persistence provider, so I would expect javax.persistence would be one of APIs needed
[16:29] <dmlloyd> okay.  maybe that's a bad example then because they should get that API anyway given that it's a JEE thing
[16:29] <dmlloyd> it seems like, generally speaking, the app doesn't really need to link to these pieces as long as the SF remembers how to access them
[16:31] <sebersole> dmlloyd: for ogm, just configuration stuff
[16:31] <sebersole> but
[16:31] <sebersole> in the case of say envers, that is not the case
[16:31] <sebersole> that does expose additional apis
[16:31] <dmlloyd> okay, so it'd depend on the piece then?
[16:32] <dmlloyd> it really only matters in the case of a custom module since the DD parser would have direct control over what it does with the other stuff
[16:32] <sebersole> are you asking in terms of exporting to the app?
[16:32] <dmlloyd> yeah
[16:32] <sebersole> i think it would be safe to just add them as an export
[16:32] <dmlloyd> okay.  limiting pollution is somewhat desirable, but not worth going nuts over
[16:33] <sebersole> dmlloyd: well lots of them add custom hibernate event listeners
[16:33] <sebersole> the app *may* want to interact with those
[16:33] <sebersole> its hard to say unequivocally that the app wont
[16:33] <dmlloyd> sounds reasonable
[16:33] <sebersole> in the case of some i can say it has to be exported
[16:33] <sebersole> envers
[16:34] <sebersole> validator
[16:34] <sebersole> search
[16:34] <sebersole> ogm... just not sure
[16:34] <sebersole> better safe than sorry i think
[16:34] <sebersole> but i hear what you are saying
[16:35] <-- jfd has left this server (Ping timeout: 245 seconds).
[16:35] <dmlloyd> as long as we don't use TCCL as the single rally point for everything we have options
[16:35] <sebersole> https://github.com/hibernate/hibernate-orm/blob/master/hibernate-core/src/main/java/org/hibernate/integrator/internal/IntegratorServiceImpl.java
[16:36] <sebersole> which makes use of this classLoaderService.loadJavaServices( Integrator.class ) call
[16:36] <sebersole> https://github.com/hibernate/hibernate-orm/blob/master/hibernate-core/src/main/java/org/hibernate/service/classloading/internal/ClassLoaderServiceImpl.java
[16:36] <sebersole> if you are interested
[16:37] <sebersole> so, no, i go out of my way to avoid tccl
[16:37] <dmlloyd> good.  If only everyone would :)
[16:37] <-- Jesmith17 has left this server (Remote host closed the connection).
[16:37] <smarlow> sebersole, dmlloyd:  I have to go offline, will catch up with you guys later...  
[16:37] <sebersole> well i take it back, i do use tccl as a fall back
[16:38] <sebersole> but the AS integration passes me in classloader to use
[16:38] <sebersole> so i dont get to that fallback
[16:38] <dmlloyd> okay so what do you think of the DD idea?
[16:39] <sebersole> i think its great
[16:39] <sebersole> i think it absolutely has to be up to deployer input
[16:39] <sebersole> deployer/developer
[16:39] <sebersole> deployer overloaded i guess :)
[16:40] <dmlloyd> develoployer!
[16:40] <sebersole> lol
[16:40] <sebersole> thanks for your time
[16:40] <-- pmuir has left this server (Quit: Leaving).
[16:40] <dmlloyd> no prob
[16:41] <-- fharms has left this server (Ping timeout: 260 seconds).
[16:42] <-- smarlow has left this server (Ping timeout: 244 seconds).
[16:44] <sebersole> dmlloyd: how do you feel about the DD solution?
[16:44] <sebersole> it was your suggestion, so i assume you cool with it
[16:46] <-- ncross has left this server (Ping timeout: 246 seconds).
[16:48] <-- smcgowan has left this server (Quit: Leaving).
[16:49] <-- ozizka-ntb has left this server (Ping timeout: 245 seconds).
[16:55] *** abstractj is now known as abstractj|away.
[16:55] <dmlloyd> yeah I prefer it
[17:00] --> mbg has joined this channel (~mbg at redhat/jboss/mbg).
[17:00] *** ChanServ gives mbg permission to talk.
[17:04] <-- jjaggars has left this server (Quit: Leaving.).
[17:06] <-- marekn has left this channel.
[17:09] <sebersole> dmlloyd: on other thought...
[17:10] <sebersole> what if the develoployer ends up bundling hibernate jars?
[17:10] <sebersole> what does the DD look like in that case?
[17:10] <dmlloyd> is that something we want to support?
[17:10] <sebersole> scott wants to.  and people do it
[17:10] <dmlloyd> okay.  I guess in that case it might make sense to have a DD option that's like "hibernate = disabled" or something
[17:11] --> doctau has joined this channel (jlivings at nat/redhat/x-vghzdeabsxpeecik).
[17:11] *** ChanServ gives doctau permission to talk.
[17:11] <dmlloyd> but that's naively assuming that we don't need to detect this and hook it up into JPA or anything like that
[17:11] <dmlloyd> (in any non-spec manner; I realize that custom JPA providers are probably something we need to support)
[17:12] <sebersole> i guess that comes down to whether we would support non-hibernate providers in the deployment
[17:12] <sebersole> or if they would need to be module too
[17:13] <sebersole> tbh, i am just thinking of the simple war case
[17:13] <sebersole> where they through hibernate and friends into web-inf/lib
[17:13] <sebersole> throw even
[17:13] <dmlloyd> sometimes they do that on purpose and sometimes they do it on accident
[17:13] <sebersole> sure
[17:13] <dmlloyd> ideally we'd be able to tell which it is
[17:13] <sebersole> i get that second part as well
[17:14] <sebersole> how?
[17:14] *** abstractj|away is now known as abstractj.
[17:14] <dmlloyd> don't know
[17:14] <sebersole> ah
[17:14] <sebersole> yea, me either
[17:14] <dmlloyd> maybe based on the presence of the DD or something
[17:14] --> pferraro has joined this channel (~paul at redhat/jboss/pferraro).
[17:14] *** ChanServ gives pferraro permission to talk.
[17:15] <sebersole> so i guess the idea with "disable" is that its already in the app classloader?
[17:15] <dmlloyd> yeah, or you want to disable it for some other reason like an alt JPA impl
[17:15] <sebersole> so we just skip doing anything special
[17:15] <dmlloyd> if that makes any sense
[17:15] <sebersole> sure
[17:16] <-- barreiro has left this server (Ping timeout: 246 seconds).
[17:16] <dmlloyd> the tricky case is when there's a hibernate JAR in the deployment and no other DDs or anything
[17:16] <dmlloyd> hard to know what the intent is in this cas
[17:17] <dmlloyd> e
[17:20] <sebersole> yep
[17:21] <sebersole> is your suggestion to just ignore the deployment provided ones in that case?
[17:21] <sebersole> its a 50/50 proposition I guess in that case
[17:23] <dmlloyd> yeah I don't have a good answer
[17:23] <dmlloyd> for logging we try to always ignore what's in the deployment unless they specifically tell us not to do that
[17:23] <dmlloyd> not sure it works 100% as well as we'd like though
[17:24] <sebersole> well the only drawback here is potentially missing one or more of those integrations that may have been in the deployment
[17:25] <sebersole> actually the "app classloader" is one scott explicitly passes into that Hibernate code I linked before
[17:26] <sebersole> so unless he explicitly excludes those jars from the app classpath we'd have problems
[17:28] <sebersole> dmlloyd: just to clarify, this is a specific DD you are talking about right?  hibernate-deployment.xml or somesuch?
[17:28] <dmlloyd> yeah
[17:29] <dmlloyd> though we were talking about the idea of a central DD for AS 8 or 9 or something, a DD analog to the central management config idea
[17:29] <dmlloyd> so maybe we'd migrate towards taht eventually if it ever happens
[17:30] <sebersole> ok
[17:33] <-- rbalent has left this server (Quit: May the Force be with you).
[17:37] <-- pgier has left this server (Quit: Leaving).