[jboss-as7-dev] Modularity is the spawn of Lucifer and a stinking donkey

[Scott Marlow]

On 05/10/2012 03:04 PM, Jason T. Greene wrote:
> On 5/10/12 11:21 AM, David M. Lloyd wrote:
>> On 05/10/2012 10:24 AM, Scott Marlow wrote:
>>> On 05/09/2012 02:01 PM, David M. Lloyd wrote:
>>>> OK I admit I LOL'ed.
>>>>
>>>> On 05/09/2012 11:50 AM, Emmanuel Bernard wrote:
>>>>> Now that I have your attention, I'd like to discuss issues we are
>>>>> experiencing when trying to modularize the Hibernate portfolio and
>>>>> make it work in AS 7.1.
>>>>>
>>>>> ## Disclaimer
>>>>>
>>>>> I perfectly understand all the coolness about modularity (speed,
>>>>> easier dependency isolation etc). I have also carefully read :
>>>>>
>>>>> - https://community.jboss.org/wiki/ModuleCompatibleClassloadingGuide
>>>>> - https://community.jboss.org/wiki/ModularSerialization
>>>>>
>>>>> But these tend to avoid the more complex cases of portable libraries
>>>>> that ought to run even outside AS 7 but have a wide variety of class
>>>>> and resource loading needs.
>>>>> I am not a complete modularity bozo but I am definitely not familiar
>>>>> with JBoss Modules nor similar solution.
>>>>>
>>>>> ## Requirements / Landscape
>>>>>
>>>>> Hibernate ORM uses the notion of service registry and integrator
>>>>> object that help during the integration or customization of the
>>>>> engine behavior by third-party frameworks.
>>>>> Enlistment of Integrators is done via the service locator pattern (a
>>>>> service file in META-INF/services/ that is looked up and contain the
>>>>> implementation class(es) at stake.
>>>>>
>>>>> Hibernate Envers is one of those customizer that depends on Hibernate
>>>>> ORM. Note that the core of Hibernate ORM does not depend on Hibernate
>>>>> Envers. The service locator file is contained in Hibernate Envers JAR.
>>>>> Hibernate OGM likewise, heavily customizes ORM and depends on
>>>>> Hibernate ORM classes - the reverse is not true. The service locator
>>>>> file is contained in Hibernate OGM JAR.
>>>>> Hibernate Search optionally depend on Hibernate ORM and JPA. The core
>>>>> of Hibernate Search is independent but an Hibernate Search ORM module
>>>>> has an integrator implementation. On top of that, Hibernate Search
>>>>> optionally depend on some JPA classes and behaves differently if they
>>>>> are there - we look them up in the classpath by reflection.
>>>>>
>>>>> On top of that, these projects do load resources (config files,
>>>>> classes):
>>>>>
>>>>> - from what Jason calls a Deployment classloader (the user
>>>>> application classes and resources really) - entities, custom analyzer
>>>>> implementations, resources files etc. A user could even write a
>>>>> custom Integrator and use the service locator pattern from his
>>>>> application.
>>>>> - from direct dependencies (Lucene is a declared dependency of
>>>>> Hibernate Search)
>>>>> - from dependencies of the deployment: for example an app developer
>>>>> adds the phonetic analyzer as a dependency of his application and ask
>>>>> Hibernate Search to use it
>>>>> - from modules that use these projects. Modeshape and Capedwarf are
>>>>> being modularized and are making use of Hibernate Search as a module.
>>>>> Properly loading the necessary classes located in Modeshape or
>>>>> Capedwarf's module but from Hibernate Search's engine proves to be
>>>>> very hard in our current approach.
>>>>>
>>>>> All of these projects should be able to run outside JBoss AS 7, so a
>>>>> modular friendly solution should somehow be abstracted and generic
>>>>> enough.
>>>>>
>>>>> ## What solution?
>>>>>
>>>>> More and more projects are being modularized including ones with
>>>>> complex resource loading dependencies like the ones I have described.
>>>>> AFAIK Infinispan is even in a worse situation as clustering and late
>>>>> class binding is at stake but let's put this one aside.
>>>>> I'd love to get a reference design outcome from this thread that
>>>>> could be copied across all these projects and future ones like Bean
>>>>> Validation.
>>>>>
>>>>> Today, we mostly use the good old and simple TCCL model which works
>>>>> fine if the jars are directly embedded in the app but fail the minute
>>>>> we start to move these dependencies into modules. Sanne, Strong,
>>>>> Scott Marlow and I are using dangerous amount of Advil to try and
>>>>> make everything work as expected. Some help would be awesome.
>>>>>
>>>>> To sum up:
>>>>>
>>>>> - can the Hibernate portfolio be supported within JBoss Module and
>>>>> how?
>>>>> - what kind of ClassloaderService contract should we use within these
>>>>> projects to be modular friendly (JBoss Modules and others)?
>>>>> - would such contract be generic enough to be extrapolated to JSRs in
>>>>> need of modular friendliness?
>>>>> - how to solve the chicken and egg issue of the bootstrapping: if we
>>>>> need to pass a ClassloaderService impl?
>>>> How do we do that best in a modular environment without forcing the
>>>> application developer to implement such godforsaken ClassloaderService
>>>> contract or even worse pass directly to us the right classloader for
>>>> each call.
>>>>
>>>> I'll just start at the beginning and you can skip over the
>>>> background if
>>>> you like.
>>>>
>>>> The key starting concept is that a class' (or package's) identity is
>>>> not
>>>> just its name but also its class loader. This is the underlying
>>>> (existing) truth that modularity brings to the fore. Corollary to this
>>>> are the fact that a single VM may have more than one class or package
>>>> with the same name, as well as the fact that not all classes/packages
>>>> are always directly visible from a given class loader.
>>>>
>>>> This problem (as you've seen) manifests itself primarily when you're
>>>> locating a class or a resource by name. You basically have two options.
>>>> You can search *relative* to a class loader (most commonly, TCCL,
>>>> though using a single explicit class loader or the caller's class
>>>> loader
>>>> also fall into this category). Or, you can use the *absolute* identity
>>>> of a class.
>>>>
>>>> Using relative resolution is often a perfectly adequate solution when
>>>> you're loading a single class or resource; in fact for some cases (like
>>>> ServiceLoader for example) it's a perfect fit in conjunction with TCCL
>>>> (in its capacity as an identifier for the "current" application). You
>>>> want the user to be able to specify their implementation of something,
>>>> and you want it to be reasonably transparent; ServiceLoader+TCCL does
>>>> this fairly well.
>>>>
>>>> ServiceLoader also does well from the perspective of APIs with a
>>>> static,
>>>> fixed number of implementations. In this case, it is appropriate for a
>>>> framework to have ServiceLoader use the class loader of the framework
>>>> itself. The framework would then be sure to import the implementations
>>>> in question (including their service descriptors); in our modular
>>>> environment, which we call a "service import". Note that this often
>>>> means there is a circular dependency between API and implementation:
>>>> that's OK!
>>>
>>> We currently use this for envers but that doesn't seem as desirable for
>>> other members of the Hibernate portfolio that may be on a separate
>>> lifecycle. For example, the Hibernate OGM is a persistence provider that
>>> depends on Hibernate ORM. If we have Hibernate ORM depend on OGM, that
>>> limits the number of OGM versions that can be in use on AS7.
>>
>> Cases where implementations are pluggable could be handled using TCCL or
>> by specifying a class loader, or by explicit registration at run time.
>> As I said above, you would only use static dependencies if the
>> implementations are more or less fixed.
>>
>>> Would it be possible, to add a MSC enhancement, that allows an inverse
>>> dependency service loader dependency to be expressed? Such that it would
>>> be enough to only have OGM depend on ORM (with an inverse service
>>> dependency specified). I'm thinking that the OGM module would need to
>>> exchange the service dependency information with the ORM module and
>>> clear it, when OGM goes away.
>>
>> MSC has absolutely nothing to do with any of this. Module loading is
>> static and is not affected by services in any way.
>>
>> In your case you probably want a registration system. But be aware that
>> if you have more than one OGM implementation, you're not going to be
>> able to use the ORM API solely to load them anyway (how would you tell
>> it which one you want?); instead you'd need some mechanism to specify,
>> so why not use TCCL or explicit class loader specification, as
>> appropriate?
>
> In case it's helpful. The way he handle JAXP is that we have a container
> default and we also support TCCL.
>
> The container default is defined by a special module:
> https://github.com/jbossas/jboss-as/blob/master/build/src/main/resources/modules/javax/xml/jaxp-provider/main/module.xml
>
>
>
> The rules are
>
> 1. Use service loader on TCCL (users deployment first), if an impl is
> found use that
> 2. If not result is find use the default implementation, loaded from the
> classloader of the "jaxp-provider" default module.
> 3. If the default jaxp-provider module doesn't contain it, we use the
> JVM default, whatever it may be.
>
> The way we implement this relies on installing a false, delegating JAXP
> provider. Although in our own code we should define SPIs of some sort.
>
> So for example, Hibernate OGM could provide a ProviderLocator interface,
> which AS would then implement. Our implementation would follow similar
> logic as above, although maybe you want slightly different rules. OGM
> would ship with a predefined ProviderLocator which would simply search
> TCCL.
>

David/Jason,

It might be helpful if we discussed the Hibernate ORM case on IRC 
(either in #Hibernate-dev or #jboss-as7 (or a private room perhaps).  Do 
you guys have any time this week for that?

Scott