[jboss-as7-dev] Modularity is the spawn of Lucifer and a stinking donkey

[Jason T. Greene]

On 5/10/12 11:21 AM, David M. Lloyd wrote:
> On 05/10/2012 10:24 AM, Scott Marlow wrote:
>> On 05/09/2012 02:01 PM, David M. Lloyd wrote:
>>> OK I admit I LOL'ed.
>>>
>>> On 05/09/2012 11:50 AM, Emmanuel Bernard wrote:
>>>> Now that I have your attention, I'd like to discuss issues we are
>>>> experiencing when trying to modularize the Hibernate portfolio and
>>>> make it work in AS 7.1.
>>>>
>>>> ## Disclaimer
>>>>
>>>> I perfectly understand all the coolness about modularity (speed,
>>>> easier dependency isolation etc). I have also carefully read :
>>>>
>>>> - https://community.jboss.org/wiki/ModuleCompatibleClassloadingGuide
>>>> - https://community.jboss.org/wiki/ModularSerialization
>>>>
>>>> But these tend to avoid the more complex cases of portable libraries
>>>> that ought to run even outside AS 7 but have a wide variety of class
>>>> and resource loading needs.
>>>> I am not a complete modularity bozo but I am definitely not familiar
>>>> with JBoss Modules nor similar solution.
>>>>
>>>> ## Requirements / Landscape
>>>>
>>>> Hibernate ORM uses the notion of service registry and integrator
>>>> object that help during the integration or customization of the
>>>> engine behavior by third-party frameworks.
>>>> Enlistment of Integrators is done via the service locator pattern (a
>>>> service file in META-INF/services/ that is looked up and contain the
>>>> implementation class(es) at stake.
>>>>
>>>> Hibernate Envers is one of those customizer that depends on Hibernate
>>>> ORM. Note that the core of Hibernate ORM does not depend on Hibernate
>>>> Envers. The service locator file is contained in Hibernate Envers JAR.
>>>> Hibernate OGM likewise, heavily customizes ORM and depends on
>>>> Hibernate ORM classes - the reverse is not true. The service locator
>>>> file is contained in Hibernate OGM JAR.
>>>> Hibernate Search optionally depend on Hibernate ORM and JPA. The core
>>>> of Hibernate Search is independent but an Hibernate Search ORM module
>>>> has an integrator implementation. On top of that, Hibernate Search
>>>> optionally depend on some JPA classes and behaves differently if they
>>>> are there - we look them up in the classpath by reflection.
>>>>
>>>> On top of that, these projects do load resources (config files,
>>>> classes):
>>>>
>>>> - from what Jason calls a Deployment classloader (the user
>>>> application classes and resources really) - entities, custom analyzer
>>>> implementations, resources files etc. A user could even write a
>>>> custom Integrator and use the service locator pattern from his
>>>> application.
>>>> - from direct dependencies (Lucene is a declared dependency of
>>>> Hibernate Search)
>>>> - from dependencies of the deployment: for example an app developer
>>>> adds the phonetic analyzer as a dependency of his application and ask
>>>> Hibernate Search to use it
>>>> - from modules that use these projects. Modeshape and Capedwarf are
>>>> being modularized and are making use of Hibernate Search as a module.
>>>> Properly loading the necessary classes located in Modeshape or
>>>> Capedwarf's module but from Hibernate Search's engine proves to be
>>>> very hard in our current approach.
>>>>
>>>> All of these projects should be able to run outside JBoss AS 7, so a
>>>> modular friendly solution should somehow be abstracted and generic
>>>> enough.
>>>>
>>>> ## What solution?
>>>>
>>>> More and more projects are being modularized including ones with
>>>> complex resource loading dependencies like the ones I have described.
>>>> AFAIK Infinispan is even in a worse situation as clustering and late
>>>> class binding is at stake but let's put this one aside.
>>>> I'd love to get a reference design outcome from this thread that
>>>> could be copied across all these projects and future ones like Bean
>>>> Validation.
>>>>
>>>> Today, we mostly use the good old and simple TCCL model which works
>>>> fine if the jars are directly embedded in the app but fail the minute
>>>> we start to move these dependencies into modules. Sanne, Strong,
>>>> Scott Marlow and I are using dangerous amount of Advil to try and
>>>> make everything work as expected. Some help would be awesome.
>>>>
>>>> To sum up:
>>>>
>>>> - can the Hibernate portfolio be supported within JBoss Module and how?
>>>> - what kind of ClassloaderService contract should we use within these
>>>> projects to be modular friendly (JBoss Modules and others)?
>>>> - would such contract be generic enough to be extrapolated to JSRs in
>>>> need of modular friendliness?
>>>> - how to solve the chicken and egg issue of the bootstrapping: if we
>>>> need to pass a ClassloaderService impl?
>>> How do we do that best in a modular environment without forcing the
>>> application developer to implement such godforsaken ClassloaderService
>>> contract or even worse pass directly to us the right classloader for
>>> each call.
>>>
>>> I'll just start at the beginning and you can skip over the background if
>>> you like.
>>>
>>> The key starting concept is that a class' (or package's) identity is not
>>> just its name but also its class loader. This is the underlying
>>> (existing) truth that modularity brings to the fore. Corollary to this
>>> are the fact that a single VM may have more than one class or package
>>> with the same name, as well as the fact that not all classes/packages
>>> are always directly visible from a given class loader.
>>>
>>> This problem (as you've seen) manifests itself primarily when you're
>>> locating a class or a resource by name. You basically have two options.
>>> You can search *relative* to a class loader (most commonly, TCCL,
>>> though using a single explicit class loader or the caller's class loader
>>> also fall into this category). Or, you can use the *absolute* identity
>>> of a class.
>>>
>>> Using relative resolution is often a perfectly adequate solution when
>>> you're loading a single class or resource; in fact for some cases (like
>>> ServiceLoader for example) it's a perfect fit in conjunction with TCCL
>>> (in its capacity as an identifier for the "current" application). You
>>> want the user to be able to specify their implementation of something,
>>> and you want it to be reasonably transparent; ServiceLoader+TCCL does
>>> this fairly well.
>>>
>>> ServiceLoader also does well from the perspective of APIs with a static,
>>> fixed number of implementations. In this case, it is appropriate for a
>>> framework to have ServiceLoader use the class loader of the framework
>>> itself. The framework would then be sure to import the implementations
>>> in question (including their service descriptors); in our modular
>>> environment, which we call a "service import". Note that this often
>>> means there is a circular dependency between API and implementation:
>>> that's OK!
>>
>> We currently use this for envers but that doesn't seem as desirable for
>> other members of the Hibernate portfolio that may be on a separate
>> lifecycle. For example, the Hibernate OGM is a persistence provider that
>> depends on Hibernate ORM. If we have Hibernate ORM depend on OGM, that
>> limits the number of OGM versions that can be in use on AS7.
>
> Cases where implementations are pluggable could be handled using TCCL or
> by specifying a class loader, or by explicit registration at run time.
> As I said above, you would only use static dependencies if the
> implementations are more or less fixed.
>
>> Would it be possible, to add a MSC enhancement, that allows an inverse
>> dependency service loader dependency to be expressed? Such that it would
>> be enough to only have OGM depend on ORM (with an inverse service
>> dependency specified). I'm thinking that the OGM module would need to
>> exchange the service dependency information with the ORM module and
>> clear it, when OGM goes away.
>
> MSC has absolutely nothing to do with any of this.  Module loading is
> static and is not affected by services in any way.
>
> In your case you probably want a registration system.  But be aware that
> if you have more than one OGM implementation, you're not going to be
> able to use the ORM API solely to load them anyway (how would you tell
> it which one you want?); instead you'd need some mechanism to specify,
> so why not use TCCL or explicit class loader specification, as appropriate?

In case it's helpful. The way he handle JAXP is that we have a container 
default and we also support TCCL.

The container default is defined by a special module:
https://github.com/jbossas/jboss-as/blob/master/build/src/main/resources/modules/javax/xml/jaxp-provider/main/module.xml


The rules are

1. Use service loader on TCCL (users deployment first), if an impl is 
found use that
2. If not result is find use the default implementation, loaded from the 
classloader of the "jaxp-provider" default module.
3. If the default jaxp-provider module doesn't contain it, we use the 
JVM default, whatever it may be.

The way we implement this relies on installing a false, delegating JAXP 
provider. Although in our own code we should define SPIs of some sort.

So for example, Hibernate OGM could provide a ProviderLocator interface, 
which AS would then implement. Our implementation would follow similar 
logic as above, although maybe you want slightly different rules. OGM 
would ship with a predefined ProviderLocator which would simply search 
TCCL.

-- 
Jason T. Greene
JBoss AS Lead / EAP Platform Architect
JBoss, a division of Red Hat