Re: [jboss-as7-dev] Modularity is the spawn of Lucifer and a stinking donkey

On 05/10/2012 03:04 PM, Jason T. Greene wrote: > On 5/10/12 11:21 AM, David M. Lloyd wrote: >> On 05/10/2012 10:24 AM, Scott Marlow wrote: >>> On 05/09/2012 02:01 PM, David M. Lloyd wrote: >>>> OK I admit I LOL'ed. >>>> >>>> On 05/09/2012 11:50 AM, Emmanuel Bernard wrote: >>>>> Now that I have your attention, I'd like to discuss issues we are >>>>> experiencing when trying to modularize the Hibernate portfolio and >>>>> make it work in AS 7.1. >>>>> >>>>> ## Disclaimer >>>>> >>>>> I perfectly understand all the coolness about modularity (speed, >>>>> easier dependency isolation etc). I have also carefully read : >>>>> >>>>> - https://community.jboss.org/wiki/ModuleCompatibleClassloadingGuide >>>>> - https://community.jboss.org/wiki/ModularSerialization >>>>> >>>>> But these tend to avoid the more complex cases of portable libraries >>>>> that ought to run even outside AS 7 but have a wide variety of class >>>>> and resource loading needs. >>>>> I am not a complete modularity bozo but I am definitely not familiar >>>>> with JBoss Modules nor similar solution. >>>>> >>>>> ## Requirements / Landscape >>>>> >>>>> Hibernate ORM uses the notion of service registry and integrator >>>>> object that help during the integration or customization of the >>>>> engine behavior by third-party frameworks. >>>>> Enlistment of Integrators is done via the service locator pattern (a >>>>> service file in META-INF/services/ that is looked up and contain the >>>>> implementation class(es) at stake. >>>>> >>>>> Hibernate Envers is one of those customizer that depends on >>>>> Hibernate >>>>> ORM. Note that the core of Hibernate ORM does not depend on >>>>> Hibernate >>>>> Envers. The service locator file is contained in Hibernate Envers >>>>> JAR. >>>>> Hibernate OGM likewise, heavily customizes ORM and depends on >>>>> Hibernate ORM classes - the reverse is not true. The service locator >>>>> file is contained in Hibernate OGM JAR. >>>>> Hibernate Search optionally depend on Hibernate ORM and JPA. The >>>>> core >>>>> of Hibernate Search is independent but an Hibernate Search ORM >>>>> module >>>>> has an integrator implementation. On top of that, Hibernate Search >>>>> optionally depend on some JPA classes and behaves differently if >>>>> they >>>>> are there - we look them up in the classpath by reflection. >>>>> >>>>> On top of that, these projects do load resources (config files, >>>>> classes): >>>>> >>>>> - from what Jason calls a Deployment classloader (the user >>>>> application classes and resources really) - entities, custom >>>>> analyzer >>>>> implementations, resources files etc. A user could even write a >>>>> custom Integrator and use the service locator pattern from his >>>>> application. >>>>> - from direct dependencies (Lucene is a declared dependency of >>>>> Hibernate Search) >>>>> - from dependencies of the deployment: for example an app developer >>>>> adds the phonetic analyzer as a dependency of his application and >>>>> ask >>>>> Hibernate Search to use it >>>>> - from modules that use these projects. Modeshape and Capedwarf are >>>>> being modularized and are making use of Hibernate Search as a >>>>> module. >>>>> Properly loading the necessary classes located in Modeshape or >>>>> Capedwarf's module but from Hibernate Search's engine proves to be >>>>> very hard in our current approach. >>>>> >>>>> All of these projects should be able to run outside JBoss AS 7, so a >>>>> modular friendly solution should somehow be abstracted and generic >>>>> enough. >>>>> >>>>> ## What solution? >>>>> >>>>> More and more projects are being modularized including ones with >>>>> complex resource loading dependencies like the ones I have >>>>> described. >>>>> AFAIK Infinispan is even in a worse situation as clustering and late >>>>> class binding is at stake but let's put this one aside. >>>>> I'd love to get a reference design outcome from this thread that >>>>> could be copied across all these projects and future ones like Bean >>>>> Validation. >>>>> >>>>> Today, we mostly use the good old and simple TCCL model which works >>>>> fine if the jars are directly embedded in the app but fail the >>>>> minute >>>>> we start to move these dependencies into modules. Sanne, Strong, >>>>> Scott Marlow and I are using dangerous amount of Advil to try and >>>>> make everything work as expected. Some help would be awesome. >>>>> >>>>> To sum up: >>>>> >>>>> - can the Hibernate portfolio be supported within JBoss Module and >>>>> how? >>>>> - what kind of ClassloaderService contract should we use within >>>>> these >>>>> projects to be modular friendly (JBoss Modules and others)? >>>>> - would such contract be generic enough to be extrapolated to >>>>> JSRs in >>>>> need of modular friendliness? >>>>> - how to solve the chicken and egg issue of the bootstrapping: if we >>>>> need to pass a ClassloaderService impl? >>>> How do we do that best in a modular environment without forcing the >>>> application developer to implement such godforsaken >>>> ClassloaderService >>>> contract or even worse pass directly to us the right classloader for >>>> each call. >>>> >>>> I'll just start at the beginning and you can skip over the >>>> background if >>>> you like. >>>> >>>> The key starting concept is that a class' (or package's) identity is >>>> not >>>> just its name but also its class loader. This is the underlying >>>> (existing) truth that modularity brings to the fore. Corollary to >>>> this >>>> are the fact that a single VM may have more than one class or package >>>> with the same name, as well as the fact that not all classes/packages >>>> are always directly visible from a given class loader. >>>> >>>> This problem (as you've seen) manifests itself primarily when you're >>>> locating a class or a resource by name. You basically have two >>>> options. >>>> You can search *relative* to a class loader (most commonly, TCCL, >>>> though using a single explicit class loader or the caller's class >>>> loader >>>> also fall into this category). Or, you can use the *absolute* >>>> identity >>>> of a class. >>>> >>>> Using relative resolution is often a perfectly adequate solution when >>>> you're loading a single class or resource; in fact for some cases >>>> (like >>>> ServiceLoader for example) it's a perfect fit in conjunction with >>>> TCCL >>>> (in its capacity as an identifier for the "current" application). You >>>> want the user to be able to specify their implementation of >>>> something, >>>> and you want it to be reasonably transparent; ServiceLoader+TCCL does >>>> this fairly well. >>>> >>>> ServiceLoader also does well from the perspective of APIs with a >>>> static, >>>> fixed number of implementations. In this case, it is appropriate >>>> for a >>>> framework to have ServiceLoader use the class loader of the framework >>>> itself. The framework would then be sure to import the >>>> implementations >>>> in question (including their service descriptors); in our modular >>>> environment, which we call a "service import". Note that this often >>>> means there is a circular dependency between API and implementation: >>>> that's OK! >>> >>> We currently use this for envers but that doesn't seem as desirable >>> for >>> other members of the Hibernate portfolio that may be on a separate >>> lifecycle. For example, the Hibernate OGM is a persistence provider >>> that >>> depends on Hibernate ORM. If we have Hibernate ORM depend on OGM, that >>> limits the number of OGM versions that can be in use on AS7. >> >> Cases where implementations are pluggable could be handled using >> TCCL or >> by specifying a class loader, or by explicit registration at run time. >> As I said above, you would only use static dependencies if the >> implementations are more or less fixed. >> >>> Would it be possible, to add a MSC enhancement, that allows an inverse >>> dependency service loader dependency to be expressed? Such that it >>> would >>> be enough to only have OGM depend on ORM (with an inverse service >>> dependency specified). I'm thinking that the OGM module would need to >>> exchange the service dependency information with the ORM module and >>> clear it, when OGM goes away. >> >> MSC has absolutely nothing to do with any of this. Module loading is >> static and is not affected by services in any way. >> >> In your case you probably want a registration system. But be aware that >> if you have more than one OGM implementation, you're not going to be >> able to use the ORM API solely to load them anyway (how would you tell >> it which one you want?); instead you'd need some mechanism to specify, >> so why not use TCCL or explicit class loader specification, as >> appropriate? > > In case it's helpful. The way he handle JAXP is that we have a container > default and we also support TCCL. > > The container default is defined by a special module: > https://github.com/jbossas/jboss-as/blob/master/build/src/main/resources/... > > > > > The rules are > > 1. Use service loader on TCCL (users deployment first), if an impl is > found use that > 2. If not result is find use the default implementation, loaded from the > classloader of the "jaxp-provider" default module. > 3. If the default jaxp-provider module doesn't contain it, we use the > JVM default, whatever it may be. > > The way we implement this relies on installing a false, delegating JAXP > provider. Although in our own code we should define SPIs of some sort. > > So for example, Hibernate OGM could provide a ProviderLocator interface, > which AS would then implement. Our implementation would follow similar > logic as above, although maybe you want slightly different rules. OGM > would ship with a predefined ProviderLocator which would simply search > TCCL. > David/Jason, It might be helpful if we discussed the Hibernate ORM case on IRC (either in #Hibernate-dev or #jboss-as7 (or a private room perhaps). Do you guys have any time this week for that? Scott