[hibernate-dev] [jdf-dev] JBoss Modules dependency vs pain POM dependency + exclusions

[Brian Stansberry]

On 6/7/13 5:14 PM, Sanne Grinovero wrote:
> Great points. Let me try attack some inline :-)
> I'm also adding Brian and Jason in CC in hope they have better
> thoughts and are willing to correct me.
>

Thanks for pointing this out yesterday, Sanne. I didn't see this last 
Friday.

> On 7 June 2013 18:25, Hardy Ferentschik <hardy at hibernate.org> wrote:
>>
>> On 7 Jan 2013, at 5:44 PM, Sanne Grinovero <sanne at hibernate.org> wrote:
>>
>>> On 7 June 2013 16:12, Hardy Ferentschik <hardy at hibernate.org> wrote:
>>>> Again, late to the party, but I agree with Emmanuel. Modules is imo to AS specific in order to encourage it
>>>> as a default/referred configuration approach. I see it more as an expert mode to configure your app server.
>>>
>>> Right we should make it easier and with clear documentation. But
>>> "dropping something in your /modules" isn't sooo different than
>>> dropping an EAR in your /deploy
>>
>> Yes it is and as Gunnar pointed out there might also be more resistance from the ops people to to something like this
>
> Those are very valid concerns. I've personally seen a variety of
> policies, ranging from "let's replace the JBoss jars with latest
> Hibernate snapshot" (done that myself in production)
> to "there's a testing process of 6 months for each configuration
> change you want on the AS; the overall cost of the tests is 2 million,
> so the first step is to file a business case to justify that" (at
> which point I've quit the employer - true story).
> But generally it used to boil down to "who is responsible" and "what
> does the documentation recommend". Which in turn boils down to what we
> can test, and what we write in the documentation.
>
> But if you look at the modules directory in WildFly, the "traditional"
> jars are bundled in a directory starting with:
> /modules/system/layers/base/
>
> There is an explanation about this structure here:
> https://community.jboss.org/wiki/LayeredDistributionsAndModulePathOrganization
>
> I couldn't find a full explanation of what we're expected to see in
> /modules other than "system",

Modules provided by the end user for their own purposes. The "system" 
subdirs are for people who are providing formal distributions that 
install on top of the AS or repackage AS with added stuff.

> or in layers other than "base" but it
> looks like quite self-explanatory that at least power users are
> expected to drop in additional modules.

Sure. For example, users often install JDBC drivers that way.

If you get our libraries via a
> package manager, that might get very easy and especially if we bless
> the process by testing for this configuration and recommend it in the
> documentation, I think ops people could be willing to accept it - I
> guess there is some kind of parallelism historically with the *-DS.xml
> datasource files to be deployed?
>
> I think it's clear that layers are expected to provide such modules,
> and the Hibernate Search project specifically should provide these for
> other projects to consume. You're right that this doesn't need to be
> "the" recommended way for Hibernate Search / ORM users but there are
> at least some benefits, mostly of monitoring and administration
> capabilities, which could make ops people specifically less resistant
> about the new approach.
>
>>> both are easily scripted, we have a not-too-nice solution which works
>>> with Maven, and if it becomes a mainstream strategy for other JBoss
>>> modules then we could certainly make a better looking Maven plugin.
>>
>> See, now we start talking. If we have the right tools we can start talking about this.
>> The current scripts are more of a hack than anything and I would not like people to
>> do the same thing. Also, what is this maven plugin supposed to do? Patch your local
>> AS instance. That's fine for your local build, but how does this help production?
>> What are you going to deliver to the guys running the actual production app? An ear file
>> and a zip with the instructions to please unzip this first here?
>
> Depends on the system. Red Hat users might "just have it there"
> automatically provided because it's actived on their subscription
> account, so the RPM automatically includes it in the application
> server; this has no downsides besides some disk space as it would only
> get activated by deployments specifying the need.
> Fedora users could "yum install" it, while developer workstations
> could get it bundled with other tooling such as Developer Studio. Some
> might need to unzip it, and while I agree that's annoying it's not
> different than unzipping WildFly in the first place, and trivial to
> specify in any build tool, even more as we deploy the zip as Maven
> modules.
> Maybe we could even have a light WildFly download, and a "WildFly with
> extensions".
>

Over the long term we want to evolve WildFly along these lines. There 
will be some engineering meetings in a couple weeks where this will be a 
topic. But it's not a WildFly 8 thing.

>>>> And is not also dangerous to fiddle too much with the modules? Adding new modules might be ok, but often
>>>> you end up updating others as well (talking EL and Bean Validation) and there you don't know whether you
>>>> break another subsystem.
>>>
>>> What fiddling? we don't update EL nor Bean Validation modules, but we
>>> provide solid modules which are tested by our CI and consumed by other
>>> projects (getting more tests done to benefit all).
>>
>> I am not talking Search in this case. It was a general comment against modifying
>> modules directories.
>
> Right, modifying the provided modules dir doesn't feel good.
> I hope the new WildFly modules directory structure addresses this concern?
>

It's intended to formalize how organizations who add on top of the base 
appserver deal with these concerns. "Organizations" in practice meaning 
Red Hat products like SOA-P that layer on top of EAP, or their upstream 
projects. "Layers" can actually override the default WF modules. Doing 
so should be avoided and done with great care. If you create a "layer" 
you're not really WF any more, you're something else. The installation 
has a new "identity" as discussed on that wiki.

An "add-on" cannot override anything and is what we want people to 
gravitate toward. An add-on provides new features, but the underlying 
installation is still WildFly.

>>> Getting more pratical: explain me how I should configure the classpath
>>> for an "embedded lib style" deployment which takes advantage of 2nd
>>> level caching via Infinispan, uses Hibernate Search to store indexes,
>>> and opens a JGroups backend. The dump solution is to include both
>>> Infinispan and JGroups; you'll have to add all dependencies like
>>> JBLogging, JBMarshaller, a provided JTA manager.. you'll get an
>>> app-server class of services but all management extensions will not be
>>> wired up to the AS core, nor Infinispan will be connected to internals
>>> like mod_cluster & al. I'm not even sure if the logger will bind
>>> correctly for dynamic configuration or if it's going to work via a
>>> Log4J bridge which gets intercepted and re-routed to the AS proper
>>> logger .. lots of problems I'd not prepared to face, I bet big time
>>> Tomcat would sound more appealing: no metrics (either can we), no
>>> admin (wouldn't work either), but at least these services don't fight
>>> against each other's resources.
>>
>> Sure, there might be situations where the module approach is better. I was just saying
>> it should not be the "default way of doing things". I trust you that the scenarios
>> you describe are best solved the way you say, but I still think it is a little unfortunate.
>
> I'm not sure it is the best way. It seems it would solve a lot of
> problems so I hope we could go to the bottom of this and verify: the
> ideal solution still needs some changes, for example how the app
> server can share a partial JGroups stack so that we can actually reuse
> it and stay independent (I'll be exploring this with Bela and Paul
> next week). Most other benefits however seem to be reachable for
> Search users by using modules today.
>
> BTW I just figured out the example I made up above was seriously evil,
> as Hibernate would need to interact both with the internal application
> server provided Infinispan and the one included in the application
> libs for index clustering "via" the Search bridge which deals . So
> it's not even about "which approach is better": I don't think it's at
> all possible without deploying custom modules, or without fully
> deactivating the JPA deployer.
>

If this is the case, something is likely incorrectly done in terms of 
the modules in WF. Hibernate core shouldn't need to see Infinispan and 
JGroups in order to provide 2nd level caching. Those modules should be 
dependencies of some other module that provides an impl of the Hibernate 
2LC SPI. That module shouldn't export Infinispan, etc.

Same pattern should apply to any of the standard uses of Infinispan in WF.

>>>> Also it adds additional constrains when deploying the app server and app. It is not just about dropping in a ear/war
>>>> anymore, suddenly I have to "patch" my app server first.
>>>
>>> Right I see your point on this: if it feels like patching it doesn't
>>> look good. But it feels much better if you look at it as "dependency
>>> resolution".
>>
>> How can it feels better by just naming it something else. What counts is that I have to
>> install a default AS instance and then modify/patch its inner configuration.
>
> We always do modify the AS server when deploying an app or configuring
> it in any form, still I never tell the top manager we're going to
> initiate an open hearth surgery on their IT system :D
> Such a terminology would only appropriate in some quite specific
> contexts right? So let's not pick the wrong name: I don't think that
> adding a jar to the place it is designed to be, in an appropriate
> extensions directory, is "patching" nor has much to do with inner
> configuration.
>
>>> I can totally see how a wildfly instance could download
>>> these on-demand from the dependency definition; by having these in
>>> Maven, corporate environments might not dislike it too much as they
>>> could have their own repository managers.
>>
>> Now that is different imo. Now we provide some proper tooling around this.
>> When are you having this ready to go?
>
> I think we could ask for help to make better tooling, if we have a
> case. We won't get tooling to deploy modules if we don't make the
> modules and verify this is a good plan. Ideally I'd like WildFly to
> automatically download extensions, so we can blame the app server for
> "downloading the internet" :)
>

<vague-hand-waving>As mentioned above, this kind of thing is under 
consideration for the long term roadmap.</vague-hand-waving>

>>> BTW this problem is only for JBoss / Wildfly as other app servers
>>> don't bundle Hibernate, so the solution is special purpose as well.
>>
>> So basically we are saying on our app server it is actually harder to get this to work than on
>> others. Interesting. Maybe we should target Glassfish then.
>
> We should target them all, including WildFly even if it needs an
> additional zip :-)
> BTW only WildFly would be able to address the monitoring, Infinispan
> and mod_cluster integrations appropriately, at least until others
> won't expose similar capabilities and modularity.
>
> -- Sanne
>
>>
>> --Hardy
>>


-- 
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat