[forge-dev] shading!?

David M. Lloyd dlloyd at redhat.com
Wed Apr 20 12:23:39 EDT 2011


I think what JBoss Modules can give you is a functional, safe 
multiple-delegate class loader which you can't get with URL class 
loaders.  Specifically, f you want to have a _graph_ of class loaders 
(as opposed to individual flat, merged classloaders) then you need 
something like Modules.

As far as Maven integration, you could write a ModuleLoader which does 
this for you.  You'd have to do version management on your own though. 
Perhaps you could use Aether for the purpose of resolving and fetching 
artifacts, and then use information exposed therefrom to construct the 
module graph.

http://www.sonatype.com/people/2010/08/introducing-aether/

Your application does not have to be modular to use JBoss Modules. 
Though of course in my opinion that makes it much nicer. :-)

On 04/20/2011 10:56 AM, Lincoln Baxter, III wrote:
> I agree with both of you guys, which is why I am soliciting ideas for
> this now.
>
> I can also say that we do not have one big Classpath right now (unless I
> am horribly mistaken.) Forge already supports classpath isolation of
> individual plugins, so we're already half-way there. If my understanding
> of classloaders is at all right, It would not be too much work to
> implement a maven resolver and additional level of Composite
> classloaders to achieve this goal.
>
> I don't believe we need to pull in a heavyweight module system like
> OSGI, or force developers to do even the simple configuration for
> something like JBoss modules (which afaik, does not have a way of
> packaging the actual dependencies with the module, or calling out to
> Maven for dependencies - which forge can already do.)
>
> *This is what we have right now:* (see today.png)
>
> Notice how the Plugin ClassLoaders may reference Forge APIs, but the
> plugins may not access each other's classloaders because there is no
> link from Forge APIs to the Plugins.
>
> *And this is all I think it should take to do what I am suggesting:*
> (see tomorrow.png)
>
> If it is technically possible to achieve with JBoss modules, via adding
> in parsing the POM and including dependencies automatically, then I am
> all for it, but from what I understand, this would not really be a 1-1
> mapping and will not be trivial. I know my understanding of ClassLoaders
> may be a bit Naive, but I don't think that we need to use JBoss modules
> (or OSGi) to achieve this goal of automatic downloading and loading of
> plugin dependency JARs.
>
> We certainly do not need the ability (yet) to drop an individual module
> while Forge is running, because Weld does not support this kind of
> disruption in the BeanStore. If you remove a JAR while Weld is running,
> you get a big fiery explosion. Same for loading a single JAR, you have
> to restart Weld because it will not be aware of the changes, and there
> is not a good API to make it so. It still has to build the bean graph
> all over again. That is one of the big reasons to use a fully-fledged
> module system, and we don't have that requirement. Forge simply dumps
> all classloaders (until JDK 7 we can't close them easily, yeah memory
> leaks, whatever, not really important right now,) then builds a new
> network of ClassLoaders and starts over.
>
> However, if we can extend JBoss modules with Maven support in this way,
> and it's something that David would be interested in, then I think
> that's one big reason to do it that way. Until then, if we can do it
> quickly my way, I'd like to get it working sooner rather than later.
> Please please tell me (I'm sure I don't even have to ask) if there are
> gaping technical holes in what I am proposing, but if there are not,
> then I'm going to stick to this position until someone bursts my bubble.
>
> ~Lincoln
>
>
> On Wed, Apr 20, 2011 at 9:33 AM, Paul Bakker <paul.bakker.nl
> <http://paul.bakker.nl>@gmail.com <http://gmail.com>> wrote:
>
>     Yeah, I'm afraid that's right. I think the module system integration
>     should get very high priority before we reach a 1.0 version. It's
>     something that affects the whole way the core is built up, and it
>     will be very difficult to change that later on.
>
>     Paul
>
>
>
>
>     On Wed, Apr 20, 2011 at 3:17 PM, Max Rydahl Andersen
>     <max.andersen at redhat.com <mailto:max.andersen at redhat.com>> wrote:
>
>         +1
>
>         I agree with what you want to do Lincoln, but don't see how that
>         can be made to happen without a module system.
>
>         Until then everything just will have to live in one big classpath.
>
>         /max
>
>         On Apr 20, 2011, at 14:24, Paul Bakker wrote:
>
>          > And that's why a module system is needed. If people can just
>         add dependencies there will be duplicate (possibly multiple
>         versions) dependencies. Each plugin should be in a separate
>         classloader, and each dependency should be too so that a plugin
>         never breaks other plugins or Forge itself. It would be a big
>         limitation if plugins can't use libraries otherwise.
>          >
>          > Paul
>          >
>          > On Tue, Apr 19, 2011 at 6:14 PM, Lincoln Baxter, III
>         <lincolnbaxter at gmail.com <mailto:lincolnbaxter at gmail.com>> wrote:
>          > That's right, but what I'm saying is that I don't want
>         developers to be responsible for anything but *their code* -- if
>         they have dependencies, those depenencies will be fetched for
>         them (or somehow bundled in the JAR file itself, which is
>         certainly possible, however not my preference.)
>          >
>          > If you could write a plugin, reference dependencies in your
>         POM, and have everything *Just Work* don't you think that would
>         be a friendlier experience?
>          >
>          > ~Lincoln
>          >
>          >
>          > On Mon, Apr 18, 2011 at 5:55 PM, Max Andersen
>         <manderse at redhat.com <mailto:manderse at redhat.com>> wrote:
>          > But if your plugin uses multiple jars it is not one jar.
>          >
>          >
>          > /max (sent from my phone)
>          >
>          >
>          > On 18/04/2011, at 19.06, "Lincoln Baxter, III"
>         <lincolnbaxter at gmail.com <mailto:lincolnbaxter at gmail.com>> wrote:
>          >
>          >> That doesn't solve the problem of having to drop jar files
>         onto the classpath in order for plugins to work. I want one JAR
>         per plugin.
>          >>
>          >> ~Lincoln
>          >>
>          >> On Mon, Apr 18, 2011 at 12:59 PM, Max Rydahl Andersen
>         <max.andersen at redhat.com <mailto:max.andersen at redhat.com>> wrote:
>          >>
>          >> On Apr 18, 2011, at 18:57, Lincoln Baxter, III wrote:
>          >>
>          >> > I don't want to force plugin-developers to create modules
>         for every dependency that their plugin requires. That's why I've
>         been avoiding OSGI or JBoss Modules.
>          >>
>          >> But then you shouldn't be forcing them to shade either - you
>         should just have one global classloader for the plugins then.
>          >>
>          >> /max
>          >>
>          >> >
>          >> > ~Lincoln
>          >> >
>          >> > On Mon, Apr 18, 2011 at 12:37 PM, Max Rydahl Andersen
>         <max.andersen at redhat.com <mailto:max.andersen at redhat.com>> wrote:
>          >> >
>          >> > > What do you think about using Maven APIs to inspect the
>         POM and fetch dependencies dynamically for each plugin, then
>         isolate them in the plugin's classloader?
>          >> >
>          >> > Why not just load them in to one classloader so you don't
>         have collisions when there are mixed dependencies on Forge it self ?
>          >> >
>          >> > How about shared data instances ? How does that work ?
>          >> >
>          >> > ...as a side note...creating our own module system now - I
>         feel that is a very bad direction :(
>          >> > Might as well adopt osgi plugin system if you want this
>         kind of separation ?
>          >> >
>          >> > /max
>          >> >
>          >> > >
>          >> > > ~Lincoln
>          >> > >
>          >> > > On Mon, Apr 18, 2011 at 11:56 AM, Max Rydahl Andersen
>         <max.andersen at redhat.com <mailto:max.andersen at redhat.com>> wrote:
>          >> > >
>          >> > > On Apr 18, 2011, at 15:15, Lincoln Baxter, III wrote:
>          >> > >
>          >> > > > "if there is a standard location for dependencies"
>          >> > > >
>          >> > > > What do you mean?
>          >> > >
>          >> > > Your "standard" for shading is that you put all classes
>         into the plugin.jar.
>          >> > >
>          >> > > A "standard" for dependencies for a plugin.jar could be
>         "next to the plugin.jar".
>          >> > >
>          >> > > Would still have the problem of overlapping jars but
>         then at least its easier to see where the duplication is.
>          >> > >
>          >> > > /max
>          >> > >
>          >> > > >
>          >> > > > Thx,
>          >> > > > ~Lincoln
>          >> > > >
>          >> > > > On Mon, Apr 18, 2011 at 6:20 AM, Max Rydahl Andersen
>         <max.andersen at redhat.com <mailto:max.andersen at redhat.com>> wrote:
>          >> > > >
>          >> > > > > I was thinking we might already be able to do that
>         using the existing pom.xml metadata that's stored in the
>         artifact itself, or is that too tricky?
>          >> > > >
>          >> > > > if there is a standard location for dependencies then
>         it should be fine - at least better than shading ;)
>          >> > > >
>          >> > > > /max
>          >> > > >
>          >> > > > >
>          >> > > > > On Mon, Apr 11, 2011 at 6:51 PM, Max Andersen
>         <manderse at redhat.com <mailto:manderse at redhat.com>> wrote:
>          >> > > > > I was thinking Plugin jar having references to
>         dependent jars via manifest.mf
>          >> > > > >
>          >> > > > > /max (sent from my phone)
>          >> > > > >
>          >> > > > >
>          >> > > > > On 12/04/2011, at 00.39, "Lincoln Baxter, III"
>         <lincolnbaxter at gmail.com <mailto:lincolnbaxter at gmail.com>> wrote:
>          >> > > > >
>          >> > > > >> Can you give an example of how you would bundle the
>         JARs? (Just put them in /META-INF/dependencies/ ... ?) And would
>         that not cause just as many class conflicts? If you
>         shade/relocate then the deps *should be* completely isolated.
>          >> > > > >>
>          >> > > > >> On Mon, Apr 11, 2011 at 4:11 PM, Max Rydahl
>         Andersen <max.andersen at redhat.com
>         <mailto:max.andersen at redhat.com>> wrote:
>          >> > > > >> well, recommending just bundling jars would be a
>         better approach than shading IMO.
>          >> > > > >>
>          >> > > > >> /max
>          >> > > > >>
>          >> > > > >> On Apr 11, 2011, at 16:00, Lincoln Baxter, III wrote:
>          >> > > > >>
>          >> > > > >> > Yeah, shading is currently the recommended
>         approach. Conflicts should be avoided by using relocations. I
>         know this is... not a great method, but for now it's all we've
>         got. Open to suggestions.
>          >> > > > >> >
>          >> > > > >> > ~Lincoln
>          >> > > > >> >
>          >> > > > >> > On Mon, Apr 11, 2011 at 3:41 AM, Max Rydahl
>         Andersen <max.andersen at redhat.com
>         <mailto:max.andersen at redhat.com>> wrote:
>          >> > > > >> > Heya,
>          >> > > > >> >
>          >> > > > >> > Lincoln, I just saw your commits to hibernattools
>         plugin at
>         (https://github.com/forge/plugin-hibernate-tools/commit/8b208b4a8e79dbb8a01d10d266ee81afd2cf7106)
>          >> > > > >> >
>          >> > > > >> > Is shading of jars really the recommended
>         approach for plugins in Forge ?
>          >> > > > >> >
>          >> > > > >> > How are you going to share/avoid collisions of
>         libraries across plugins if they need to bundle via shading ?
>          >> > > > >> >
>          >> > > > >> > /max
>          >> > > > >> > http://about.me/maxandersen
>          >> > > > >> >
>          >> > > > >> >
>          >> > > > >> >
>          >> > > > >> >
>          >> > > > >> > _______________________________________________
>          >> > > > >> > forge-dev mailing list
>          >> > > > >> > forge-dev at lists.jboss.org
>         <mailto:forge-dev at lists.jboss.org>
>          >> > > > >> > https://lists.jboss.org/mailman/listinfo/forge-dev
>          >> > > > >> >
>          >> > > > >> >
>          >> > > > >> >
>          >> > > > >> > --
>          >> > > > >> > Lincoln Baxter, III
>          >> > > > >> > http://ocpsoft.com
>          >> > > > >> > http://scrumshark.com
>          >> > > > >> > "Keep it Simple"
>          >> > > > >>
>          >> > > > >> /max
>          >> > > > >> http://about.me/maxandersen
>          >> > > > >>
>          >> > > > >>
>          >> > > > >>
>          >> > > > >>
>          >> > > > >>
>          >> > > > >>
>          >> > > > >> --
>          >> > > > >> Lincoln Baxter, III
>          >> > > > >> http://ocpsoft.com
>          >> > > > >> http://scrumshark.com
>          >> > > > >> "Keep it Simple"
>          >> > > > >
>          >> > > > >
>          >> > > > >
>          >> > > > > --
>          >> > > > > Lincoln Baxter, III
>          >> > > > > http://ocpsoft.com
>          >> > > > > http://scrumshark.com
>          >> > > > > "Keep it Simple"
>          >> > > >
>          >> > > > /max
>          >> > > > http://about.me/maxandersen
>          >> > > >
>          >> > > >
>          >> > > >
>          >> > > >
>          >> > > >
>          >> > > >
>          >> > > > --
>          >> > > > Lincoln Baxter, III
>          >> > > > http://ocpsoft.com
>          >> > > > http://scrumshark.com
>          >> > > > "Keep it Simple"
>          >> > >
>          >> > > /max
>          >> > > http://about.me/maxandersen
>          >> > >
>          >> > >
>          >> > >
>          >> > >
>          >> > >
>          >> > >
>          >> > > --
>          >> > > Lincoln Baxter, III
>          >> > > http://ocpsoft.com
>          >> > > http://scrumshark.com
>          >> > > "Keep it Simple"
>          >> >
>          >> > /max
>          >> > http://about.me/maxandersen
>          >> >
>          >> >
>          >> >
>          >> >
>          >> >
>          >> >
>          >> > --
>          >> > Lincoln Baxter, III
>          >> > http://ocpsoft.com
>          >> > http://scrumshark.com
>          >> > "Keep it Simple"
>          >>
>          >> /max
>          >> http://about.me/maxandersen
>          >>
>          >>
>          >>
>          >>
>          >>
>          >>
>          >> --
>          >> Lincoln Baxter, III
>          >> http://ocpsoft.com
>          >> http://scrumshark.com
>          >> "Keep it Simple"
>          >
>          >
>          >
>          > --
>          > Lincoln Baxter, III
>          > http://ocpsoft.com
>          > http://scrumshark.com
>          > "Keep it Simple"
>          >
>          > _______________________________________________
>          > forge-dev mailing list
>          > forge-dev at lists.jboss.org <mailto:forge-dev at lists.jboss.org>
>          > https://lists.jboss.org/mailman/listinfo/forge-dev
>          >
>          >
>          > _______________________________________________
>          > forge-dev mailing list
>          > forge-dev at lists.jboss.org <mailto:forge-dev at lists.jboss.org>
>          > https://lists.jboss.org/mailman/listinfo/forge-dev
>
>         /max
>         http://about.me/maxandersen
>
>
>
>
>         _______________________________________________
>         forge-dev mailing list
>         forge-dev at lists.jboss.org <mailto:forge-dev at lists.jboss.org>
>         https://lists.jboss.org/mailman/listinfo/forge-dev
>
>
>
>     _______________________________________________
>     forge-dev mailing list
>     forge-dev at lists.jboss.org <mailto:forge-dev at lists.jboss.org>
>     https://lists.jboss.org/mailman/listinfo/forge-dev
>
>
>
>
> --
> Lincoln Baxter, III
> http://ocpsoft.com
> http://scrumshark.com
> "Keep it Simple"


-- 
- DML


More information about the forge-dev mailing list