[forge-dev] shading!?

Wed Apr 20 11:56:10 EDT 2011

I agree with both of you guys, which is why I am soliciting ideas for this
now.

I can also say that we do not have one big Classpath right now (unless I am
horribly mistaken.) Forge already supports classpath isolation of individual
plugins, so we're already half-way there. If my understanding of
classloaders is at all right, It would not be too much work to implement a
maven resolver and additional level of Composite classloaders to achieve
this goal.

I don't believe we need to pull in a heavyweight module system like OSGI, or
force developers to do even the simple configuration for something like
JBoss modules (which afaik, does not have a way of packaging the actual
dependencies with the module, or calling out to Maven for dependencies -
which forge can already do.)

*This is what we have right now:* (see today.png)

Notice how the Plugin ClassLoaders may reference Forge APIs, but the plugins
may not access each other's classloaders because there is no link from Forge
APIs to the Plugins.

*And this is all I think it should take to do what I am suggesting:* (see
tomorrow.png)

If it is technically possible to achieve with JBoss modules, via adding in
parsing the POM and including dependencies automatically, then I am all for
it, but from what I understand, this would not really be a 1-1 mapping and
will not be trivial. I know my understanding of ClassLoaders may be a bit
Naive, but I don't think that we need to use JBoss modules (or OSGi) to
achieve this goal of automatic downloading and loading of plugin dependency
JARs.

We certainly do not need the ability (yet) to drop an individual module
while Forge is running, because Weld does not support this kind of
disruption in the BeanStore. If you remove a JAR while Weld is running, you
get a big fiery explosion. Same for loading a single JAR, you have to
restart Weld because it will not be aware of the changes, and there is not a
good API to make it so. It still has to build the bean graph all over again.
That is one of the big reasons to use a fully-fledged module system, and we
don't have that requirement. Forge simply dumps all classloaders (until JDK
7 we can't close them easily, yeah memory leaks, whatever, not really
important right now,) then builds a new network of ClassLoaders and starts
over.

However, if we can extend JBoss modules with Maven support in this way, and
it's something that David would be interested in, then I think that's one
big reason to do it that way. Until then, if we can do it quickly my way,
I'd like to get it working sooner rather than later. Please please tell me
(I'm sure I don't even have to ask) if there are gaping technical holes in
what I am proposing, but if there are not, then I'm going to stick to this
position until someone bursts my bubble.

~Lincoln

On Wed, Apr 20, 2011 at 9:33 AM, Paul Bakker <paul.bakker.nl at gmail.com>wrote:

> Yeah, I'm afraid that's right. I think the module system integration should
> get very high priority before we reach a 1.0 version. It's something that
> affects the whole way the core is built up, and it will be very difficult to
> change that later on.
>
> Paul
>
>
>
>
> On Wed, Apr 20, 2011 at 3:17 PM, Max Rydahl Andersen <
> max.andersen at redhat.com> wrote:
>
>> +1
>>
>> I agree with what you want to do Lincoln, but don't see how that can be
>> made to happen without a module system.
>>
>> Until then everything just will have to live in one big classpath.
>>
>> /max
>>
>> On Apr 20, 2011, at 14:24, Paul Bakker wrote:
>>
>> > And that's why a module system is needed. If people can just add
>> dependencies there will be duplicate (possibly multiple versions)
>> dependencies. Each plugin should be in a separate classloader, and each
>> dependency should be too so that a plugin never breaks other plugins or
>> Forge itself. It would be a big limitation if plugins can't use libraries
>> otherwise.
>> >
>> > Paul
>> >
>> > On Tue, Apr 19, 2011 at 6:14 PM, Lincoln Baxter, III <
>> lincolnbaxter at gmail.com> wrote:
>> > That's right, but what I'm saying is that I don't want developers to be
>> responsible for anything but *their code* -- if they have dependencies,
>> those depenencies will be fetched for them (or somehow bundled in the JAR
>> file itself, which is certainly possible, however not my preference.)
>> >
>> > If you could write a plugin, reference dependencies in your POM, and
>> have everything *Just Work* don't you think that would be a friendlier
>> experience?
>> >
>> > ~Lincoln
>> >
>> >
>> > On Mon, Apr 18, 2011 at 5:55 PM, Max Andersen <manderse at redhat.com>
>> wrote:
>> > But if your plugin uses multiple jars it is not one jar.
>> >
>> >
>> > /max (sent from my phone)
>> >
>> >
>> > On 18/04/2011, at 19.06, "Lincoln Baxter, III" <lincolnbaxter at gmail.com>
>> wrote:
>> >
>> >> That doesn't solve the problem of having to drop jar files onto the
>> classpath in order for plugins to work. I want one JAR per plugin.
>> >>
>> >> ~Lincoln
>> >>
>> >> On Mon, Apr 18, 2011 at 12:59 PM, Max Rydahl Andersen <
>> max.andersen at redhat.com> wrote:
>> >>
>> >> On Apr 18, 2011, at 18:57, Lincoln Baxter, III wrote:
>> >>
>> >> > I don't want to force plugin-developers to create modules for every
>> dependency that their plugin requires. That's why I've been avoiding OSGI or
>> JBoss Modules.
>> >>
>> >> But then you shouldn't be forcing them to shade either - you should
>> just have one global classloader for the plugins then.
>> >>
>> >> /max
>> >>
>> >> >
>> >> > ~Lincoln
>> >> >
>> >> > On Mon, Apr 18, 2011 at 12:37 PM, Max Rydahl Andersen <
>> max.andersen at redhat.com> wrote:
>> >> >
>> >> > > What do you think about using Maven APIs to inspect the POM and
>> fetch dependencies dynamically for each plugin, then isolate them in the
>> plugin's classloader?
>> >> >
>> >> > Why not just load them in to one classloader so you don't have
>> collisions when there are mixed dependencies on Forge it self ?
>> >> >
>> >> > How about shared data instances ? How does that work ?
>> >> >
>> >> > ...as a side note...creating our own module system now - I feel that
>> is a very bad direction :(
>> >> > Might as well adopt osgi plugin system if you want this kind of
>> separation ?
>> >> >
>> >> > /max
>> >> >
>> >> > >
>> >> > > ~Lincoln
>> >> > >
>> >> > > On Mon, Apr 18, 2011 at 11:56 AM, Max Rydahl Andersen <
>> max.andersen at redhat.com> wrote:
>> >> > >
>> >> > > On Apr 18, 2011, at 15:15, Lincoln Baxter, III wrote:
>> >> > >
>> >> > > > "if there is a standard location for dependencies"
>> >> > > >
>> >> > > > What do you mean?
>> >> > >
>> >> > > Your "standard" for shading is that you put all classes into the
>> plugin.jar.
>> >> > >
>> >> > > A "standard" for dependencies for a plugin.jar could be "next to
>> the plugin.jar".
>> >> > >
>> >> > > Would still have the problem of overlapping jars but then at least
>> its easier to see where the duplication is.
>> >> > >
>> >> > > /max
>> >> > >
>> >> > > >
>> >> > > > Thx,
>> >> > > > ~Lincoln
>> >> > > >
>> >> > > > On Mon, Apr 18, 2011 at 6:20 AM, Max Rydahl Andersen <
>> max.andersen at redhat.com> wrote:
>> >> > > >
>> >> > > > > I was thinking we might already be able to do that using the
>> existing pom.xml metadata that's stored in the artifact itself, or is that
>> too tricky?
>> >> > > >
>> >> > > > if there is a standard location for dependencies then it should
>> be fine - at least better than shading ;)
>> >> > > >
>> >> > > > /max
>> >> > > >
>> >> > > > >
>> >> > > > > On Mon, Apr 11, 2011 at 6:51 PM, Max Andersen <
>> manderse at redhat.com> wrote:
>> >> > > > > I was thinking Plugin jar having references to dependent jars
>> via manifest.mf
>> >> > > > >
>> >> > > > > /max (sent from my phone)
>> >> > > > >
>> >> > > > >
>> >> > > > > On 12/04/2011, at 00.39, "Lincoln Baxter, III" <
>> lincolnbaxter at gmail.com> wrote:
>> >> > > > >
>> >> > > > >> Can you give an example of how you would bundle the JARs?
>> (Just put them in /META-INF/dependencies/ ... ?) And would that not cause
>> just as many class conflicts? If you shade/relocate then the deps *should
>> be* completely isolated.
>> >> > > > >>
>> >> > > > >> On Mon, Apr 11, 2011 at 4:11 PM, Max Rydahl Andersen <
>> max.andersen at redhat.com> wrote:
>> >> > > > >> well, recommending just bundling jars would be a better
>> approach than shading IMO.
>> >> > > > >>
>> >> > > > >> /max
>> >> > > > >>
>> >> > > > >> On Apr 11, 2011, at 16:00, Lincoln Baxter, III wrote:
>> >> > > > >>
>> >> > > > >> > Yeah, shading is currently the recommended approach.
>> Conflicts should be avoided by using relocations. I know this is... not a
>> great method, but for now it's all we've got. Open to suggestions.
>> >> > > > >> >
>> >> > > > >> > ~Lincoln
>> >> > > > >> >
>> >> > > > >> > On Mon, Apr 11, 2011 at 3:41 AM, Max Rydahl Andersen <
>> max.andersen at redhat.com> wrote:
>> >> > > > >> > Heya,
>> >> > > > >> >
>> >> > > > >> > Lincoln, I just saw your commits to hibernattools plugin at
>> (
>> https://github.com/forge/plugin-hibernate-tools/commit/8b208b4a8e79dbb8a01d10d266ee81afd2cf7106
>> )
>> >> > > > >> >
>> >> > > > >> > Is shading of jars really the recommended approach for
>> plugins in Forge ?
>> >> > > > >> >
>> >> > > > >> > How are you going to share/avoid collisions of libraries
>> across plugins if they need to bundle via shading ?
>> >> > > > >> >
>> >> > > > >> > /max
>> >> > > > >> > http://about.me/maxandersen
>> >> > > > >> >
>> >> > > > >> >
>> >> > > > >> >
>> >> > > > >> >
>> >> > > > >> > _______________________________________________
>> >> > > > >> > forge-dev mailing list
>> >> > > > >> > forge-dev at lists.jboss.org
>> >> > > > >> > https://lists.jboss.org/mailman/listinfo/forge-dev
>> >> > > > >> >
>> >> > > > >> >
>> >> > > > >> >
>> >> > > > >> > --
>> >> > > > >> > Lincoln Baxter, III
>> >> > > > >> > http://ocpsoft.com
>> >> > > > >> > http://scrumshark.com
>> >> > > > >> > "Keep it Simple"
>> >> > > > >>
>> >> > > > >> /max
>> >> > > > >> http://about.me/maxandersen
>> >> > > > >>
>> >> > > > >>
>> >> > > > >>
>> >> > > > >>
>> >> > > > >>
>> >> > > > >>
>> >> > > > >> --
>> >> > > > >> Lincoln Baxter, III
>> >> > > > >> http://ocpsoft.com
>> >> > > > >> http://scrumshark.com
>> >> > > > >> "Keep it Simple"
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > > --
>> >> > > > > Lincoln Baxter, III
>> >> > > > > http://ocpsoft.com
>> >> > > > > http://scrumshark.com
>> >> > > > > "Keep it Simple"
>> >> > > >
>> >> > > > /max
>> >> > > > http://about.me/maxandersen
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > > --
>> >> > > > Lincoln Baxter, III
>> >> > > > http://ocpsoft.com
>> >> > > > http://scrumshark.com
>> >> > > > "Keep it Simple"
>> >> > >
>> >> > > /max
>> >> > > http://about.me/maxandersen
>> >> > >
>> >> > >
>> >> > >
>> >> > >
>> >> > >
>> >> > >
>> >> > > --
>> >> > > Lincoln Baxter, III
>> >> > > http://ocpsoft.com
>> >> > > http://scrumshark.com
>> >> > > "Keep it Simple"
>> >> >
>> >> > /max
>> >> > http://about.me/maxandersen
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Lincoln Baxter, III
>> >> > http://ocpsoft.com
>> >> > http://scrumshark.com
>> >> > "Keep it Simple"
>> >>
>> >> /max
>> >> http://about.me/maxandersen
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Lincoln Baxter, III
>> >> http://ocpsoft.com
>> >> http://scrumshark.com
>> >> "Keep it Simple"
>> >
>> >
>> >
>> > --
>> > Lincoln Baxter, III
>> > http://ocpsoft.com
>> > http://scrumshark.com
>> > "Keep it Simple"
>> >
>> > _______________________________________________
>> > forge-dev mailing list
>> > forge-dev at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/forge-dev
>> >
>> >
>> > _______________________________________________
>> > forge-dev mailing list
>> > forge-dev at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/forge-dev
>>
>> /max
>> http://about.me/maxandersen
>>
>>
>>
>>
>> _______________________________________________
>> forge-dev mailing list
>> forge-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/forge-dev
>>
>
>
> _______________________________________________
> forge-dev mailing list
> forge-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/forge-dev
>
>

-- 
Lincoln Baxter, III
http://ocpsoft.com
http://scrumshark.com
"Keep it Simple"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/forge-dev/attachments/20110420/7eff5724/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: today.png
Type: image/png
Size: 79351 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/forge-dev/attachments/20110420/7eff5724/attachment-0002.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tomorrow.png
Type: image/png
Size: 89625 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/forge-dev/attachments/20110420/7eff5724/attachment-0003.png 