Re: [forge-dev] shading!?

I agree with both of you guys, which is why I am soliciting ideas for this now. I can also say that we do not have one big Classpath right now (unless I am horribly mistaken.) Forge already supports classpath isolation of individual plugins, so we're already half-way there. If my understanding of classloaders is at all right, It would not be too much work to implement a maven resolver and additional level of Composite classloaders to achieve this goal. I don't believe we need to pull in a heavyweight module system like OSGI, or force developers to do even the simple configuration for something like JBoss modules (which afaik, does not have a way of packaging the actual dependencies with the module, or calling out to Maven for dependencies - which forge can already do.) *This is what we have right now:* (see today.png) Notice how the Plugin ClassLoaders may reference Forge APIs, but the plugins may not access each other's classloaders because there is no link from Forge APIs to the Plugins. *And this is all I think it should take to do what I am suggesting:* (see tomorrow.png) If it is technically possible to achieve with JBoss modules, via adding in parsing the POM and including dependencies automatically, then I am all for it, but from what I understand, this would not really be a 1-1 mapping and will not be trivial. I know my understanding of ClassLoaders may be a bit Naive, but I don't think that we need to use JBoss modules (or OSGi) to achieve this goal of automatic downloading and loading of plugin dependency JARs. We certainly do not need the ability (yet) to drop an individual module while Forge is running, because Weld does not support this kind of disruption in the BeanStore. If you remove a JAR while Weld is running, you get a big fiery explosion. Same for loading a single JAR, you have to restart Weld because it will not be aware of the changes, and there is not a good API to make it so. It still has to build the bean graph all over again. That is one of the big reasons to use a fully-fledged module system, and we don't have that requirement. Forge simply dumps all classloaders (until JDK 7 we can't close them easily, yeah memory leaks, whatever, not really important right now,) then builds a new network of ClassLoaders and starts over. However, if we can extend JBoss modules with Maven support in this way, and it's something that David would be interested in, then I think that's one big reason to do it that way. Until then, if we can do it quickly my way, I'd like to get it working sooner rather than later. Please please tell me (I'm sure I don't even have to ask) if there are gaping technical holes in what I am proposing, but if there are not, then I'm going to stick to this position until someone bursts my bubble. ~Lincoln On Wed, Apr 20, 2011 at 9:33 AM, Paul Bakker <paul.bakker.nl(a)gmail.com&gt;wrote: > Yeah, I'm afraid that's right. I think the module system integration > should get very high priority before we reach a 1.0 version. It's something > that affects the whole way the core is built up, and it will be very > difficult to change that later on. > > Paul > > > > > On Wed, Apr 20, 2011 at 3:17 PM, Max Rydahl Andersen < > max.andersen(a)redhat.com&gt; wrote: > >> +1 >> >> I agree with what you want to do Lincoln, but don't see how that can be >> made to happen without a module system. >> >> Until then everything just will have to live in one big classpath. >> >> /max >> >> On Apr 20, 2011, at 14:24, Paul Bakker wrote: >> >> > And that's why a module system is needed. If people can just add >> dependencies there will be duplicate (possibly multiple versions) >> dependencies. Each plugin should be in a separate classloader, and each >> dependency should be too so that a plugin never breaks other plugins or >> Forge itself. It would be a big limitation if plugins can't use libraries >> otherwise. >> > >> > Paul >> > >> > On Tue, Apr 19, 2011 at 6:14 PM, Lincoln Baxter, III < >> lincolnbaxter(a)gmail.com&gt; wrote: >> > That's right, but what I'm saying is that I don't want developers to be >> responsible for anything but *their code* -- if they have dependencies, >> those depenencies will be fetched for them (or somehow bundled in the JAR >> file itself, which is certainly possible, however not my preference.) >> > >> > If you could write a plugin, reference dependencies in your POM, and >> have everything *Just Work* don't you think that would be a friendlier >> experience? >> > >> > ~Lincoln >> > >> > >> > On Mon, Apr 18, 2011 at 5:55 PM, Max Andersen <manderse(a)redhat.com&gt; >> wrote: >> > But if your plugin uses multiple jars it is not one jar. >> > >> > >> > /max (sent from my phone) >> > >> > >> > On 18/04/2011, at 19.06, "Lincoln Baxter, III" < >> lincolnbaxter(a)gmail.com&gt; wrote: >> > >> >> That doesn't solve the problem of having to drop jar files onto the >> classpath in order for plugins to work. I want one JAR per plugin. >> >> >> >> ~Lincoln >> >> >> >> On Mon, Apr 18, 2011 at 12:59 PM, Max Rydahl Andersen < >> max.andersen(a)redhat.com&gt; wrote: >> >> >> >> On Apr 18, 2011, at 18:57, Lincoln Baxter, III wrote: >> >> >> >> > I don't want to force plugin-developers to create modules for every >> dependency that their plugin requires. That's why I've been avoiding OSGI or >> JBoss Modules. >> >> >> >> But then you shouldn't be forcing them to shade either - you should >> just have one global classloader for the plugins then. >> >> >> >> /max >> >> >> >> > >> >> > ~Lincoln >> >> > >> >> > On Mon, Apr 18, 2011 at 12:37 PM, Max Rydahl Andersen < >> max.andersen(a)redhat.com&gt; wrote: >> >> > >> >> > > What do you think about using Maven APIs to inspect the POM and >> fetch dependencies dynamically for each plugin, then isolate them in the >> plugin's classloader? >> >> > >> >> > Why not just load them in to one classloader so you don't have >> collisions when there are mixed dependencies on Forge it self ? >> >> > >> >> > How about shared data instances ? How does that work ? >> >> > >> >> > ...as a side note...creating our own module system now - I feel that >> is a very bad direction :( >> >> > Might as well adopt osgi plugin system if you want this kind of >> separation ? >> >> > >> >> > /max >> >> > >> >> > > >> >> > > ~Lincoln >> >> > > >> >> > > On Mon, Apr 18, 2011 at 11:56 AM, Max Rydahl Andersen < >> max.andersen(a)redhat.com&gt; wrote: >> >> > > >> >> > > On Apr 18, 2011, at 15:15, Lincoln Baxter, III wrote: >> >> > > >> >> > > > "if there is a standard location for dependencies" >> >> > > > >> >> > > > What do you mean? >> >> > > >> >> > > Your "standard" for shading is that you put all classes into the >> plugin.jar. >> >> > > >> >> > > A "standard" for dependencies for a plugin.jar could be "next to >> the plugin.jar". >> >> > > >> >> > > Would still have the problem of overlapping jars but then at least >> its easier to see where the duplication is. >> >> > > >> >> > > /max >> >> > > >> >> > > > >> >> > > > Thx, >> >> > > > ~Lincoln >> >> > > > >> >> > > > On Mon, Apr 18, 2011 at 6:20 AM, Max Rydahl Andersen < >> max.andersen(a)redhat.com&gt; wrote: >> >> > > > >> >> > > > > I was thinking we might already be able to do that using the >> existing pom.xml metadata that's stored in the artifact itself, or is that >> too tricky? >> >> > > > >> >> > > > if there is a standard location for dependencies then it should >> be fine - at least better than shading ;) >> >> > > > >> >> > > > /max >> >> > > > >> >> > > > > >> >> > > > > On Mon, Apr 11, 2011 at 6:51 PM, Max Andersen < >> manderse(a)redhat.com&gt; wrote: >> >> > > > > I was thinking Plugin jar having references to dependent jars >> via manifest.mf >> >> > > > > >> >> > > > > /max (sent from my phone) >> >> > > > > >> >> > > > > >> >> > > > > On 12/04/2011, at 00.39, "Lincoln Baxter, III" < >> lincolnbaxter(a)gmail.com&gt; wrote: >> >> > > > > >> >> > > > >> Can you give an example of how you would bundle the JARs? >> (Just put them in /META-INF/dependencies/ ... ?) And would that not cause >> just as many class conflicts? If you shade/relocate then the deps *should >> be* completely isolated. >> >> > > > >> >> >> > > > >> On Mon, Apr 11, 2011 at 4:11 PM, Max Rydahl Andersen < >> max.andersen(a)redhat.com&gt; wrote: >> >> > > > >> well, recommending just bundling jars would be a better >> approach than shading IMO. >> >> > > > >> >> >> > > > >> /max >> >> > > > >> >> >> > > > >> On Apr 11, 2011, at 16:00, Lincoln Baxter, III wrote: >> >> > > > >> >> >> > > > >> > Yeah, shading is currently the recommended approach. >> Conflicts should be avoided by using relocations. I know this is... not a >> great method, but for now it's all we've got. Open to suggestions. >> >> > > > >> > >> >> > > > >> > ~Lincoln >> >> > > > >> > >> >> > > > >> > On Mon, Apr 11, 2011 at 3:41 AM, Max Rydahl Andersen < >> max.andersen(a)redhat.com&gt; wrote: >> >> > > > >> > Heya, >> >> > > > >> > >> >> > > > >> > Lincoln, I just saw your commits to hibernattools plugin at >> ( >> https://github.com/forge/plugin-hibernate-tools/commit/8b208b4a8e79dbb8a0... >> ) >> >> > > > >> > >> >> > > > >> > Is shading of jars really the recommended approach for >> plugins in Forge ? >> >> > > > >> > >> >> > > > >> > How are you going to share/avoid collisions of libraries >> across plugins if they need to bundle via shading ? >> >> > > > >> > >> >> > > > >> > /max >> >> > > > >> > http://about.me/maxandersen >> >> > > > >> > >> >> > > > >> > >> >> > > > >> > >> >> > > > >> > >> >> > > > >> > _______________________________________________ >> >> > > > >> > forge-dev mailing list >> >> > > > >> > forge-dev(a)lists.jboss.org >> >> > > > >> > https://lists.jboss.org/mailman/listinfo/forge-dev >> >> > > > >> > >> >> > > > >> > >> >> > > > >> > >> >> > > > >> > -- >> >> > > > >> > Lincoln Baxter, III >> >> > > > >> > http://ocpsoft.com >> >> > > > >> > http://scrumshark.com >> >> > > > >> > "Keep it Simple" >> >> > > > >> >> >> > > > >> /max >> >> > > > >> http://about.me/maxandersen >> >> > > > >> >> >> > > > >> >> >> > > > >> >> >> > > > >> >> >> > > > >> >> >> > > > >> >> >> > > > >> -- >> >> > > > >> Lincoln Baxter, III >> >> > > > >> http://ocpsoft.com >> >> > > > >> http://scrumshark.com >> >> > > > >> "Keep it Simple" >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > -- >> >> > > > > Lincoln Baxter, III >> >> > > > > http://ocpsoft.com >> >> > > > > http://scrumshark.com >> >> > > > > "Keep it Simple" >> >> > > > >> >> > > > /max >> >> > > > http://about.me/maxandersen >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > -- >> >> > > > Lincoln Baxter, III >> >> > > > http://ocpsoft.com >> >> > > > http://scrumshark.com >> >> > > > "Keep it Simple" >> >> > > >> >> > > /max >> >> > > http://about.me/maxandersen >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > -- >> >> > > Lincoln Baxter, III >> >> > > http://ocpsoft.com >> >> > > http://scrumshark.com >> >> > > "Keep it Simple" >> >> > >> >> > /max >> >> > http://about.me/maxandersen >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > -- >> >> > Lincoln Baxter, III >> >> > http://ocpsoft.com >> >> > http://scrumshark.com >> >> > "Keep it Simple" >> >> >> >> /max >> >> http://about.me/maxandersen >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> Lincoln Baxter, III >> >> http://ocpsoft.com >> >> http://scrumshark.com >> >> "Keep it Simple" >> > >> > >> > >> > -- >> > Lincoln Baxter, III >> > http://ocpsoft.com >> > http://scrumshark.com >> > "Keep it Simple" >> > >> > _______________________________________________ >> > forge-dev mailing list >> > forge-dev(a)lists.jboss.org >> > https://lists.jboss.org/mailman/listinfo/forge-dev >> > >> > >> > _______________________________________________ >> > forge-dev mailing list >> > forge-dev(a)lists.jboss.org >> > https://lists.jboss.org/mailman/listinfo/forge-dev >> >> /max >> http://about.me/maxandersen >> >> >> >> >> _______________________________________________ >> forge-dev mailing list >> forge-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/forge-dev >> > > > _______________________________________________ > forge-dev mailing list > forge-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/forge-dev > > -- Lincoln Baxter, III http://ocpsoft.com http://scrumshark.com "Keep it Simple"