Re: [forge-dev] shading!?

David, If we use a blank/empty modules descriptor, will JARs in plugin.jar:/lib/* be loaded and isolated from the rest of the system? If so, that might be exactly what we need, if combined with an appopriate maven packaging strategy at build time. ~Lincoln On Wed, Apr 20, 2011 at 12:34 PM, Lincoln Baxter, III < lincolnbaxter(a)gmail.com&gt; wrote: > > > I think what JBoss Modules can give you is a functional, safe >> multiple-delegate class loader which you can't get with URL class loaders. >> Specifically, f you want to have a _graph_ of class loaders (as opposed to >> individual flat, merged classloaders) then you need something like Modules. >> > > I think I just embarrassed myself ;) But can weld even handle this kind of > separate class-loader situation? > > >> As far as Maven integration, you could write a ModuleLoader which does >> this for you. You'd have to do version management on your own though. >> Perhaps you could use Aether for the purpose of resolving and fetching >> artifacts, and then use information exposed therefrom to construct the >> module graph. >> > > Right, which is something Forge can already do pretty handily (using > Aether): > https://github.com/seam/forge/blob/master/shell-api/src/main/java/org/jbo... > > >> http://www.sonatype.com/people/2010/08/introducing-aether/ >> >> Your application does not have to be modular to use JBoss Modules. Though >> of course in my opinion that makes it much nicer. :-) >> >> >> On 04/20/2011 10:56 AM, Lincoln Baxter, III wrote: >> >>> I agree with both of you guys, which is why I am soliciting ideas for >>> this now. >>> >>> I can also say that we do not have one big Classpath right now (unless I >>> am horribly mistaken.) Forge already supports classpath isolation of >>> individual plugins, so we're already half-way there. If my understanding >>> of classloaders is at all right, It would not be too much work to >>> implement a maven resolver and additional level of Composite >>> classloaders to achieve this goal. >>> >>> I don't believe we need to pull in a heavyweight module system like >>> OSGI, or force developers to do even the simple configuration for >>> something like JBoss modules (which afaik, does not have a way of >>> packaging the actual dependencies with the module, or calling out to >>> Maven for dependencies - which forge can already do.) >>> >>> *This is what we have right now:* (see today.png) >>> >>> Notice how the Plugin ClassLoaders may reference Forge APIs, but the >>> plugins may not access each other's classloaders because there is no >>> link from Forge APIs to the Plugins. >>> >>> *And this is all I think it should take to do what I am suggesting:* >>> (see tomorrow.png) >>> >>> If it is technically possible to achieve with JBoss modules, via adding >>> in parsing the POM and including dependencies automatically, then I am >>> all for it, but from what I understand, this would not really be a 1-1 >>> mapping and will not be trivial. I know my understanding of ClassLoaders >>> may be a bit Naive, but I don't think that we need to use JBoss modules >>> (or OSGi) to achieve this goal of automatic downloading and loading of >>> plugin dependency JARs. >>> >>> We certainly do not need the ability (yet) to drop an individual module >>> while Forge is running, because Weld does not support this kind of >>> disruption in the BeanStore. If you remove a JAR while Weld is running, >>> you get a big fiery explosion. Same for loading a single JAR, you have >>> to restart Weld because it will not be aware of the changes, and there >>> is not a good API to make it so. It still has to build the bean graph >>> all over again. That is one of the big reasons to use a fully-fledged >>> module system, and we don't have that requirement. Forge simply dumps >>> all classloaders (until JDK 7 we can't close them easily, yeah memory >>> leaks, whatever, not really important right now,) then builds a new >>> network of ClassLoaders and starts over. >>> >>> However, if we can extend JBoss modules with Maven support in this way, >>> and it's something that David would be interested in, then I think >>> that's one big reason to do it that way. Until then, if we can do it >>> quickly my way, I'd like to get it working sooner rather than later. >>> Please please tell me (I'm sure I don't even have to ask) if there are >>> gaping technical holes in what I am proposing, but if there are not, >>> then I'm going to stick to this position until someone bursts my bubble. >>> >>> ~Lincoln >>> >>> >>> On Wed, Apr 20, 2011 at 9:33 AM, Paul Bakker <paul.bakker.nl >>> <http://paul.bakker.nl>@gmail.com <http://gmail.com>> wrote: >>> >>> Yeah, I'm afraid that's right. I think the module system integration >>> should get very high priority before we reach a 1.0 version. It's >>> something that affects the whole way the core is built up, and it >>> will be very difficult to change that later on. >>> >>> Paul >>> >>> >>> >>> >>> On Wed, Apr 20, 2011 at 3:17 PM, Max Rydahl Andersen >>> <max.andersen(a)redhat.com <mailto:max.andersen@redhat.com>> wrote: >>> >>> +1 >>> >>> I agree with what you want to do Lincoln, but don't see how that >>> can be made to happen without a module system. >>> >>> Until then everything just will have to live in one big >>> classpath. >>> >>> /max >>> >>> On Apr 20, 2011, at 14:24, Paul Bakker wrote: >>> >>> > And that's why a module system is needed. If people can just >>> add dependencies there will be duplicate (possibly multiple >>> versions) dependencies. Each plugin should be in a separate >>> classloader, and each dependency should be too so that a plugin >>> never breaks other plugins or Forge itself. It would be a big >>> limitation if plugins can't use libraries otherwise. >>> > >>> > Paul >>> > >>> > On Tue, Apr 19, 2011 at 6:14 PM, Lincoln Baxter, III >>> <lincolnbaxter(a)gmail.com <mailto:lincolnbaxter@gmail.com>> >>> wrote: >>> > That's right, but what I'm saying is that I don't want >>> developers to be responsible for anything but *their code* -- if >>> they have dependencies, those depenencies will be fetched for >>> them (or somehow bundled in the JAR file itself, which is >>> certainly possible, however not my preference.) >>> > >>> > If you could write a plugin, reference dependencies in your >>> POM, and have everything *Just Work* don't you think that would >>> be a friendlier experience? >>> > >>> > ~Lincoln >>> > >>> > >>> > On Mon, Apr 18, 2011 at 5:55 PM, Max Andersen >>> <manderse(a)redhat.com <mailto:manderse@redhat.com>> wrote: >>> > But if your plugin uses multiple jars it is not one jar. >>> > >>> > >>> > /max (sent from my phone) >>> > >>> > >>> > On 18/04/2011, at 19.06, "Lincoln Baxter, III" >>> <lincolnbaxter(a)gmail.com <mailto:lincolnbaxter@gmail.com>> >>> wrote: >>> > >>> >> That doesn't solve the problem of having to drop jar files >>> onto the classpath in order for plugins to work. I want one JAR >>> per plugin. >>> >> >>> >> ~Lincoln >>> >> >>> >> On Mon, Apr 18, 2011 at 12:59 PM, Max Rydahl Andersen >>> <max.andersen(a)redhat.com <mailto:max.andersen@redhat.com>> >>> wrote: >>> >> >>> >> On Apr 18, 2011, at 18:57, Lincoln Baxter, III wrote: >>> >> >>> >> > I don't want to force plugin-developers to create modules >>> for every dependency that their plugin requires. That's why I've >>> been avoiding OSGI or JBoss Modules. >>> >> >>> >> But then you shouldn't be forcing them to shade either - you >>> should just have one global classloader for the plugins then. >>> >> >>> >> /max >>> >> >>> >> > >>> >> > ~Lincoln >>> >> > >>> >> > On Mon, Apr 18, 2011 at 12:37 PM, Max Rydahl Andersen >>> <max.andersen(a)redhat.com <mailto:max.andersen@redhat.com>> >>> wrote: >>> >> > >>> >> > > What do you think about using Maven APIs to inspect the >>> POM and fetch dependencies dynamically for each plugin, then >>> isolate them in the plugin's classloader? >>> >> > >>> >> > Why not just load them in to one classloader so you don't >>> have collisions when there are mixed dependencies on Forge it >>> self ? >>> >> > >>> >> > How about shared data instances ? How does that work ? >>> >> > >>> >> > ...as a side note...creating our own module system now - I >>> feel that is a very bad direction :( >>> >> > Might as well adopt osgi plugin system if you want this >>> kind of separation ? >>> >> > >>> >> > /max >>> >> > >>> >> > > >>> >> > > ~Lincoln >>> >> > > >>> >> > > On Mon, Apr 18, 2011 at 11:56 AM, Max Rydahl Andersen >>> <max.andersen(a)redhat.com <mailto:max.andersen@redhat.com>> >>> wrote: >>> >> > > >>> >> > > On Apr 18, 2011, at 15:15, Lincoln Baxter, III wrote: >>> >> > > >>> >> > > > "if there is a standard location for dependencies" >>> >> > > > >>> >> > > > What do you mean? >>> >> > > >>> >> > > Your "standard" for shading is that you put all classes >>> into the plugin.jar. >>> >> > > >>> >> > > A "standard" for dependencies for a plugin.jar could be >>> "next to the plugin.jar". >>> >> > > >>> >> > > Would still have the problem of overlapping jars but >>> then at least its easier to see where the duplication is. >>> >> > > >>> >> > > /max >>> >> > > >>> >> > > > >>> >> > > > Thx, >>> >> > > > ~Lincoln >>> >> > > > >>> >> > > > On Mon, Apr 18, 2011 at 6:20 AM, Max Rydahl Andersen >>> <max.andersen(a)redhat.com <mailto:max.andersen@redhat.com>> >>> wrote: >>> >> > > > >>> >> > > > > I was thinking we might already be able to do that >>> using the existing pom.xml metadata that's stored in the >>> artifact itself, or is that too tricky? >>> >> > > > >>> >> > > > if there is a standard location for dependencies then >>> it should be fine - at least better than shading ;) >>> >> > > > >>> >> > > > /max >>> >> > > > >>> >> > > > > >>> >> > > > > On Mon, Apr 11, 2011 at 6:51 PM, Max Andersen >>> <manderse(a)redhat.com <mailto:manderse@redhat.com>> wrote: >>> >> > > > > I was thinking Plugin jar having references to >>> dependent jars via manifest.mf >>> >> > > > > >>> >> > > > > /max (sent from my phone) >>> >> > > > > >>> >> > > > > >>> >> > > > > On 12/04/2011, at 00.39, "Lincoln Baxter, III" >>> <lincolnbaxter(a)gmail.com <mailto:lincolnbaxter@gmail.com>> >>> wrote: >>> >> > > > > >>> >> > > > >> Can you give an example of how you would bundle the >>> JARs? (Just put them in /META-INF/dependencies/ ... ?) And would >>> that not cause just as many class conflicts? If you >>> shade/relocate then the deps *should be* completely isolated. >>> >> > > > >> >>> >> > > > >> On Mon, Apr 11, 2011 at 4:11 PM, Max Rydahl >>> Andersen <max.andersen(a)redhat.com >>> <mailto:max.andersen@redhat.com>> wrote: >>> >> > > > >> well, recommending just bundling jars would be a >>> better approach than shading IMO. >>> >> > > > >> >>> >> > > > >> /max >>> >> > > > >> >>> >> > > > >> On Apr 11, 2011, at 16:00, Lincoln Baxter, III >>> wrote: >>> >> > > > >> >>> >> > > > >> > Yeah, shading is currently the recommended >>> approach. Conflicts should be avoided by using relocations. I >>> know this is... not a great method, but for now it's all we've >>> got. Open to suggestions. >>> >> > > > >> > >>> >> > > > >> > ~Lincoln >>> >> > > > >> > >>> >> > > > >> > On Mon, Apr 11, 2011 at 3:41 AM, Max Rydahl >>> Andersen <max.andersen(a)redhat.com >>> <mailto:max.andersen@redhat.com>> wrote: >>> >> > > > >> > Heya, >>> >> > > > >> > >>> >> > > > >> > Lincoln, I just saw your commits to hibernattools >>> plugin at >>> ( >>> https://github.com/forge/plugin-hibernate-tools/commit/8b208b4a8e79dbb8a0... >>> ) >>> >> > > > >> > >>> >> > > > >> > Is shading of jars really the recommended >>> approach for plugins in Forge ? >>> >> > > > >> > >>> >> > > > >> > How are you going to share/avoid collisions of >>> libraries across plugins if they need to bundle via shading ? >>> >> > > > >> > >>> >> > > > >> > /max >>> >> > > > >> > http://about.me/maxandersen >>> >> > > > >> > >>> >> > > > >> > >>> >> > > > >> > >>> >> > > > >> > >>> >> > > > >> > _______________________________________________ >>> >> > > > >> > forge-dev mailing list >>> >> > > > >> > forge-dev(a)lists.jboss.org >>> <mailto:forge-dev@lists.jboss.org> >>> >>> >> > > > >> > >>> https://lists.jboss.org/mailman/listinfo/forge-dev >>> >> > > > >> > >>> >> > > > >> > >>> >> > > > >> > >>> >> > > > >> > -- >>> >> > > > >> > Lincoln Baxter, III >>> >> > > > >> > http://ocpsoft.com >>> >> > > > >> > http://scrumshark.com >>> >> > > > >> > "Keep it Simple" >>> >> > > > >> >>> >> > > > >> /max >>> >> > > > >> http://about.me/maxandersen >>> >> > > > >> >>> >> > > > >> >>> >> > > > >> >>> >> > > > >> >>> >> > > > >> >>> >> > > > >> >>> >> > > > >> -- >>> >> > > > >> Lincoln Baxter, III >>> >> > > > >> http://ocpsoft.com >>> >> > > > >> http://scrumshark.com >>> >> > > > >> "Keep it Simple" >>> >> > > > > >>> >> > > > > >>> >> > > > > >>> >> > > > > -- >>> >> > > > > Lincoln Baxter, III >>> >> > > > > http://ocpsoft.com >>> >> > > > > http://scrumshark.com >>> >> > > > > "Keep it Simple" >>> >> > > > >>> >> > > > /max >>> >> > > > http://about.me/maxandersen >>> >> > > > >>> >> > > > >>> >> > > > >>> >> > > > >>> >> > > > >>> >> > > > >>> >> > > > -- >>> >> > > > Lincoln Baxter, III >>> >> > > > http://ocpsoft.com >>> >> > > > http://scrumshark.com >>> >> > > > "Keep it Simple" >>> >> > > >>> >> > > /max >>> >> > > http://about.me/maxandersen >>> >> > > >>> >> > > >>> >> > > >>> >> > > >>> >> > > >>> >> > > >>> >> > > -- >>> >> > > Lincoln Baxter, III >>> >> > > http://ocpsoft.com >>> >> > > http://scrumshark.com >>> >> > > "Keep it Simple" >>> >> > >>> >> > /max >>> >> > http://about.me/maxandersen >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > -- >>> >> > Lincoln Baxter, III >>> >> > http://ocpsoft.com >>> >> > http://scrumshark.com >>> >> > "Keep it Simple" >>> >> >>> >> /max >>> >> http://about.me/maxandersen >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> -- >>> >> Lincoln Baxter, III >>> >> http://ocpsoft.com >>> >> http://scrumshark.com >>> >> "Keep it Simple" >>> > >>> > >>> > >>> > -- >>> > Lincoln Baxter, III >>> > http://ocpsoft.com >>> > http://scrumshark.com >>> > "Keep it Simple" >>> > >>> > _______________________________________________ >>> > forge-dev mailing list >>> > forge-dev(a)lists.jboss.org <mailto:forge-dev@lists.jboss.org> >>> >>> > https://lists.jboss.org/mailman/listinfo/forge-dev >>> > >>> > >>> > _______________________________________________ >>> > forge-dev mailing list >>> > forge-dev(a)lists.jboss.org <mailto:forge-dev@lists.jboss.org> >>> >>> > https://lists.jboss.org/mailman/listinfo/forge-dev >>> >>> /max >>> http://about.me/maxandersen >>> >>> >>> >>> >>> _______________________________________________ >>> forge-dev mailing list >>> forge-dev(a)lists.jboss.org <mailto:forge-dev@lists.jboss.org> >>> >>> https://lists.jboss.org/mailman/listinfo/forge-dev >>> >>> >>> >>> _______________________________________________ >>> forge-dev mailing list >>> forge-dev(a)lists.jboss.org <mailto:forge-dev@lists.jboss.org> >>> >>> https://lists.jboss.org/mailman/listinfo/forge-dev >>> >>> >>> >>> >>> -- >>> Lincoln Baxter, III >>> http://ocpsoft.com >>> http://scrumshark.com >>> "Keep it Simple" >>> >> >> >> -- >> - DML >> > > > > -- > Lincoln Baxter, III > http://ocpsoft.com > http://scrumshark.com > "Keep it Simple" > -- Lincoln Baxter, III http://ocpsoft.com http://scrumshark.com "Keep it Simple"