Hi Everyone, Since this weekends hudson meltdown seems likely related to a memory leak, I have reverted VFS to M1. Recent updates to the VFS have been observed to leak (JBAS-6805). This is potentially caused by a thread safety issue introduced in M2. Shelly has kicked off another 5.1 testrun with this update, and is checking to see see if M1 resolved 6805. In the meantime I am working on an update to the VFS that does not have the recent TS issues.

Ales, So, we want to test with VFS 2.2.0.M1 and seam-int 5.2.0-SNAPSHOT? Shelly ----- Original Message ----- From: "Ales Justin" <ales.justin(a)gmail.com&gt; To: "JBoss.org development list" <jboss-development(a)lists.jboss.org&gt; Sent: Monday, April 20, 2009 12:13:24 PM GMT -05:00 US/Canada Eastern Subject: Re: [jboss-dev] Reverted VFS to 2.2.0.M1 (& hudson madness) Can we also change the Seam-int to 5.2.0-SNAPSHOT? (Or Shelly, you already did this?) This should fix the performance issue EmbJopr is having with VFSScanner going "into" plugins/*.jar files. Jason T. Greene wrote: > Hi Everyone, > > Since this weekends hudson meltdown seems likely related to a memory > leak, I have reverted VFS to M1. Recent updates to the VFS have been > observed to leak (JBAS-6805). This is potentially caused by a thread > safety issue introduced in M2. Shelly has kicked off another 5.1 testrun > with this update, and is checking to see see if M1 resolved 6805. > > In the meantime I am working on an update to the VFS that does not have > the recent TS issues. > _______________________________________________ jboss-development mailing list jboss-development(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-development _______________________________________________ jboss-development mailing list jboss-development(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-development

Unfortunately the tempinfo TS fixes resurrects JBAS-6715, although in a slightly different manner (the VFS test created for 6715 passes). The root problem still seems to be a race that leads to the deployers somehow holding on to a stale ZipFileWrapper instance. This is going to take awhile to track down. Ales Justin wrote: > VFS 2.2.0.M1 has .war cleanup bug > VFS 2.2.0.M2 has TS issues + Iterator.remove bug > VFS 2.2.0.M3 has TS issues > > Seam-int 5.2.0-SNAPSHOT excludes plugins/*.jar from scan > > If you could wait for some VFS 2.2.0-SNAPSHOT from Jason > (it depends how fast he can provide it) > that would be the best choice. > > Otherwise do your pick. :-) > > Shelly McGowan wrote: >> Ales, >> >> So, we want to test with VFS 2.2.0.M1 and seam-int 5.2.0-SNAPSHOT? >> >> >> Shelly >> >> >> ----- Original Message ----- >> From: "Ales Justin" <ales.justin(a)gmail.com&gt; >> To: "JBoss.org development list" <jboss-development(a)lists.jboss.org&gt; >> Sent: Monday, April 20, 2009 12:13:24 PM GMT -05:00 US/Canada Eastern >> Subject: Re: [jboss-dev] Reverted VFS to 2.2.0.M1 (& hudson madness) >> >> Can we also change the Seam-int to 5.2.0-SNAPSHOT? >> (Or Shelly, you already did this?) >> >> This should fix the performance issue EmbJopr is having with >> VFSScanner going "into" plugins/*.jar files. >> >> Jason T. Greene wrote: >>> Hi Everyone, >>> >>> Since this weekends hudson meltdown seems likely related to a memory >>> leak, I have reverted VFS to M1. Recent updates to the VFS have been >>> observed to leak (JBAS-6805). This is potentially caused by a thread >>> safety issue introduced in M2. Shelly has kicked off another 5.1 >>> testrun with this update, and is checking to see see if M1 resolved >>> 6805. >>> >>> In the meantime I am working on an update to the VFS that does not >>> have the recent TS issues. >>> >> _______________________________________________ >> jboss-development mailing list >> jboss-development(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-development >> _______________________________________________ >> jboss-development mailing list >> jboss-development(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-development >> > _______________________________________________ > jboss-development mailing list > jboss-development(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-development

Ales Justin wrote: > So, you're saying that even after you fixed ts issues, > the 6715 is still there? Yes, after fixing just the TS issues with TempInfo usage (wanted to limit the scope of this), the DeploymentManager test in the AS fails, but the override test in the VFS (and all other tests) pass. I have not committed this though, since I wanted to make sure the changes did not lead to regression first. I could if you want to take a look. > I don't see a race issue. The DeploymentManager test succeeds several times before failing, so I was assuming timing related, but it might not be. Ill hopefully have more info soon. > It must be something similar to what we already had, > something is not cleaning up properly behind itself. > That was the issue before -> due to overridden TempInfo, > nobody reset the parent VFSContext. > Jason T. Greene wrote: >> Unfortunately the tempinfo TS fixes resurrects JBAS-6715, although >> in a slightly different manner (the VFS test created for 6715 >> passes). The root problem still seems to be a race that leads to >> the deployers somehow holding on to a stale ZipFileWrapper >> instance. This is going to take awhile to track down. >> >> Ales Justin wrote: >>> VFS 2.2.0.M1 has .war cleanup bug >>> VFS 2.2.0.M2 has TS issues + Iterator.remove bug >>> VFS 2.2.0.M3 has TS issues >>> >>> Seam-int 5.2.0-SNAPSHOT excludes plugins/*.jar from scan >>> >>> If you could wait for some VFS 2.2.0-SNAPSHOT from Jason >>> (it depends how fast he can provide it) >>> that would be the best choice. >>> >>> Otherwise do your pick. :-) >>> >>> Shelly McGowan wrote: >>>> Ales, >>>> >>>> So, we want to test with VFS 2.2.0.M1 and seam-int 5.2.0-SNAPSHOT? >>>> >>>> >>>> Shelly >>>> >>>> >>>> ----- Original Message ----- >>>> From: "Ales Justin" <ales.justin(a)gmail.com&gt; >>>> To: "JBoss.org development list" <jboss-development(a)lists.jboss.org >>>> > >>>> Sent: Monday, April 20, 2009 12:13:24 PM GMT -05:00 US/Canada >>>> Eastern >>>> Subject: Re: [jboss-dev] Reverted VFS to 2.2.0.M1 (& hudson >>>> madness) >>>> >>>> Can we also change the Seam-int to 5.2.0-SNAPSHOT? >>>> (Or Shelly, you already did this?) >>>> >>>> This should fix the performance issue EmbJopr is having with >>>> VFSScanner going "into" plugins/*.jar files. >>>> >>>> Jason T. Greene wrote: >>>>> Hi Everyone, >>>>> >>>>> Since this weekends hudson meltdown seems likely related to a >>>>> memory leak, I have reverted VFS to M1. Recent updates to the >>>>> VFS have been observed to leak (JBAS-6805). This is potentially >>>>> caused by a thread safety issue introduced in M2. Shelly has >>>>> kicked off another 5.1 testrun with this update, and is >>>>> checking to see see if M1 resolved 6805. >>>>> >>>>> In the meantime I am working on an update to the VFS that does >>>>> not have the recent TS issues. >>>>> >>>> _______________________________________________ >>>> jboss-development mailing list >>>> jboss-development(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-development >>>> _______________________________________________ >>>> jboss-development mailing list >>>> jboss-development(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-development >>>> >>> _______________________________________________ >>> jboss-development mailing list >>> jboss-development(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-development >> >> > _______________________________________________ > jboss-development mailing list > jboss-development(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-development -- Jason T. Greene JBoss, a division of Red Hat _______________________________________________ jboss-development mailing list jboss-development(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-development

I finally have the cause: Deploy ------ 1. Structure scanning creates tempX for the nested war (ZipEntryContext-A) 2. During deployment JEEDeployedObject calls getResource, which opens a vfs URL 3. The vfs registry cache lookup fails because ZipEntryContext-A is a nested resource, thus the original deployment is rescanned. 4. ZipEntryContext-B is created in the same file context as A (the deploy dir), therefore *tempX is reused* (see initEntries:539)

Undeploy -------- 5. tempX is removed 6. ZipEntyContext-A is cleaned, and it's container is reset Redeploy -------- 7. During structure scanning ZipEntryContext-B is retrieved (which still refers to tempX)is opened *BOOM* So the fundamental problem is that we reuse temps for entities which have different lifecycles. We also always purge temps based on the source location, so even if we keep them separate, the first tempinfo cleanup will remove them all.

Ales Justin wrote: > Sure, commit and I'll have a look tomorrow. > > Sent from my iPhone > > On Apr 21, 2009, at 19:12, "Jason T. Greene" <jason.greene(a)redhat.com&gt; > wrote: > >> Ales Justin wrote: >>> So, you're saying that even after you fixed ts issues, >>> the 6715 is still there? >> >> Yes, after fixing just the TS issues with TempInfo usage (wanted to >> limit the scope of this), the DeploymentManager test in the AS fails, >> but the override test in the VFS (and all other tests) pass. I have >> not committed this though, since I wanted to make sure the changes >> did not lead to regression first. I could if you want to take a look. >> >>> I don't see a race issue. >> >> The DeploymentManager test succeeds several times before failing, so >> I was assuming timing related, but it might not be. Ill hopefully >> have more info soon. >> >>> It must be something similar to what we already had, >>> something is not cleaning up properly behind itself. >>> That was the issue before -> due to overridden TempInfo, >>> nobody reset the parent VFSContext. >>> Jason T. Greene wrote: >>>> Unfortunately the tempinfo TS fixes resurrects JBAS-6715, although >>>> in a slightly different manner (the VFS test created for 6715 >>>> passes). The root problem still seems to be a race that leads to >>>> the deployers somehow holding on to a stale ZipFileWrapper >>>> instance. This is going to take awhile to track down. >>>> >>>> Ales Justin wrote: >>>>> VFS 2.2.0.M1 has .war cleanup bug >>>>> VFS 2.2.0.M2 has TS issues + Iterator.remove bug >>>>> VFS 2.2.0.M3 has TS issues >>>>> >>>>> Seam-int 5.2.0-SNAPSHOT excludes plugins/*.jar from scan >>>>> >>>>> If you could wait for some VFS 2.2.0-SNAPSHOT from Jason >>>>> (it depends how fast he can provide it) >>>>> that would be the best choice. >>>>> >>>>> Otherwise do your pick. :-) >>>>> >>>>> Shelly McGowan wrote: >>>>>> Ales, >>>>>> >>>>>> So, we want to test with VFS 2.2.0.M1 and seam-int 5.2.0-SNAPSHOT? >>>>>> >>>>>> >>>>>> Shelly >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>> From: "Ales Justin" <ales.justin(a)gmail.com&gt; >>>>>> To: "JBoss.org development list" <jboss-development(a)lists.jboss.org&gt; >>>>>> Sent: Monday, April 20, 2009 12:13:24 PM GMT -05:00 US/Canada >>>>>> Eastern >>>>>> Subject: Re: [jboss-dev] Reverted VFS to 2.2.0.M1 (& hudson madness) >>>>>> >>>>>> Can we also change the Seam-int to 5.2.0-SNAPSHOT? >>>>>> (Or Shelly, you already did this?) >>>>>> >>>>>> This should fix the performance issue EmbJopr is having with >>>>>> VFSScanner going "into" plugins/*.jar files. >>>>>> >>>>>> Jason T. Greene wrote: >>>>>>> Hi Everyone, >>>>>>> >>>>>>> Since this weekends hudson meltdown seems likely related to a >>>>>>> memory leak, I have reverted VFS to M1. Recent updates to the >>>>>>> VFS have been observed to leak (JBAS-6805). This is potentially >>>>>>> caused by a thread safety issue introduced in M2. Shelly has >>>>>>> kicked off another 5.1 testrun with this update, and is checking >>>>>>> to see see if M1 resolved 6805. >>>>>>> >>>>>>> In the meantime I am working on an update to the VFS that does >>>>>>> not have the recent TS issues. >>>>>>> >>>>>> _______________________________________________ >>>>>> jboss-development mailing list >>>>>> jboss-development(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/jboss-development >>>>>> _______________________________________________ >>>>>> jboss-development mailing list >>>>>> jboss-development(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/jboss-development >>>>>> >>>>> _______________________________________________ >>>>> jboss-development mailing list >>>>> jboss-development(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/jboss-development >>>> >>>> >>> _______________________________________________ >>> jboss-development mailing list >>> jboss-development(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-development >> >> >> -- >> Jason T. Greene >> JBoss, a division of Red Hat >> _______________________________________________ >> jboss-development mailing list >> jboss-development(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-development > _______________________________________________ > jboss-development mailing list > jboss-development(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-development

Ales Justin wrote: >> I finally have the cause: >> >> Deploy >> ------ >> 1. Structure scanning creates tempX for the nested war >> (ZipEntryContext-A) >> >> 2. During deployment JEEDeployedObject calls getResource, which opens >> a vfs URL >> >> 3. The vfs registry cache lookup fails because ZipEntryContext-A is a >> nested resource, thus the original deployment is rescanned. >> >> 4. ZipEntryContext-B is created in the same file context as A (the >> deploy dir), therefore *tempX is reused* (see initEntries:539) > > Why exactly is ZEC-B created? > It should be only one, as that's the point of having a cache. It looks like nested zip entries are never registered in the cache (see mountZipStream) in ZipEntryContext.

>> Why exactly is ZEC-B created? >> It should be only one, as that's the point of having a cache. > > It looks like nested zip entries are never registered in the cache > (see mountZipStream) in ZipEntryContext. I still don't see how that could happen. As the only api to get a VirtualFile is over VFS class. And VFS goes over VFSRegistry, which knows about the cache. New mounted ZEC should be a child of some already cached context. And when someone asks for URL --> VirtualFile, it should actually hit that parent context, (see your newly added VFSContext::getFurthestParentTemp(String path)) which would then know about that newly mounted ZEC.

>> If it was only a single instance, as the cache presumes, >> there should be no problem, as the entity is reset no cleanup. > > That would definitely be more efficient, but what if the cache evicts > the entry (lru etc)? By keeping a VirtualFile ref, you should be aware that it might be cleaned. The only time we currently do it, is at undeploy, and the only ref we keep is the top deployment root. If things are properly reset, there should not be a problem. I agree it's not a bullet proof concept, but I doubt we need to impl GC like functionality for this usecase.

> That would also mean making sure that the deployment code never holds > on to a ref during redeploy. That would be the simplest workaround, but I would still like to see VFS's cleanup doing things properly, what ever that might eventually be - which impl, ...

> I can think of two potential ways to solve this, although both would > exceed the timeline we need to get this fixed by (last week :) > > 1. We could cleanup just the calling handler's temp, and refcount the > temp (although this would require that people call cleanup on virtual > file entries) This would never work, as you don't have all the handles available at cleaup or/and people would have to code against a notion of doing a cleanup.

> 2. We could make VirtualFile refs rediscover themselves after they are > cleaned This is probably doable, but going backwards on a structure like nested zips is a pita. (been there, done that :-)) e.g. if you wanna keep things minimal + have valid fs structure you would have to somehow change or reuse the existing parent refs

> Well, I am exhausted, so I am going to crash. Having nightmares about the VFS? :-)

> I just committed the JBVFS-108 changes. In case you have trouble > reproducing on Windows, it > 's easy to find if you modify ZipEntryContext.getChild to check if the > child's temp is still registered and dump an error if it is not. I'll try to see why that ZEC-B is created. Or if you could provide me with your usecase, either as instructions on how that new creation happens or putting it directly into VFS's tests.

>> Why exactly is ZEC-B created? >> It should be only one, as that's the point of having a cache. > > It looks like nested zip entries are never registered in the cache > (see mountZipStream) in ZipEntryContext. I still don't see how that could happen. As the only api to get a VirtualFile is over VFS class. And VFS goes over VFSRegistry, which knows about the cache. New mounted ZEC should be a child of some already cached context. And when someone asks for URL --> VirtualFile, it should actually hit that parent context, (see your newly added VFSContext::getFurthestParentTemp(String path)) which would then know about that newly mounted ZEC.

> 7. VFSRegistry.getFile is called by the JSR77 deployment code > (J2EEDeployment) I need to check this.

> 1. ProfileService calls VFS.getRoot() on the ear > 2. The VirtualFile is passed to MainDeployer.addDeployment > 3. Structure scanning is performed, which holds on to the roots. Only client side top root is kept, all other (sub)roots are cleaned up at undeploy, and new DeploymentContext (+ roots) are created an (re)deploy. > 4. MainDeployer.process() > 5. Undeploy is performed which invalidates the EAR ZEC > 6. Deploy is performed which *reuses* the root in 2, so everything > using the VDF gets the old ZEC Only top root should get reused - the client side, all other should be re-created.

> The VDF code creates new nested ZECs rooted at the old EAR ZEC. This > is the true owner of the temp. Yes, old .ear ZEC was reset, hence it needs to re-initialize. Creating new nested ZEC + new temp copy is expected. We could actually leave some thing if we knew it was a re-deploy, but this is done per VDF bases, which is outside the VFS goal. > The scan in the middle of the deployment process gets the new EAR ZEC, > and thus creates a new nested entry under that. I don't see why a new EAR ZEC is created. If we're not explicitly using VFS::createNewRoot, then .ear's ZEC should always come off cached deploy/ root (its cached VFSContext).

> I should be able to do a VFS deployment test that simulates this now. I committed a test to deployers that reproduces this: org/jboss/test/deployers/vfs/deployer/bean/test/VFSRedeployTestCase It verifies that the VirtualFileHandler in the DeploymentUnit is the same as the one in the registry (accessed via VFS.getRoot()).

> I don't see why a new EAR ZEC is created. > > If we're not explicitly using VFS::createNewRoot, > then .ear's ZEC should always come off cached deploy/ root (its cached > VFSContext). The cache on the FileSystemContext(/deploy) is wiped because undeploy calls cleanup on the EAR VF, and that calls removeChild on the DelgateHandler's parent, which is the FSC(/deploy). I should be able to do a VFS deployment test that simulates this now. Although I still want to try out the multiple handle change to tempinfo.