[ https://jira.jboss.org/browse/JBVFS-159?page=com.atlassian.jira.plugin.sy... ] Marko Strukelj commented on JBVFS-159: -------------------------------------- ZipEntryInputStream (later upgraded to CertificateReaderInputStream which just added extra functionality for reading certificates) is there to provide reference counting for usages of underlying ZipFileWrapper. Without it ZipFileWrapper will increase refCount for every newly read class/resource but never decrease it. The result is a permanent file handle leak which on windows systems makes it impossible to delete the file. After enough time the counter would go beyond MAX_VALUE and into negative territory potentially causing closing of underlying ZipFile despite outstanding yet unread ZipEntryInputStream - triggering IOException on class/resource loading. If incrementing of refCount is disabled such an arbitrary closing will go into effect immediately. To put it simply - this class plays a role, you may eventually experience problems having commented it out. One possible reason for increased memory usage might be the use of finalize(). The finalize() method is not particularly important - it's just a final safety net for code that forgets to close unfully read streams - mainly here for possible buggy third party libraries on which you as a developer may not have much influence. Removing it could cause existing applications using buggy libraries, and running on Windows to start develop file locking problems, and this can not be made configurable as a class either has a finalize() method or doesn't - and that is not configurable ... although there might be complicated workarounds for that as well ... Could you repeat your test by removing your initial code comments and by commenting out just the finalize() method, so that we see if this is the cause? -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBVFS-159?page=com.atlassian.jira.plugin.sy... ] Marko Strukelj commented on JBVFS-159: -------------------------------------- To clarify, my proposal regarding finalize() method is not really about any memory leak, but about garbage collection dynamics which is affected by finalizers, and causes such objects to take more gc cycles to remove. As far as memory leak is concerned ... I tried some tests with, and without the proposed removal of ZipEntryInputStream, using vfs 2.1.2.GA. I am unable to see any noticable difference in memory usage between the two - which means the difference is less than the variance. My test works in isolation from jboss-as. It uses VFS directly to scan and fully read the content of all the archives in jbossas deploy directory, where I have some extra ears, some 50Mb big. In this scenario the memory goes up to about 150M in both cases. If all the code that uses resource/class streams properly closes streams after reading or simply just fully reads the streams, then there should be no memory leaking here - and I can't find one. If there are outstanding open streams on the zip file, then of course it would be wrong to forcefully close it (equivallent of pulling the rug from under the open stream - causing a potential IOException). We shouldn't blame VFS for such leaks. Again, the function of ZipEntryInputStream is precisely to support timely, and correct release of ZipFile objects - therefore to release memory, and file locks as soon as possible. I'm attaching the test. Place it in jboss_vfs/src/test/java/org/jboss/test/virtual/test. Adjust the value of DIR variable to point to your deploy dir or wherever you have many java archive files. Run the test with: mvn test -Dtest=VFSZipMemoryTestCase You can tweak memory options using: mvn test -Dtest=VFSZipMemoryTestCase "-DargLine=-server -Xss128k -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC" Use jvisualvm in parallel, connect to SurefireBooter process when it appears in the list of processes, observe memory usage, do some memory profiling, some snapshots ... During the run you can see instances of ZipEntryInputStream being created. But at the end of the run their number is zero - meaning they were all released and GC-ed. Ergo - with proper usage there is no memory leak. We need to differentiate between a memory leak, and a runtime memory footprint. There doesn't seem to be any memory leak, and runtime footprint for the isolated test is indistinguishable between current code and your proposed fix. My hunch is that in your case you have some code somewhere - maybe some third party library, that holds on to the ZipEntryInputStream preventing the release of underlying ZipFileWrapper, and transitively the ZipFile - the one that actually consumes memory, as everything else are just lightweight wrappers around it. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBVFS-159?page=com.atlassian.jira.plugin.sy... ] Samuel Cai commented on JBVFS-159: ---------------------------------- Marko, thanks again for you test. Last week I did several tests and found the root cause. I'll attach all materials in a zip. Also I'll list all testings I've done, there were some interesting findings. 1. I wrote a java class to utilize VFS directly, see the Test1.java, also modified ZipFileWrapper.java to allow system property jboss.vfs.useNativeStream to use ZipInputStream instead of ZipEntryInputStream. By doing this on a directory (has 81 jars, total 48M), I can clearly see process size issue when using ZipEntryInputStream, but not if use ZipInputStream. Since this is single thread and noReaperOverride is true, ZipFile was closed every time after one ZipEntry was processed, additional log proved this. Note, I skipped processing directory, if not, I can see process size issue even using ZipInputStream, just always a little better than using ZipEntryInputStream. So there seems have some other issue in method recomposeZipAsInputStream, but no time to check it. This is an interesting finding. 2. Based on test #1, we can see the difference is, if use ZipEntryInputStream, steam may be left opened, ZipFile is closed (according to JDK, which then close all of the input streams previously returned by invocations of the {@link #getInputStream getInputStream} method). If use ZipInputStream, ZipFile isn't closed (maybe after GC), instead I only close the input streams. This is clearly a JDK related issue, so I wrote a pure test (Test2.java) which only uses JDK's API. It reads three arguments, first one is the jar/zip file it'll process, second is the loop times, third is whether close stream before closing zip file. For example (Test2.java is in leakdetector.jar): /data/jdk/bin/java -cp leakdetector.jar Test2 /tmp/vfs-nested.tmp/80d72123_jboss-xa-jdbc.jar 10000 false /data/jdk/bin/java -cp leakdetector.jar Test2 /tmp/vfs-nested.tmp/80d72123_jboss-xa-jdbc.jar 10000 true I wrote a simple script(jps_test.sh) to record process size (VSZ) and RSS, run it in another window first, here is the result: False: Logging jps info Mon Jun 21 00:33:39 PDT 2010 1257728 48548 Logging jps info Mon Jun 21 00:33:40 PDT 2010 1339648 94380 Logging jps info Mon Jun 21 00:33:41 PDT 2010 1422752 141176 Logging jps info Mon Jun 21 00:33:42 PDT 2010 1508768 188620 Logging jps info Mon Jun 21 00:33:43 PDT 2010 1589476 238240 True: Logging jps info Mon Jun 21 00:34:17 PDT 2010 1195024 16436 Logging jps info Mon Jun 21 00:34:18 PDT 2010 1195024 20476 Logging jps info Mon Jun 21 00:34:19 PDT 2010 1196784 26280 Logging jps info Mon Jun 21 00:34:20 PDT 2010 1196784 30728 Logging jps info Mon Jun 21 00:34:21 PDT 2010 1196784 33912 3. Based on #2, we can see if don't close stream before closing ZipFile, will have native memory leak. JAVA uses ZLIB for zip native operation, we tested three version of libz.so: 1.2.1.2 (come with OS), 1.2.3 (in JBoss's native), 1.2.5 (I compiled by myself), no luck. My coworker tested IBM JDK, no luck. All were done on Redhat, if want to test other Linux, need more time. Based on above, could you try the Test2.java on your local Linux machine? If can reproduce, then means we have to close stream before closing ZipFile. (Since I can easily reproduce locally with pure test, sorry I can't try you test, also due to requirement maven setup on Linux server, I right now only setup local windows) Note, I found several old bugs/issues for JDK, related to native memory leak of zip under multi-core CPU system, if you can't reproduce, maybe you can try on that kind of machine. If you still can't reproduce, then maybe our system is different to others, I may file a bug to JAVA to see if they can help. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBVFS-159?page=com.atlassian.jira.plugin.sy... ] Samuel Cai commented on JBVFS-159: ---------------------------------- The JDK bug/issue I mentioned: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6734186 -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBVFS-159?page=com.atlassian.jira.plugin.sy... ] Samuel Cai commented on JBVFS-159: ---------------------------------- Marko, I found a confirmed JDK bug: http://bugs.sun.com/view_bug.do?bug_id=6735255, which is the same as my finding. All JDK on all system have this problem, only except JDK 7(b62) or after. Not sure why Sun didn't fix it for JDK6, based on current situation, updating VFS to use the workaround should be the only way: Ensure that your application calls ZipFileInputStream.close() explicitly. I'll provide my suggestion patch later. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBVFS-159?page=com.atlassian.jira.plugin.sy... ] Marko Strukelj commented on JBVFS-159: -------------------------------------- One important thing I forgot to mention: the patch relies on ZipFileLockReaper, so make sure you _don't_ have -Djboss.vfs.forceNoReaper=true set. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBVFS-159?page=com.atlassian.jira.plugin.sy... ] Samuel Cai updated JBVFS-159: ----------------------------- Attachment: change.tar.gz -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBVFS-159?page=com.atlassian.jira.plugin.sy... ] Samuel Cai commented on JBVFS-159: ---------------------------------- Hi Marko, I did some tests locally: Start JBoss (with our war deployed), and do nothing in 10 mins, test three times. I monitor three values: Mem Usage, Peak Mem Usage, VM Size. Original: Mem Usage: 470M-700M after deployment, wide range, every time is totally different. After about 5 mins, drop to around 10M suddenly, and then increase slowly. Peak Mem Usage: up to 750M VM Size: 950-1150M after deployment, wide range, every time is totally different. Almost no changes later. My patch: Mem Usuage: around 550M after deployment. It drops to around 10M suddenly after several mins, and then increase slowly. Peak Mem Usage: around 550M. VM Size: around 930M after deployment. Almost no changes later. Your patch: Mem Usuage: around 700M after deployment. It drops to around 10M suddenly after several mins, and then increase slowly. Peak Mem Usage: around 750M VM Size: around 1089M after deployment. Almost no changes later. I'm using Windows XP and 2 cores CPU, seems windows manages Mem Usage and VM Size better, but test result still indicates that not closing stream will result in higher VM Size. JVM startup parameter: -Xss128k -Xmn250m -XX:PermSize=128m -XX:MaxPermSize=128m -Xms750m -Xmx750m -server Could you do some more tests on windows, maybe you'll see the Mem Usuage dropped too. Also about your patch, you mentioned that it's not for prod, how can we make it for prod? Thanks, Samuel -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBVFS-159?page=com.atlassian.jira.plugin.sy... ] Samuel Cai commented on JBVFS-159: ---------------------------------- Marko, firstly thanks for you patch, just seems it doesn't work for our specific environment. Since the bug is confirmed by Sun, I think we need to close stream, or at least providing switchable options as you said. I'm eager to see if my patch (or any other that can fix our problem) can be reviewed by JBoss team and agreed formally, this way our manager will feel safer to put the patch on QA for further testing and finally move it onto PROD. Thanks, Samuel -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBVFS-159?page=com.atlassian.jira.plugin.sy... ] Marko Strukelj commented on JBVFS-159: -------------------------------------- Samuel, everything that has to do with support belongs more to a customer portal. So let's move that part of the conversation there. Thanks. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBVFS-159?page=com.atlassian.jira.plugin.sy... ] Samuel Cai commented on JBVFS-159: ---------------------------------- Hi Marko, I think this is a generic problem for all users, just looks worse in our environment, I'm hoping JBoss can provide an official path. Thanks, Samuel -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBVFS-159?page=com.atlassian.jira.plugin.sy... ] Samuel Cai commented on JBVFS-159: ---------------------------------- Thanks, we're using Redhat. I'll check with our manager to see if can follow you suggestion. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBVFS-159?page=com.atlassian.jira.plugin.sy... ] Marko Strukelj commented on JBVFS-159: -------------------------------------- JBoss AS 6 M3 introduced a next version of VFS (VFS3) which is very different from 2.x version, so it doesn't necessarily suffer this same problem, but it would be worth checking out. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBVFS-159?page=com.atlassian.jira.plugin.sy... ] Shakil Momin commented on JBVFS-159: ------------------------------------ One simple possible solution is to deploy application in uncompressed form. I have tested it and its working. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JBVFS-159?page=com.atlassian.jira.plugin.... ] Alexander Nemsadze commented on JBVFS-159: ------------------------------------------ I suppose, I'm having same issue http://community.jboss.org/thread/160201 Can't find any straight-forward solution yet. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira