Re: [jboss-dev] Scanning jars for package capabilities

Ok, great. Now we need to get a vfs handler implementation for your jzipfile - vfsjzip. David M. Lloyd wrote: > When I say "ZipFile" I should really say "zip file". I of course > would heavily advocate using my "jzipfile" library that I wrote last > week which is exactly what you describe: a pure Java implementation > without the native and locking bullshit. :-) > > That said however, even jzipfile requires a java.io.File and shuns the > scanning approach. Here's why. > > The first reason is simple. To stream a zip file to find a single > entry, you have to read the entire zip file's data up to that entry > rather than just jumping to the exact offset of the data you're > interested in and pulling down the data. If you scan the zip file > completely even once, you might as well have just copied it and read > the directory instead. > > The second reason is more involved. A zip file has two parts: the > local file section and the directory. The local file section looks > something like this: > > [local file 0] > signature word 0x04034b50 > compression = deflate > local CRC* > local compressed size* > local uncompressed size* > name = foo/bar/foobar.class > [..possibly compressed file data goes here..] > real CRC** > real compressed size** > real uncompressed size** > [local file 1] > signature word 0x04034b50 > compression = deflate > local CRC* > local compressed size* > local uncompressed size* > name = foo/bar/foobaz.class > [..possibly compressed file data goes here..] > real CRC** > real compressed size** > real uncompressed size** > [...] > > The asterisk * indicates that this field is a valid header field but > it is almost *always* zero, with the actual information specified > *after* the compressed data (**). (There's a few extra things in > there that I don't mention, including some odd stuff that the JAR tool > does to distinguish a real JAR file from a file named .jar that was > created by some other zip-like tool). > > This means that if you're scanning from the start of a zip file (like > ZipInputStream), you usually have no way to know how long each file is > other than just reading the bytes with e.g. InflaterInputStream and > hope that the EOF matches up correctly (which it doesn't always do). > But there is one worse case. If a nested zip file is "stored" rather > than "deflated", there may be *no* way to distinguish between records > in the nested zip from the parent zip, so you may reach the end of the > stored data of a nested zip entry and the streamer would get confused > and think that it hit the end of the parent entry. Very messy. > > Now the directory is at the *end* of the zip file and looks like this: > [directory entry 0] > signature word 0x02014b50 > compression = deflate > timestamp = xxxx > real CRC > real compressed size > real uncompressed size > offset of local header = 0 > name = foo/bar/foobar.class > comment = blah blah > [directory entry 1] > signature word 0x02014b50 > compression = deflate > timestamp = xxxx > real CRC > real compressed size > real uncompressed size > offset of local header = 12345 > name = foo/bar/foobaz.class > comment = blah blah > [...] > [end of directory] > signature word 0x06054b50 > offset of first directory entry = 24680 > entry count = N > comment = "" > [EOF] > > As you can see the directory entries give us hard values we can use > for the lengths (both compressed and uncompressed) which makes it far > more resilient against errors. Also, the directory is all packed > together, meaning that we can read just a small portion of the zip > file and thus fully index it (and, if we copied the zip to a "safe" > location, we can cache the directory and close the zip file, thus > avoiding one aspect of the infamous locking issues on Windows). So > ideally, we always use the directory to locate zip entries. > > The question is, how do you find the directory? You can't seek from > the beginning of the file because of the problems already outlined > with that approach. The only reasonably accurate way to get the > directory is to read the *end* of the file first, which generally > means that InputStream is not an option to get this data. The only > real option is to use a random-access file method (FileChannel or > RandomAccessFile) to get at this data. This means that if you have an > InputStream (which is what you can generally get out of VFS), you need > to create a temp file and slurp the data into it before you can index it. > > I realize that this is a full iteration of the zip file, *however* it > means that once you do this, you have instant access to any zip entry > from that point on, *and* the lifetime of the zip file is no longer > tied to its parent (be it a directory or another zip file), which > fixes a wide variety of issues, including the requirement that > resources don't start disappearing before undeploy() is called (in the > case where the FS-backed deploy/ mechanism is used), more Windows > locking issues, etc. I also believe that if we *always* copy and > *never* scan the zip file, we'll have a net performance gain in the > end (assuming we only ever copy and index each zip exactly one time). > And of course things are a lot less likely to blow up surprisingly if > you have a copy of the zip file. > > - DML > > On 06/19/2009 11:25 AM, Scott Stark wrote: >> I don't view the ZipFile api as flexible enough as it requires a >> java.io.File and nested jars. In addition all jdk implementations >> have a native component that has tended to introduce locking and >> memory faults. That is why I believe we need a pure java jar/zip file >> implementation that addresses these limitations. _______________________________________________ jboss-development mailing list jboss-development(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-development