[infinispan-dev] Getting rid of GZipInputStream

galder at jboss.org galder at jboss.org
Mon Apr 19 09:14:59 EDT 2010 

Hi Phillipe,

See below:

----- "Philippe Van Dyck" <pvdyck at gmail.com> wrote:

> On Mon, Apr 19, 2010 at 12:36 PM, <galder at jboss.org> wrote:
> 
> > Hi Philippe,
> >
> > So, to sum up, trying to fix
> https://jira.jboss.org/jira/browse/ISPN-133won't solve your problem,
> correct? Did you test this out by any chance?
> >
> 
> Yep, "Basically InputStream#available() is utterly broken.".
> GzipIbputStream's implementation is :
>  public int available() throws IOException {
>         ensureOpen();
>         if (reachEOF) {
>             return 0;
>         } else {
>             return 1;
>         }
>     }
> 
> You definitely cannot use the return value to set your buffer size.

Interesting. It seems like it's not only Apache's HTTP client library that did not care much about available() implementations, but also JDK classes as well. I'd rather read on 1k batches than 1 byte batches. See things like this, I'm even less inclined to urgently look into https://jira.jboss.org/jira/browse/ISPN-133.

> 
> 
> > Did you test Bzip2Compressor streams to see if that worked?
> >
> >
> Yes, perfectly, one more dependency to add to commons compression...
> any
> problem with that ?

I suppose that's a dependency to the cloud cache store module, correct? If so, I'm fine with it :)

> 
> The updated CloudCacheStore with bzip2 compression is ready...

Cool, thanks very much for working on this!!

> 
> Cheers,
> 
> Phil
> 
> 
> > Cheers,
> >
> > ----- "philippe van dyck" <pvdyck at gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > the problem is in the JDK and linked to a 10 years old -never
> fixed-
> > > bug in GZipInputStream...
> > > I propose to get rid of GZipInputStream and use Apache Commons
> > > Bzip2Compressor streams in the CloudCacheStore.
> > >
> > > (And BTW, using len=inputStream.available() is a very bad idea
> with
> > > GZipInputStream...)
> > >
> > > WDYT ?
> > >
> > > phil
> > >
> > > Début du message réexpédié :
> > >
> > > > De : Philippe Van Dyck <pvdyck at gmail.com>
> > > > Date : 16 avril 2010 10:45:19 HAEC
> > > > À : infinispan -Dev List <infinispan-dev at lists.jboss.org>
> > > > Objet : Bug : read after end of stream @ AbstractMarshaller
> > > >
> > > > Hi all,
> > > >
> > > > since I use an InflaterInputStream to send objects to S3 using
> > > JClouds Blobstore, I need a very strict management of streams.
> > > > In AbstractMarshaller,  while ((bytesRead =
> inputStream.read(buf, 0,
> > > buf.length)) != -1) bytes.write(buf, 0, bytesRead) will read after
> the
> > > stream's end, waiting for '-1' to happen.
> > > > You cannot do that with a GZIPInputStream because you will get
> a
> > > "java.io.EOFException: Unexpected end of ZLIB input stream".
> > > > Should I file a bug or correct the code ?
> > > >
> > > > phil
> > > Début du message réexpédié :
> > >
> > > > De : Philippe Van Dyck <pvdyck at gmail.com>
> > > > Date : 16 avril 2010 10:52:22 HAEC
> > > > À : infinispan -Dev List <infinispan-dev at lists.jboss.org>
> > > > Objet : Réexp : Bug : read after end of stream @
> AbstractMarshaller
> > > >
> > > > Ok, I took a closer look.
> > > > Actually, it is related to HTTPCORE-199, and this bug is fixed.
> > > > What about using int len = inputStream.available(); in
> > > AbstractMarshaller again ?
> > > >
> > > > phil
> > > >
> > > > ---------- Forwarded message ----------
> > > > From: Philippe Van Dyck <pvdyck at gmail.com>
> > > > Date: Fri, Apr 16, 2010 at 10:45 AM
> > > > Subject: Bug : read after end of stream @ AbstractMarshaller
> > > > To: infinispan -Dev List <infinispan-dev at lists.jboss.org>
> > > >
> > > >
> > > > Hi all,
> > > >
> > > > since I use an InflaterInputStream to send objects to S3 using
> > > JClouds Blobstore, I need a very strict management of streams.
> > > > In AbstractMarshaller,  while ((bytesRead =
> inputStream.read(buf, 0,
> > > buf.length)) != -1) bytes.write(buf, 0, bytesRead) will read after
> the
> > > stream's end, waiting for '-1' to happen.
> > > > You cannot do that with a GZIPInputStream because you will get
> a
> > > "java.io.EOFException: Unexpected end of ZLIB input stream".
> > > > Should I file a bug or correct the code ?
> > > >
> > > > phil
> > > >
> > > Début du message réexpédié :
> > >
> > > > De : Philippe Van Dyck <pvdyck at gmail.com>
> > > > Date : 16 avril 2010 11:50:54 HAEC
> > > > À : infinispan -Dev List <infinispan-dev at lists.jboss.org>
> > > > Objet : Réexp : Bug : read after end of stream @
> AbstractMarshaller
> > > >
> > > > Ok, JDK Bug
> > > http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6519463
> > > >
> > > > The workaround is to simply ignore (!) the EOFException...
> > > >
> > > > WDYT ?
> > > >
> > > > phil
> > > >
> > > > ---------- Forwarded message ----------
> > > > From: Philippe Van Dyck <pvdyck at gmail.com>
> > > > Date: Fri, Apr 16, 2010 at 10:52 AM
> > > > Subject: Fwd: Bug : read after end of stream @
> AbstractMarshaller
> > > > To: infinispan -Dev List <infinispan-dev at lists.jboss.org>
> > > >
> > > >
> > > > Ok, I took a closer look.
> > > > Actually, it is related to HTTPCORE-199, and this bug is fixed.
> > > > What about using int len = inputStream.available(); in
> > > AbstractMarshaller again ?
> > > >
> > > > phil
> > > >
> > > >
> > > > ---------- Forwarded message ----------
> > > > From: Philippe Van Dyck <pvdyck at gmail.com>
> > > > Date: Fri, Apr 16, 2010 at 10:45 AM
> > > > Subject: Bug : read after end of stream @ AbstractMarshaller
> > > > To: infinispan -Dev List <infinispan-dev at lists.jboss.org>
> > > >
> > > >
> > > > Hi all,
> > > >
> > > > since I use an InflaterInputStream to send objects to S3 using
> > > JClouds Blobstore, I need a very strict management of streams.
> > > > In AbstractMarshaller,  while ((bytesRead =
> inputStream.read(buf, 0,
> > > buf.length)) != -1) bytes.write(buf, 0, bytesRead) will read after
> the
> > > stream's end, waiting for '-1' to happen.
> > > > You cannot do that with a GZIPInputStream because you will get
> a
> > > "java.io.EOFException: Unexpected end of ZLIB input stream".
> > > > Should I file a bug or correct the code ?
> > > >
> > > > phil
> > > >
> > > >
> > >
> > >
> > > _______________________________________________
> > > infinispan-dev mailing list
> > > infinispan-dev at lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

More information about the infinispan-dev mailing list