[infinispan-dev] Hot Rod - pt3
Alex Kluge
java_kluge at yahoo.com
Mon Jan 4 16:44:23 EST 2010
> > - No events. I used events, that is asynchronous messages originating
> > from the server when certain conditions are met, to notify clients
> > that data was written to the server. This was mostly driven by the
> > need to notify an L1 cache of a write to the cache server. It would
> > be good to allow for this usage in this protocol. Note that this is
> > another case where the op code is useful to have as part of the
> > message.
>
> Isn't this expensive? Doesn't this mean the server
> has to fire off messages to each and every connected client,
> whether the clients are interested in these messages or
> not?
Not necessarily. There are a number of options. It wouldn't be
difficult to allow the clients to register for the events, and
then only send them to the interested clients. The events can
be sent asynchronously (a separate thread), thus they won't
delay the response to a write. Piratically speaking, they aren't
that expensive.
> > - What happens if the key or the value is not text? I have a way of
> > representing the data to allow for a wide variety of data types,
> > even allowing for arrays or maps. This will make the protocol more
> > complex, but the assumption that the data is a string is rather
> > limiting. This is already sketched out in the wiki.
>
> There is some work around non-text data in the RESTful
> cache server. Adding MIME types to any opaque data
> allows us to attach handlers to this data, such as XML or
> JSON data. What do you have in mind?
The idea is to prefix each data block with a data type. This is a
lightweight binary protocol, so a full fledged mime type would be
overkill. There is a discussion and example in the Encoding Data
section of this page:
http://community.jboss.org/wiki/RemoteCacheInteractions
Data types are limited to things like integer, byte, string, boolean, etc.
Or, if it isn't a recognised type, the native platform serialisation is
used. There can be arrays of these types, or maps as well.
Each data type is represented by a bit, and they can be used in
combinations. An array of bytes would have the array, byte and
primitive bits set. The set of recognised data types can of course be
expanded.
> > - Is the full message size still there as a header field? Is this
> > necessary? It precludes you from generating messages where you
> > don't know the full size of the message beforehand. For example,
> > in the future you might want to write a map of the data in the
> > cache. I do this, and can chunk the data, thus allowing me to send
> > an arbitrarily large map. This becomes, to put it mildly, difficult
> > if you need the size of the request as a header field.
>
> +1. If there is a way to work without this size
> header that would be better.
In my implementation I have a size attached to each field, and this
allows the messages to be handled easily. I retrieve more data from
the network if there is not enough data to complete the processing of
a field. There is no need to know the size of the full message.
Thanks,
Alex
--- On Mon, 1/4/10, Manik Surtani <manik at jboss.org> wrote:
> From: Manik Surtani <manik at jboss.org>
> Subject: Re: [infinispan-dev] Hot Rod - pt3
> To: "infinispan -Dev List" <infinispan-dev at lists.jboss.org>
> Date: Monday, January 4, 2010, 11:17 AM
>
> On 29 Dec 2009, at 18:56, Alex Kluge wrote:
>
> > Hi,
> >
> > I hope you have all enjoyed the holiday season.
> >
> > The protocol is an interesting read. There are a
> few things that stand
> > out.
> >
> > - No events. I used events, that is asynchronous
> messages originating
> > from the server when certain conditions
> are met, to notify clients
> > that data was written to the
> server. This was mostly driven by the
> > need to notify an L1 cache of a write to
> the cache server. It would
> > be good to allow for this usage in this
> protocol. Note that this is
> > another case where the op code is useful
> to have as part of the
> > message.
>
> Isn't this expensive? Doesn't this mean the server
> has to fire off messages to each and every connected client,
> whether the clients are interested in these messages or
> not?
>
> > - What happens if the key or the value is not
> text? I have a way of
> > representing the data to allow for a wide
> variety of data types,
> > even allowing for arrays or maps. This
> will make the protocol more
> > complex, but the assumption that the data
> is a string is rather
> > limiting. This is already sketched out in
> the wiki.
>
> There is some work around non-text data in the RESTful
> cache server. Adding MIME types to any opaque data
> allows us to attach handlers to this data, such as XML or
> JSON data. What do you have in mind?
>
> > - Is the full message size still there as a
> header field? Is this
> > necessary? It precludes you from
> generating messages where you
> > don't know the full size of the message
> beforehand. For example,
> > in the future you might want to write a
> map of the data in the
> > cache. I do this, and can chunk the data,
> thus allowing me to send
> > an arbitrarily large map. This becomes,
> to put it mildly, difficult
> > if you need the size of the request as a
> header field.
>
> +1. If there is a way to work without this size
> header that would be better.
>
> Cheers
> Manik
>
> >
> >
>
> Thanks,
> >
>
> Alex
> >
> > --- On Mon, 12/21/09, Manik Surtani <manik at jboss.org>
> wrote:
> >
> >> From: Manik Surtani <manik at jboss.org>
> >> Subject: Re: [infinispan-dev] Hot Rod - pt3
> >> To: "infinispan -Dev List" <infinispan-dev at lists.jboss.org>
> >> Date: Monday, December 21, 2009, 5:36 AM
> >>
> >> On 21 Dec 2009, at 11:08, Galder Zamarreno wrote:
> >>
> >>> Hi all,
> >>>
> >>> Re: http://community.jboss.org/wiki/HotRodProtocol
> >>>
> >>> First, I've made the corresponding changes
> based on
> >> the feedback I got
> >>> from pt2. This included reducing the response
> header
> >> since clients are
> >>> already aware of what they sent, addition of
> topology
> >> view id to to non
> >>> dumb requests and further specification in
> responses
> >> when topology
> >>> changes have happened...etc.
> >>>
> >>> I've also added flags to the request header
> that allow
> >> sending
> >>> Infinispan flags like: skip cache store, zero
> lock
> >> acquisition
> >>> timeout...etc.
> >>
> >>> Note that I've noted this as being N * 1 byte
> where
> >> each byte represents
> >>> a flag. However, I think this could maybe be
> sent more
> >> efficiently by
> >>> using XOR, i.e.
> >>>
> >>> 0x00 -> no flag
> >>> 0x01 (0000 0001) -> zero lock acquisition
> >>> 0x02 (0000 0010) -> cache mode local
> >>> 0x03 (0000 0011) -> zero lock acquisition +
> cache
> >> mode local
> >>> 0x04 (0000 0100) -> skip locking
> >>> ...etc.
> >>>
> >>> With 2 bytes, we could implement 16 Flags, we
> >> currently 11. However, we
> >>> could use vint as well, making sure that the
> most
> >> significant bit does
> >>> not mean anything flag wise. Iow, with vint,
> in 1 byte
> >> we'd be able to
> >>> define 7 diff flags. Thoughts?
> >>
> >> +1
> >>
> >>> I've also added the quit command that
> disconnects
> >> clients.
> >>
> >> Does this have any effect on the servers?
> >>
> >>> Finally, as far as I'm concerned, the
> specification is
> >> complete. I'm
> >>> leaving the quiet commands out of this initial
> scope
> >> (see
> >>> http://code.google.com/p/memcached/wiki/MemcacheBinaryProtocol).
> >>
> >>> Remember that quiet commands could be used so
> that the
> >> server buffers
> >>> responses and only when you send a non-quiet
> command,
> >> the server replies
> >>> with all the pending answers.
> >>>
> >>> As you can see at the bottom of the wiki, I've
> added a
> >> local only put
> >>> request/response example so that readers get
> an idea
> >> of what a full
> >>> command looks like. I had received some
> feedback from
> >> readers saying
> >>> that it was difficult to understand how it all
> fit
> >> together.
> >>
> >> The examples look good, however I would not use
> the
> >> LOCAL_ONLY flag since that flag really only has
> meaning in a
> >> p2p context. In a client-server context,
> this flag is
> >> useless, and probably meaningless. I would
> suggest a
> >> different flag for the example, e.g.,
> ZERO_LOCK_TIMEOUT.
> >>
> >>> I'll probably add a couple more examples for
> >> non-so-dumb and clever
> >>> request/responses but I'll held them until we
> have a
> >> final round of
> >>> feedback and people can indicate whether they
> want any
> >> other examples
> >>> appearing in the wiki.
> >>>
> >>> Cheers,
> >>> --
> >>> Galder Zamarreño
> >>> Sr. Software Engineer
> >>> Infinispan, JBoss Cache
> >>>
> _______________________________________________
> >>> infinispan-dev mailing list
> >>> infinispan-dev at lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >> --
> >> Manik Surtani
> >> manik at jboss.org
> >> Lead, Infinispan
> >> Lead, JBoss Cache
> >> http://www.infinispan.org
> >> http://www.jbosscache.org
> >>
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >
> >
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> --
> Manik Surtani
> manik at jboss.org
> Lead, Infinispan
> Lead, JBoss Cache
> http://www.infinispan.org
> http://www.jbosscache.org
>
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
More information about the infinispan-dev
mailing list