Regarding ISPN-29 [1], I've made some notes about what this will provide on this wiki page [2]. I'm kinda tired of referring to the 'custom binary protocol' as a 'custom binary protocol'! Can anyone think of a snazzy name for this protocol? Keep in mind that we would want others to implement clients using this protocol as well on other platforms. Here are a few thoughts to get the creative juices flowing: * ICBP (Infinispan cache binary protocol - BORING!) * Adhesive (the 'glue' between the client and server) * Elastiglue * StickyFingers (after the Rolling Stones album?) - Manik [1] https://jira.jboss.org/jira/browse/ISPN-29 [2] http://www.jboss.org/community/wiki/Clientandservermodules -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

From: Manik Surtani <manik(a)jboss.org&gt; Subject: Re: [infinispan-dev] ISPN-29 and a "custom protocol" To: "Jeff Ramsdale" <jeff.ramsdale(a)gmail.com&gt; Cc: infinispan-dev(a)lists.jboss.org Date: Wednesday, August 19, 2009, 12:05 PM Nice one Jeff ... so far this is winning in my mind! On 19 Aug 2009, at 18:01, Jeff Ramsdale wrote: How about Hot Rod? It has a connection to the word "custom" and implies speed... -jeff On Wed, Aug 19, 2009 at 7:05 AM, Manik Surtani <manik(a)jboss.org&gt; wrote: Regarding ISPN-29 [1], I've made some notes about what this will provide on this wiki page [2].  I'm kinda tired of referring to the 'custom binary protocol' as a 'custom binary protocol'!  Can anyone think of a snazzy name for this protocol?  Keep in mind that we would want others to implement clients using this protocol as well on other platforms. Here are a few thoughts to get the creative juices flowing:        * ICBP (Infinispan cache binary protocol - BORING!)        * Adhesive (the 'glue' between the client and server)        * Elastiglue        * StickyFingers (after the Rolling Stones album?) - Manik [1] https://jira.jboss.org/jira/browse/ISPN-29 [2] http://www.jboss.org/community/wiki/Clientandservermodules -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev --Manik Surtanimanik(a)jboss.orgLead, InfinispanLead, JBoss Cachehttp://www.infinispan.orghttp://www.jbosscache.org -----Inline Attachment Follows----- _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Is this protocol documented somewhere?

Galder is working on the HotRod protocol which may well benefit from this.

The plan is for HotRod to support (but not mandate) client-side CH

From: Manik Surtani <manik(a)jboss.org&gt; Subject: Re: [infinispan-dev] ISPN-29 and a "custom protocol" To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; Date: Monday, November 16, 2009, 5:06 AM Hi Alex - comments in line: On 16 Nov 2009, at 07:25, Alex Kluge wrote: > Hi, > >  It is worth mentioning that I have a full implementation of a > client-server binary protocol used for jboss-cache, which is similar > to the current project and can be easily adapted to it. There are a > number of interesting points > >   - Built on Apache MINA framework, but the layers are well separated, >     so replacing one would not be too difficult. Nice.  >   - Language neutral binary protocol (non Java clients are planned). Is this protocol documented somewhere?  Galder is working on the HotRod protocol which may well benefit from this. >   - Integrated into Jboss Cache as an L2 cache, but easily used >     independently. >   - Performance is quite reasonable, with request/response cycles on >     the order of 600 microseconds for small request/response. >   - Easily extensible, a different codec can be supplied to support >     different protocols. Some refactoring could be done to make this >     much easier. >   - Non trivially tested already. >   - Inherently asynchronous - synchronous responses are achieved by >     immediately waiting on the response future object. >   - Server side consistent hashing, clients connect to any server. The plan is for HotRod to support (but not mandate) client-side CH as well for "smart" connections. > There is raw source at >   http://www.vizitsolutions.com/org.jboss.cache.tcpcache.tar.gz > > I'll see about some explanations and examples over the next weeks. Great! > This was intended to be contributed back to the Jboss Cache project from the beginning, hence the organization into jboss.cache packages. Oh, and > I never gave it a snazzy name - I just called it the Jboss Cache binary > protocol. :) > >                                    Alex > > --- On Wed, 8/19/09, Manik Surtani <manik(a)jboss.org&gt; wrote: > >> From: Manik Surtani <manik(a)jboss.org&gt; >> Subject: Re: [infinispan-dev] ISPN-29 and a "custom protocol" >> To: "Jeff Ramsdale" <jeff.ramsdale(a)gmail.com&gt; >> Cc: infinispan-dev(a)lists.jboss.org >> Date: Wednesday, August 19, 2009, 12:05 PM >> Nice one Jeff ... so far this is >> winning in my mind! >> >> On 19 Aug 2009, at 18:01, Jeff Ramsdale >> wrote: >> How about Hot Rod? It has a connection to the >> word "custom" and implies speed... >> -jeff >> >> On Wed, Aug 19, 2009 at 7:05 AM, >> Manik Surtani <manik(a)jboss.org&gt; >> wrote: >> Regarding >> ISPN-29 [1], I've made some notes about what this will >> provide on this wiki page [2].  I'm kinda tired >> of referring to the >> 'custom binary protocol' as a 'custom binary >> protocol'!  Can anyone >> think of a snazzy name for this protocol?  Keep in >> mind that we would >> want others to implement clients using this protocol as >> well on other >> platforms. >> >> Here are a few thoughts to get the creative juices >> flowing: >> >>        * ICBP (Infinispan cache binary >> protocol - BORING!) >>        * Adhesive (the 'glue' >> between the client and server) >>        * Elastiglue >>        * StickyFingers (after the >> Rolling Stones album?) >> >> - Manik >> >> [1] https://jira.jboss.org/jira/browse/ISPN-29 >> [2] http://www.jboss.org/community/wiki/Clientandservermodules >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> --Manik >> Surtanimanik(a)jboss.orgLead, >> InfinispanLead, JBoss Cachehttp://www.infinispan.orghttp://www.jbosscache.org >> >> >> >> >> -----Inline Attachment Follows----- >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hi, > Is this protocol documented somewhere? Putting this online is one of the main things I will be able to do along this path over the next week. > Galder is working on the HotRod protocol which may well benefit > from this. Well, it may not be necessary to reinvent it. We'll see. Are there requirements for an infinispan protocol documented anywhere?

> The plan is for HotRod to support (but not mandate) > client-side CH I did this originally, however, I moved to a server side message routing for a few reasons. - It is easier to coordinate the smaller number of servers when a new one comes online, or goes offline.

- It allows fewer connections per server, client side routing requires that each client connect to each server. For our scale, thousands of servers, this is an issue.

- It guarantees a consistent mapping of data to servers for potentially disperate clients. - It makes the client side code easier to write.

That said, if the client does do the hashing, the server will still do it, and make sure that the data arrives at the correct server. So nothing prevents the client from doing the hashing. Alex --- On Mon, 11/16/09, Manik Surtani&lt;manik(a)jboss.org&gt; wrote: > From: Manik Surtani&lt;manik(a)jboss.org&gt; > Subject: Re: [infinispan-dev] ISPN-29 and a "custom protocol" > To: "infinispan -Dev List&quot;&lt;infinispan-dev(a)lists.jboss.org&gt; > Date: Monday, November 16, 2009, 5:06 AM > Hi Alex - comments in line: > > On 16 Nov 2009, at 07:25, Alex Kluge wrote: > >> Hi, >> >> It is worth mentioning that I have a full > implementation of a >> client-server binary protocol used for jboss-cache, > which is similar >> to the current project and can be easily adapted to > it. There are a >> number of interesting points >> >> - Built on Apache MINA framework, but > the layers are well separated, >> so replacing one would not be > too difficult. > > Nice. > >> - Language neutral binary protocol > (non Java clients are planned). > > Is this protocol documented somewhere? Galder is > working on the HotRod protocol which may well benefit from > this. > >> - Integrated into Jboss Cache as an > L2 cache, but easily used >> independently. >> - Performance is quite reasonable, > with request/response cycles on >> the order of 600 microseconds > for small request/response. >> - Easily extensible, a different > codec can be supplied to support >> different protocols. Some > refactoring could be done to make this >> much easier. >> - Non trivially tested already. >> - Inherently asynchronous - > synchronous responses are achieved by >> immediately waiting on the > response future object. >> - Server side consistent hashing, > clients connect to any server. > > The plan is for HotRod to support (but not mandate) > client-side CH as well for "smart" connections. > >> There is raw source at >> http://www.vizitsolutions.com/org.jboss.cache.tcpcache.tar.gz >> >> I'll see about some explanations and examples over the > next weeks. > > Great! > >> This was intended to be contributed back to the Jboss > Cache project from the beginning, hence the organization > into jboss.cache packages. Oh, and >> I never gave it a snazzy name - I just called it the > Jboss Cache binary >> protocol. > > :) > > >> >> > > Alex >> >> --- On Wed, 8/19/09, Manik Surtani&lt;manik(a)jboss.org&gt; > wrote: >> >>> From: Manik Surtani&lt;manik(a)jboss.org&gt; >>> Subject: Re: [infinispan-dev] ISPN-29 and a > "custom protocol" >>> To: "Jeff Ramsdale&quot;&lt;jeff.ramsdale(a)gmail.com&gt; >>> Cc: infinispan-dev(a)lists.jboss.org >>> Date: Wednesday, August 19, 2009, 12:05 PM >>> Nice one Jeff ... so far this is >>> winning in my mind! >>> >>> On 19 Aug 2009, at 18:01, Jeff Ramsdale >>> wrote: >>> How about Hot Rod? It has a connection to the >>> word "custom" and implies speed... >>> -jeff >>> >>> On Wed, Aug 19, 2009 at 7:05 AM, >>> Manik Surtani&lt;manik(a)jboss.org&gt; >>> wrote: >>> Regarding >>> ISPN-29 [1], I've made some notes about what this > will >>> provide on this wiki page [2]. I'm kinda > tired >>> of referring to the >>> 'custom binary protocol' as a 'custom binary >>> protocol'! Can anyone >>> think of a snazzy name for this protocol? > Keep in >>> mind that we would >>> want others to implement clients using this > protocol as >>> well on other >>> platforms. >>> >>> Here are a few thoughts to get the creative > juices >>> flowing: >>> >>> * ICBP (Infinispan > cache binary >>> protocol - BORING!) >>> * Adhesive (the 'glue' >>> between the client and server) >>> * Elastiglue >>> * StickyFingers (after > the >>> Rolling Stones album?) >>> >>> - Manik >>> >>> [1] https://jira.jboss.org/jira/browse/ISPN-29 >>> [2] http://www.jboss.org/community/wiki/Clientandservermodules >>> -- >>> Manik Surtani >>> manik(a)jboss.org >>> Lead, Infinispan >>> Lead, JBoss Cache >>> http://www.infinispan.org >>> http://www.jbosscache.org >>> >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> >>> --Manik >>> Surtanimanik(a)jboss.orgLead, >>> InfinispanLead, JBoss Cachehttp://www.infinispan.orghttp://www.jbosscache.org >>> >>> >>> >>> >>> -----Inline Attachment Follows----- >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hi, I've been a little busy recently tracking what appears to be an issue with JBoss Cache under significant concurrent load, but I have had some time to look at this. >> Are there requirements for an infinispan >> protocol documented anywhere? > > Manik and I discussed the extra bits that we think the > binary protocol > should contain and these are now explained in: > http://www.jboss.org/community/wiki/HotRodProtocol I don see anything here that would be particularly difficult. The only thing that confuses me is the exact meaning of "piggy back". It sounds like "piggy back" as used here refers to overloading a response and providing additional information in some cases.

I tend to like clearly defined requests and responses, so clients can ask for information that they want. There are also events, so that when something changes, the clients can receive asynchronous events.

Perhaps if I had a slightly better understanding of the desired behaviour I could fit the protocol to it.

>> - It is easier to coordinate the smaller number of servers when a new >> one comes online, or goes offline. > Hmmmm, I'm not sure I understand why this would be a problem for client > side routing. The client's would get new cluster formation information > next time they make a call. IOW, the response to the client would > contain cluster formation info of the nodes configured with hot rod > server, so no need to coordinate server join/leaves with clients. This > is how clustered EJBs work in AS. Consider the case where you have a thousand clients, and three servers. Each client sends a put or get request to the appropriate server according to a consistent hash. Now, add a new server. If the clients are doing the CH, then they all need to be consistent at the same time. Otherwise one client may write data to one server, and another client may read or write the same data to another server.

Consistency among a large number of clients becomes a hard problem. It is simply easier to manage the dynamics when you have a simpler system under tighter control, such as a set of servers.

Alex --- On Fri, 11/20/09, Galder Zamarreno <galder(a)redhat.com&gt; wrote: > From: Galder Zamarreno <galder(a)redhat.com&gt; > Subject: Re: [infinispan-dev] ISPN-29 and a "custom protocol" > To: infinispan-dev(a)lists.jboss.org > Date: Friday, November 20, 2009, 4:50 AM > Hi Alex, > > On 11/19/2009 06:02 PM, Alex Kluge wrote: >> Hi, >> >> > Is this protocol > documented somewhere? >> >> Putting this online is one of the > main things I will be able to do >> along this path over the next week. >> >> > Galder is working on the > HotRod protocol which may well benefit >> > from this. >> >> Well, it may not be necessary to > reinvent it. We'll see. Are there >> requirements for an infinispan > protocol documented anywhere? > > Manik and I discussed the extra bits that we think the > binary protocol > should contain and these are now explained in: > http://www.jboss.org/community/wiki/HotRodProtocol > > If you can add docu of your binary protocol there, we can > look into > accomodation this stuff into your binary protocol > potentially? > >> >> > The plan is for HotRod to > support (but not mandate) >> > client-side CH >> >> I did this originally, however, I > moved to a server side message >> routing for a few reasons. >> >> - It is easier to coordinate the > smaller number of servers when a new >> one comes online, or goes > offline. > > Hmmmm, I'm not sure I understand why this would be a > problem for client > side routing. The client's would get new cluster formation > information > next time they make a call. IOW, the response to the client > would > contain cluster formation info of the nodes configured with > hot rod > server, so no need to coordinate server join/leaves with > clients. This > is how clustered EJBs work in AS. > >> >> - It allows fewer connections per > server, client side routing requires >> that each client connect to > each server. For our scale, thousands of >> servers, this is an issue. > > That's a good point. > >> >> - It guarantees a consistent mapping > of data to servers for potentially >> disperate clients. >> >> - It makes the client side code > easier to write. > > True :) > >> >> >> That said, if the client does do the > hashing, the server will still >> do it, and make sure that the data > arrives at the correct server. So >> nothing prevents the client from > doing the hashing. >> >> > > Alex >> >> >> --- On Mon, 11/16/09, Manik Surtani&lt;manik(a)jboss.org&gt; > wrote: >> >>> From: Manik Surtani&lt;manik(a)jboss.org&gt; >>> Subject: Re: [infinispan-dev] ISPN-29 and a > "custom protocol" >>> To: "infinispan -Dev List&quot;&lt;infinispan-dev(a)lists.jboss.org&gt; >>> Date: Monday, November 16, 2009, 5:06 AM >>> Hi Alex - comments in line: >>> >>> On 16 Nov 2009, at 07:25, Alex Kluge wrote: >>> >>>> Hi, >>>> >>>> It is worth mentioning that I > have a full >>> implementation of a >>>> client-server binary protocol used for > jboss-cache, >>> which is similar >>>> to the current project and can be easily > adapted to >>> it. There are a >>>> number of interesting points >>>> >>>> - Built on Apache MINA > framework, but >>> the layers are well separated, >>>> so replacing > one would not be >>> too difficult. >>> >>> Nice. >>> >>>> - Language neutral > binary protocol >>> (non Java clients are planned). >>> >>> Is this protocol documented somewhere? > Galder is >>> working on the HotRod protocol which may well > benefit from >>> this. >>> >>>> - Integrated into > Jboss Cache as an >>> L2 cache, but easily used >>>> independently. >>>> - Performance is quite > reasonable, >>> with request/response cycles on >>>> the order of > 600 microseconds >>> for small request/response. >>>> - Easily extensible, a > different >>> codec can be supplied to support >>>> different > protocols. Some >>> refactoring could be done to make this >>>> much easier. >>>> - Non trivially tested > already. >>>> - Inherently > asynchronous - >>> synchronous responses are achieved by >>>> immediately > waiting on the >>> response future object. >>>> - Server side > consistent hashing, >>> clients connect to any server. >>> >>> The plan is for HotRod to support (but not > mandate) >>> client-side CH as well for "smart" connections. >>> >>>> There is raw source at >>>> http://www.vizitsolutions.com/org.jboss.cache.tcpcache.tar.gz >>>> >>>> I'll see about some explanations and examples > over the >>> next weeks. >>> >>> Great! >>> >>>> This was intended to be contributed back to > the Jboss >>> Cache project from the beginning, hence the > organization >>> into jboss.cache packages. Oh, and >>>> I never gave it a snazzy name - I just called > it the >>> Jboss Cache binary >>>> protocol. >>> >>> :) >>> >>> >>>> >>>> >>> >>> Alex >>>> >>>> --- On Wed, 8/19/09, Manik Surtani&lt;manik(a)jboss.org&gt; >>> wrote: >>>> >>>>> From: Manik Surtani&lt;manik(a)jboss.org&gt; >>>>> Subject: Re: [infinispan-dev] ISPN-29 and > a >>> "custom protocol" >>>>> To: "Jeff Ramsdale&quot;&lt;jeff.ramsdale(a)gmail.com&gt; >>>>> Cc: infinispan-dev(a)lists.jboss.org >>>>> Date: Wednesday, August 19, 2009, 12:05 > PM >>>>> Nice one Jeff ... so far this is >>>>> winning in my mind! >>>>> >>>>> On 19 Aug 2009, at 18:01, Jeff Ramsdale >>>>> wrote: >>>>> How about Hot Rod? It has a connection to > the >>>>> word "custom" and implies speed... >>>>> -jeff >>>>> >>>>> On Wed, Aug 19, 2009 at 7:05 AM, >>>>> Manik Surtani&lt;manik(a)jboss.org&gt; >>>>> wrote: >>>>> Regarding >>>>> ISPN-29 [1], I've made some notes about > what this >>> will >>>>> provide on this wiki page [2]. I'm > kinda >>> tired >>>>> of referring to the >>>>> 'custom binary protocol' as a 'custom > binary >>>>> protocol'! Can anyone >>>>> think of a snazzy name for this protocol? >>> Keep in >>>>> mind that we would >>>>> want others to implement clients using > this >>> protocol as >>>>> well on other >>>>> platforms. >>>>> >>>>> Here are a few thoughts to get the > creative >>> juices >>>>> flowing: >>>>> >>>>> * ICBP > (Infinispan >>> cache binary >>>>> protocol - BORING!) >>>>> * > Adhesive (the 'glue' >>>>> between the client and server) >>>>> * > Elastiglue >>>>> * > StickyFingers (after >>> the >>>>> Rolling Stones album?) >>>>> >>>>> - Manik >>>>> >>>>> [1] https://jira.jboss.org/jira/browse/ISPN-29 >>>>> [2] http://www.jboss.org/community/wiki/Clientandservermodules >>>>> -- >>>>> Manik Surtani >>>>> manik(a)jboss.org >>>>> Lead, Infinispan >>>>> Lead, JBoss Cache >>>>> http://www.infinispan.org >>>>> http://www.jbosscache.org >>>>> >>>>> >>>>> >>>>> >>>>> > _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>>> >>>>> --Manik >>>>> Surtanimanik(a)jboss.orgLead, >>>>> InfinispanLead, JBoss Cachehttp://www.infinispan.orghttp://www.jbosscache.org >>>>> >>>>> >>>>> >>>>> >>>>> -----Inline Attachment Follows----- >>>>> >>>>> > _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> >>>> >>>> >>>> > _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> -- >>> Manik Surtani >>> manik(a)jboss.org >>> Lead, Infinispan >>> Lead, JBoss Cache >>> http://www.infinispan.org >>> http://www.jbosscache.org >>> >>> >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

HI, > I tend to like clearly defined requests and responses, so clients > can ask for information that they want. There are also events, so > that when something changes, the clients can receive asynchronous > events. > The problem with that approach is the way connections work. > Explicit > server -> client messages would require either open > client-side ServerSockets, or persistent connections which cannot > be guaranteed if there is a firewall separating things. In reading this you may notice that I favor use case driven design :) Yes, it is difficult to ensure that a client has accurate information, especially about the server topology. That is what drove me to the server side routing approach. For example. a client connects for the first time, how do they get the server topology? I would suggest either it be unneeded (the server does the routing), or the first thing they send is a request for it.

The same question applies to a client that has been disconnected for a period of time. The available solutions are a bit different here, because the client may have a server topology, but it may be out of date. So if we expect the client to be able to do the routing, it will have to provide a topology version to the server with the requests, and expect an update back when the version is out of date. Or, it could simply send a request for a topology update as the first request. > This is actually dealt-with. When a rebalance occurs, > requests sent to the "old" owner are proxied on to the new > one. Transparent to the caller. When could this stop? That is, when would you be guaranteed that all clients have an up to date topology? Or is this always available? If it is always available, how much value is provided by having the client determine the server to rout the request to?

Does the client attempt to maintain connections to all the servers? Or does it open and close connections for each request?

> persistent connections which cannot be guaranteed if there is a > firewall separating things. How prevalent is this case? I have used and seen a remote cache in several applications, and it has always been comparatively local to the application. If a reasonably long lived connection is not available, a lot of issues arise. Consider the use as an L2 cache, how can you guarantee the consistancy of the various L1 caches if they do not receive events from the L2 cache when data is written? Indeed, the protocol that I have is designed around the L2 cache usage scenario. I'll see what I can do about documenting it...

Alex --- On Wed, 11/25/09, Manik Surtani <manik(a)jboss.org&gt; wrote: > From: Manik Surtani <manik(a)jboss.org&gt; > Subject: Re: [infinispan-dev] HotRod?? [ISPN-29 and a "custom protocol"] > To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; > Date: Wednesday, November 25, 2009, 5:01 AM > > On 24 Nov 2009, at 22:25, Alex Kluge wrote: > >> Hi, >> >> I've been a little busy recently tracking what > appears to be an issue >> with JBoss Cache under significant concurrent load, > but I have had some >> time to look at this. >> >>>> Are there requirements for an infinispan >>>> protocol documented anywhere? >>> >>> Manik and I discussed the extra bits that we think > the >>> binary protocol >>> should contain and these are now explained in: >>> http://www.jboss.org/community/wiki/HotRodProtocol >> >> I don see anything here that would be particularly > difficult. >> The only thing that confuses me is the exact meaning > of "piggy back". >> It sounds like "piggy back" as used here refers to > overloading a >> response and providing additional information in some > cases. > > That is precisely what it means. :-) > >> I tend >> to like clearly defined requests and responses, so > clients can ask >> for information that they want. There are also events, > so that when >> something changes, the clients can receive > asynchronous events. > > The problem with that approach is the way connections > work. Explicit server -> client messages would > require either open client-side ServerSockets, or persistent > connections which cannot be guaranteed if there is a > firewall separating things. > >> Perhaps if I had a slightly better understanding of > the desired >> behaviour I could fit the protocol to it. > > Sending additional information as a piggyback to usual > payload is a common technique, this is how HA EJB connectors > are commonly implemented in app servers. Essentially all > response streams a client may see would be in the following > format: > > [num-headers: > byte][headers][response-size: int][response: byte array] > > On the client, when parsing the stream, the response is > parsed and passed up to higher layers, along with a map of > headers which may contain additional info such as > server-side topology, which can be used to update the > client's list of targets and will affect client-side logic > when choosing where to send future requests. > >> >>>> - It is easier to coordinate the smaller > number of servers when a new >>>> one comes online, or goes > offline. >> >>> Hmmmm, I'm not sure I understand why this would be > a problem for client >>> side routing. The client's would get new cluster > formation information >>> next time they make a call. IOW, the response to > the client would >>> contain cluster formation info of the nodes > configured with hot rod >>> server, so no need to coordinate server > join/leaves with clients. This >>> is how clustered EJBs work in AS. >> >> Consider the case where you have a thousand clients, > and three servers. >> Each client sends a put or get request to the > appropriate server according >> to a consistent hash. >> >> Now, add a new server. If the clients are doing the > CH, then they all >> need to be consistent at the same time. Otherwise one > client may write >> data to one server, and another client may read or > write the same data >> to another server. > > This is actually dealt-with. When a rebalance occurs, > requests sent to the "old" owner are proxied on to the new > one. Transparent to the caller. > >> Consistency among a large number of clients becomes >> a hard problem. It is simply easier to manage the > dynamics when you >> have a simpler system under tighter control, such as a > set of servers. > > Right, and in such a case you may "dumb down" the > client. The way we see it, there are 3 degrees of > "intelligence" in clients, and this "degree of intelligence" > could be encoded (a single byte?) in the request when > talking to servers. > > 1. Dumb. Just spray requests to a load balancer which > uses a round-robin policy to talk to servers. Very > simple clients, only configured with a single endpoint which > is the LB. No extra information sent back to clients > from the servers. > > 2. Semi-smart: Clients have knowledge of server > topology, but clients do not engage in CH. No LB > needed, clients round-robin around the servers, potentially > use session affinity, clients resilient to connection > failure (can fail over). Servers need to send back > server topology whenever there are changes to the backend. > > 3. Smart: Clients have knowledge of server topology > as well as the CH function in effect. Clients can > direct requests to specific back-end nodes based on keys > involved, can load balance and fail over. Servers need > to send back server topology and hash codes of each server > node whenever there are changes to the backend. > > Good discussions anyway, lets keep these going... > > Cheers > Manik > >> >> > > Alex >> >> >> >> --- On Fri, 11/20/09, Galder Zamarreno <galder(a)redhat.com&gt; > wrote: >> >>> From: Galder Zamarreno <galder(a)redhat.com&gt; >>> Subject: Re: [infinispan-dev] ISPN-29 and a > "custom protocol" >>> To: infinispan-dev(a)lists.jboss.org >>> Date: Friday, November 20, 2009, 4:50 AM >>> Hi Alex, >>> >>> On 11/19/2009 06:02 PM, Alex Kluge wrote: >>>> Hi, >>>> >>>> > Is this protocol >>> documented somewhere? >>>> >>>> Putting this online is one of > the >>> main things I will be able to do >>>> along this path over the next > week. >>>> >>>> > Galder is working on > the >>> HotRod protocol which may well benefit >>>> > from this. >>>> >>>> Well, it may not be necessary to >>> reinvent it. We'll see. Are there >>>> requirements for an infinispan >>> protocol documented anywhere? >>> >>> Manik and I discussed the extra bits that we think > the >>> binary protocol >>> should contain and these are now explained in: >>> http://www.jboss.org/community/wiki/HotRodProtocol >>> >>> If you can add docu of your binary protocol there, > we can >>> look into >>> accomodation this stuff into your binary protocol >>> potentially? >>> >>>> >>>> > The plan is for HotRod > to >>> support (but not mandate) >>>> > client-side CH >>>> >>>> I did this originally, however, > I >>> moved to a server side message >>>> routing for a few reasons. >>>> >>>> - It is easier to coordinate the >>> smaller number of servers when a new >>>> one comes online, or goes >>> offline. >>> >>> Hmmmm, I'm not sure I understand why this would be > a >>> problem for client >>> side routing. The client's would get new cluster > formation >>> information >>> next time they make a call. IOW, the response to > the client >>> would >>> contain cluster formation info of the nodes > configured with >>> hot rod >>> server, so no need to coordinate server > join/leaves with >>> clients. This >>> is how clustered EJBs work in AS. >>> >>>> >>>> - It allows fewer connections > per >>> server, client side routing requires >>>> that each client connect > to >>> each server. For our scale, thousands of >>>> servers, this is an > issue. >>> >>> That's a good point. >>> >>>> >>>> - It guarantees a consistent > mapping >>> of data to servers for potentially >>>> disperate clients. >>>> >>>> - It makes the client side code >>> easier to write. >>> >>> True :) >>> >>>> >>>> >>>> That said, if the client does do > the >>> hashing, the server will still >>>> do it, and make sure that the > data >>> arrives at the correct server. So >>>> nothing prevents the client from >>> doing the hashing. >>>> >>>> > >>> > >>> > Alex >>>> >>>> >>>> --- On Mon, 11/16/09, Manik Surtani&lt;manik(a)jboss.org&gt; > >>> wrote: >>>> >>>>> From: Manik Surtani&lt;manik(a)jboss.org&gt; >>>>> Subject: Re: [infinispan-dev] ISPN-29 and > a >>> "custom protocol" >>>>> To: "infinispan -Dev List&quot;&lt;infinispan-dev(a)lists.jboss.org&gt; >>>>> Date: Monday, November 16, 2009, 5:06 AM >>>>> Hi Alex - comments in line: >>>>> >>>>> On 16 Nov 2009, at 07:25, Alex Kluge > wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> It is worth > mentioning that I >>> have a full >>>>> implementation of a >>>>>> client-server binary protocol used > for >>> jboss-cache, >>>>> which is similar >>>>>> to the current project and can be > easily >>> adapted to >>>>> it. There are a >>>>>> number of interesting points >>>>>> >>>>>> - Built on Apache > MINA >>> framework, but >>>>> the layers are well separated, >>>>>> so > replacing >>> one would not be >>>>> too difficult. >>>>> >>>>> Nice. >>>>> >>>>>> - Language > neutral >>> binary protocol >>>>> (non Java clients are planned). >>>>> >>>>> Is this protocol documented somewhere? >>> Galder is >>>>> working on the HotRod protocol which may > well >>> benefit from >>>>> this. >>>>> >>>>>> - Integrated into >>> Jboss Cache as an >>>>> L2 cache, but easily used >>>>>> > independently. >>>>>> - Performance is > quite >>> reasonable, >>>>> with request/response cycles on >>>>>> the order > of >>> 600 microseconds >>>>> for small request/response. >>>>>> - Easily > extensible, a >>> different >>>>> codec can be supplied to support >>>>>> different >>> protocols. Some >>>>> refactoring could be done to make this >>>>>> much > easier. >>>>>> - Non trivially > tested >>> already. >>>>>> - Inherently >>> asynchronous - >>>>> synchronous responses are achieved by >>>>>> > immediately >>> waiting on the >>>>> response future object. >>>>>> - Server side >>> consistent hashing, >>>>> clients connect to any server. >>>>> >>>>> The plan is for HotRod to support (but > not >>> mandate) >>>>> client-side CH as well for "smart" > connections. >>>>> >>>>>> There is raw source at >>>>>> http://www.vizitsolutions.com/org.jboss.cache.tcpcache.tar.gz >>>>>> >>>>>> I'll see about some explanations and > examples >>> over the >>>>> next weeks. >>>>> >>>>> Great! >>>>> >>>>>> This was intended to be contributed > back to >>> the Jboss >>>>> Cache project from the beginning, hence > the >>> organization >>>>> into jboss.cache packages. Oh, and >>>>>> I never gave it a snazzy name - I just > called >>> it the >>>>> Jboss Cache binary >>>>>> protocol. >>>>> >>>>> :) >>>>> >>>>> >>>>>> >>>>>> >>>>> >>>>> Alex >>>>>> >>>>>> --- On Wed, 8/19/09, Manik > Surtani&lt;manik(a)jboss.org&gt; >>>>> wrote: >>>>>> >>>>>>> From: Manik Surtani&lt;manik(a)jboss.org&gt; >>>>>>> Subject: Re: [infinispan-dev] > ISPN-29 and >>> a >>>>> "custom protocol" >>>>>>> To: "Jeff Ramsdale&quot;&lt;jeff.ramsdale(a)gmail.com&gt; >>>>>>> Cc: infinispan-dev(a)lists.jboss.org >>>>>>> Date: Wednesday, August 19, 2009, > 12:05 >>> PM >>>>>>> Nice one Jeff ... so far this is >>>>>>> winning in my mind! >>>>>>> >>>>>>> On 19 Aug 2009, at 18:01, Jeff > Ramsdale >>>>>>> wrote: >>>>>>> How about Hot Rod? It has a > connection to >>> the >>>>>>> word "custom" and implies > speed... >>>>>>> -jeff >>>>>>> >>>>>>> On Wed, Aug 19, 2009 at 7:05 AM, >>>>>>> Manik Surtani&lt;manik(a)jboss.org&gt; >>>>>>> wrote: >>>>>>> Regarding >>>>>>> ISPN-29 [1], I've made some notes > about >>> what this >>>>> will >>>>>>> provide on this wiki page > [2]. I'm >>> kinda >>>>> tired >>>>>>> of referring to the >>>>>>> 'custom binary protocol' as a > 'custom >>> binary >>>>>>> protocol'! Can anyone >>>>>>> think of a snazzy name for this > protocol? >>>>> Keep in >>>>>>> mind that we would >>>>>>> want others to implement clients > using >>> this >>>>> protocol as >>>>>>> well on other >>>>>>> platforms. >>>>>>> >>>>>>> Here are a few thoughts to get > the >>> creative >>>>> juices >>>>>>> flowing: >>>>>>> >>>>>>> > * ICBP >>> (Infinispan >>>>> cache binary >>>>>>> protocol - BORING!) >>>>>>> > * >>> Adhesive (the 'glue' >>>>>>> between the client and server) >>>>>>> > * >>> Elastiglue >>>>>>> > * >>> StickyFingers (after >>>>> the >>>>>>> Rolling Stones album?) >>>>>>> >>>>>>> - Manik >>>>>>> >>>>>>> [1] https://jira.jboss.org/jira/browse/ISPN-29 >>>>>>> [2] http://www.jboss.org/community/wiki/Clientandservermodules >>>>>>> -- >>>>>>> Manik Surtani >>>>>>> manik(a)jboss.org >>>>>>> Lead, Infinispan >>>>>>> Lead, JBoss Cache >>>>>>> http://www.infinispan.org >>>>>>> http://www.jbosscache.org >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>> _______________________________________________ >>>>>>> infinispan-dev mailing list >>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>> >>>>>>> >>>>>>> --Manik >>>>>>> Surtanimanik(a)jboss.orgLead, >>>>>>> InfinispanLead, JBoss Cachehttp://www.infinispan.orghttp://www.jbosscache.org >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -----Inline Attachment > Follows----- >>>>>>> >>>>>>> >>> _______________________________________________ >>>>>>> infinispan-dev mailing list >>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>>> -- >>>>> Manik Surtani >>>>> manik(a)jboss.org >>>>> Lead, Infinispan >>>>> Lead, JBoss Cache >>>>> http://www.infinispan.org >>>>> http://www.jbosscache.org >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> > _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>> >>>> >>>> >>>> >>>> > _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> -- >>> Galder Zamarreño >>> Sr. Software Engineer >>> Infinispan, JBoss Cache >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hi, >> I'll see what I can do about documenting it... > > Yeah lets do that - I would recommend using the JBoss.org > wiki pages so that the designs can be distilled by all of us > and then linked to from the other Infinispan design pages. I've started this at http://www.jboss.org/community/wiki/RemoteCacheInteractions I will start with a description of what I have, which is a good multilevel cache implementation. Then get to a description of some of the additional motivations and requirements for a data grid, which is a bit more general, and use that to motivate changes in the protocol. Alex _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hi all, I've updated with a very early draft of the hot rod protocol: http://www.jboss.org/community/wiki/HotRodProtocol As a reference, I've been using http://code.google.com/p/memcached/wiki/MemcacheBinaryProtocol to see what memcached did. Alex, as I emailed you yesterday, it'd be interesting to see what you did too. First, several operations are still missing from the spec: remove, removeIfPresent, *Async ops (putAsync, removeAsync...etc), putAll, putForExternalRead, evict, version, name (get cache's name). Also, I still need to think how to add flags into each operation sent across the network. There're some other operations that I'm consider whether to include or not. Here's what I'm thinking at the moment: - keySet, entrySet, values: I don't plan to include them since they'd be very expensive both from a cache iteration perspective and from sending that stuff back to the client. - containsKey: I don't plan to include it, you can figure it out with get command. - containsValue: No plan to include it since we don't support it. - size, isEmpty: Maybe include them as part of stats command like memcached does? Also, it might be wised to implement operations like replaceIfEquals using with a cas long, like memcached does (http://code.google.com/p/memcached/wiki/MemcacheBinaryProtocol), rather than sending the value cos that would be more network efficient. The old value could be much more longer. Another thing to note here is that for expiry and maxIdle, I followed the Java TimeUnit based definition. To make it easier for client implementers, we could follow the memcached style, where seconds are used always and if the number of seconds is bigger than 30 days, those seconds represent UNIX time (number of seconds since 1st January 1970). WDYT? Finally, memcached has this notion of getq where, as far as I can understand the spec, it's a quiet operation where the server holds up the return until you send a non quiet operation, i.e. a get or a noop. I need to look further into it to understand how exactly works, but it'd be interesting to know the thoughts of anyone who's dealt with this already. Cheers, On 12/03/2009 06:22 PM, Manik Surtani wrote: > Great, thanks Alex! > > On 3 Dec 2009, at 05:00, Alex Kluge wrote: > > >> Hi, >> >> >>>> I'll see what I can do about documenting it... >>>> >>> Yeah lets do that - I would recommend using the JBoss.org >>> wiki pages so that the designs can be distilled by all of us >>> and then linked to from the other Infinispan design pages. >>> >> I've started this at >> >> http://www.jboss.org/community/wiki/RemoteCacheInteractions >> >> I will start with a description of what I have, which is a good multilevel cache implementation. Then get to a description of some of the additional motivations and requirements for a data grid, which is a bit more general, and use that to motivate changes in the protocol. >> >> Alex >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev >

Quick note. You don't specify TCP/UDP, though I assume it's intended to be TCP? - DML On 12/08/2009 01:36 PM, Galder Zamarreno wrote: > Hi all, > > I've updated with a very early draft of the hot rod protocol: > http://www.jboss.org/community/wiki/HotRodProtocol > > As a reference, I've been using > http://code.google.com/p/memcached/wiki/MemcacheBinaryProtocol to see > what memcached did. Alex, as I emailed you yesterday, it'd be > interesting to see what you did too. > > First, several operations are still missing from the spec: remove, > removeIfPresent, *Async ops (putAsync, removeAsync...etc), putAll, > putForExternalRead, evict, version, name (get cache's name). > > Also, I still need to think how to add flags into each operation sent > across the network. > > There're some other operations that I'm consider whether to include or > not. Here's what I'm thinking at the moment: > > - keySet, entrySet, values: I don't plan to include them since they'd be > very expensive both from a cache iteration perspective and from sending > that stuff back to the client. > > - containsKey: I don't plan to include it, you can figure it out with > get command. > > - containsValue: No plan to include it since we don't support it. > > - size, isEmpty: Maybe include them as part of stats command like > memcached does? > > Also, it might be wised to implement operations like replaceIfEquals > using with a cas long, like memcached does > (http://code.google.com/p/memcached/wiki/MemcacheBinaryProtocol), rather > than sending the value cos that would be more network efficient. The old > value could be much more longer. > > Another thing to note here is that for expiry and maxIdle, I followed > the Java TimeUnit based definition. To make it easier for client > implementers, we could follow the memcached style, where seconds are > used always and if the number of seconds is bigger than 30 days, those > seconds represent UNIX time (number of seconds since 1st January 1970). > WDYT? > > Finally, memcached has this notion of getq where, as far as I can > understand the spec, it's a quiet operation where the server holds up > the return until you send a non quiet operation, i.e. a get or a noop. I > need to look further into it to understand how exactly works, but it'd > be interesting to know the thoughts of anyone who's dealt with this already. > > Cheers, > > On 12/03/2009 06:22 PM, Manik Surtani wrote: >> Great, thanks Alex! >> >> On 3 Dec 2009, at 05:00, Alex Kluge wrote: >> >>> Hi, >>> >>>>> I'll see what I can do about documenting it... >>>> >>>> Yeah lets do that - I would recommend using the JBoss.org >>>> wiki pages so that the designs can be distilled by all of us >>>> and then linked to from the other Infinispan design pages. >>> >>> I've started this at >>> >>> http://www.jboss.org/community/wiki/RemoteCacheInteractions >>> >>> I will start with a description of what I have, which is a good multilevel cache implementation. Then get to a description of some of the additional motivations and requirements for a data grid, which is a bit more general, and use that to motivate changes in the protocol. >>> >>> Alex >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 8 Dec 2009, at 19:36, Galder Zamarreno wrote: > Hi all, > > I've updated with a very early draft of the hot rod protocol: > http://www.jboss.org/community/wiki/HotRodProtocol > > As a reference, I've been using > http://code.google.com/p/memcached/wiki/MemcacheBinaryProtocol to see > what memcached did. Alex, as I emailed you yesterday, it'd be > interesting to see what you did too. > > First, several operations are still missing from the spec: remove, > removeIfPresent, *Async ops (putAsync, removeAsync...etc), putAll, > putForExternalRead, evict, version, name (get cache's name). > > Also, I still need to think how to add flags into each operation sent > across the network. > > There're some other operations that I'm consider whether to include or > not. Here's what I'm thinking at the moment: > > - keySet, entrySet, values: I don't plan to include them since they'd be > very expensive both from a cache iteration perspective and from sending > that stuff back to the client. Yes, I think this is unnecessary. > - containsKey: I don't plan to include it, you can figure it out with > get command. This would be much cheaper than a GET, from a network overhead perspective.

> - containsValue: No plan to include it since we don't support it. Agreed. > - size, isEmpty: Maybe include them as part of stats command like > memcached does? Perhaps. > Also, it might be wised to implement operations like replaceIfEquals > using with a cas long, like memcached does > (http://code.google.com/p/memcached/wiki/MemcacheBinaryProtocol), rather > than sending the value cos that would be more network efficient. The old > value could be much more longer. Not sure I understand. So we have 2 replace methods (thanks to ConcurrentMap) - replace(k, v) which succeeds if k is mapped to something, and replace(k, v1, v2) which succeeds if k is mapped to v1. From what I see, only the former is supported by memcached's binary protocol and not the latter. Are you suggesting supporting the latter in HotRod?

> Another thing to note here is that for expiry and maxIdle, I followed > the Java TimeUnit based definition. To make it easier for client > implementers, we could follow the memcached style, where seconds are > used always and if the number of seconds is bigger than 30 days, those > seconds represent UNIX time (number of seconds since 1st January 1970). > WDYT? This is done to reduce bytes on the wire, since expressing something in millis often means extra bytes and often meaningless since network latency can be worth several hundred millis already. So it makes sense for the protocol to express this in seconds. Not sure I understand your comment re: seconds> 30 days. Are you referring to the time being treated as relative to the UNIX epoch [1] as opposed to relative to the current time snapshot?

[1] http://www.techmount.com/index.php/20040830/epoch-time-unix-time-stamp > Finally, memcached has this notion of getq where, as far as I can > understand the spec, it's a quiet operation where the server holds up > the return until you send a non quiet operation, i.e. a get or a noop. I > need to look further into it to understand how exactly works, but it'd > be interesting to know the thoughts of anyone who's dealt with this already. > > Cheers, > > On 12/03/2009 06:22 PM, Manik Surtani wrote: >> Great, thanks Alex! >> >> On 3 Dec 2009, at 05:00, Alex Kluge wrote: >> >>> Hi, >>> >>>>> I'll see what I can do about documenting it... >>>> >>>> Yeah lets do that - I would recommend using the JBoss.org >>>> wiki pages so that the designs can be distilled by all of us >>>> and then linked to from the other Infinispan design pages. >>> >>> I've started this at >>> >>> http://www.jboss.org/community/wiki/RemoteCacheInteractions >>> >>> I will start with a description of what I have, which is a good multilevel cache implementation. Then get to a description of some of the additional motivations and requirements for a data grid, which is a bit more general, and use that to motivate changes in the protocol. >>> >>> Alex >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 10 Dec 2009, at 12:23, Galder Zamarreno wrote: >>> Also, it might be wised to implement operations like replaceIfEquals >>> using with a cas long, like memcached does >>> (http://code.google.com/p/memcached/wiki/MemcacheBinaryProtocol), rather >>> than sending the value cos that would be more network efficient. The old >>> value could be much more longer. >> >> Not sure I understand. So we have 2 replace methods (thanks to ConcurrentMap) - replace(k, v) which succeeds if k is mapped to something, and replace(k, v1, v2) which succeeds if k is mapped to v1. From what I see, only the former is supported by memcached's binary protocol and not the latter. Are you suggesting supporting the latter in HotRod? > > I'm suggesting supporting both actually. > > replace(k, v) which succeeds if k is mapped to something - that's the > same as memcached's replace command. I plan to support that. > > replace(k, v1, v2) is what memcached does with 'cas' but instead of > passing the old value, it passes the cas id, which is 8 bytes at the > most. The old value could be a lot longer. In ConcurrentMap, replace > works by passing the old value. Sure, and that's in the same VM so you pass by reference. Across the wire, this may involve serializing the entire (old) value which may be quite large. > I think, to save network space, I think we should implement it the way > memcached does. IOW, whenever something gets stored, it's stored with a > cas id and this can later be used to replace a value, only of the cas is > the one as we've passed. You get this cas id back as a return value? Which you can then store and use in future 'replace' commands? Is this it?

> In the txt protocol, I implemented the cas id using System.nanoTime(). Is this cas id meant to be unique? If so, System.nanoTime() may not be ...

>>> Another thing to note here is that for expiry and maxIdle, I followed >>> the Java TimeUnit based definition. To make it easier for client >>> implementers, we could follow the memcached style, where seconds are >>> used always and if the number of seconds is bigger than 30 days, those >>> seconds represent UNIX time (number of seconds since 1st January 1970). >>> WDYT? >> >> This is done to reduce bytes on the wire, since expressing something in millis often means extra bytes and often meaningless since network latency can be worth several hundred millis already. So it makes sense for the protocol to express this in seconds. Not sure I understand your comment re: seconds> 30 days. Are you referring to the time being treated as relative to the UNIX epoch [1] as opposed to relative to the current time snapshot? > > Indeed, when you pass a number of seconds that is bigger than 30 days, > memcached understands it as the offset from 1/1/1970. Otherwise, if > below 30 days, it thinks it's relative to the current time. > > Good point about sending anything below seconds, it's wasteful and > netwrok latency probably invalidates. Assuming we only allowed, seconds, > minutes, hours, and days as time units, we'll be saving space compared > to memcached. > > I'm not sure of the frequency of hours/days as time units, so I'd > inclined to only support seconds and use the same algorithm as memcached > simply, as this reduces complexity of existing clients that might be > used to memcached protocol. Yes, the protocol should just use seconds. Client libs on top could add syntactic sugar like TimeUnit.*, etc. Cheers -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 10 Dec 2009, at 15:24, Galder Zamarreno wrote: > > > On 12/10/2009 03:15 PM, Manik Surtani wrote: >> >> On 10 Dec 2009, at 12:23, Galder Zamarreno wrote: >>>>> Also, it might be wised to implement operations like replaceIfEquals >>>>> using with a cas long, like memcached does >>>>> (http://code.google.com/p/memcached/wiki/MemcacheBinaryProtocol), rather >>>>> than sending the value cos that would be more network efficient. The old >>>>> value could be much more longer. >>>> >>>> Not sure I understand. So we have 2 replace methods (thanks to ConcurrentMap) - replace(k, v) which succeeds if k is mapped to something, and replace(k, v1, v2) which succeeds if k is mapped to v1. From what I see, only the former is supported by memcached's binary protocol and not the latter. Are you suggesting supporting the latter in HotRod? >>> >>> I'm suggesting supporting both actually. >>> >>> replace(k, v) which succeeds if k is mapped to something - that's the >>> same as memcached's replace command. I plan to support that. >>> >>> replace(k, v1, v2) is what memcached does with 'cas' but instead of >>> passing the old value, it passes the cas id, which is 8 bytes at the >>> most. The old value could be a lot longer. In ConcurrentMap, replace >>> works by passing the old value. >> >> Sure, and that's in the same VM so you pass by reference. Across the wire, this may involve serializing the entire (old) value which may be quite large. >> >>> I think, to save network space, I think we should implement it the way >>> memcached does. IOW, whenever something gets stored, it's stored with a >>> cas id and this can later be used to replace a value, only of the cas is >>> the one as we've passed. >> >> You get this cas id back as a return value? Which you can then store and use in future 'replace' commands? Is this it? > > Yeah, u can get the cas id via gets command. > >> >>> In the txt protocol, I implemented the cas id using System.nanoTime(). >> >> Is this cas id meant to be unique? If so, System.nanoTime() may not be ... > > Not over time. You want to be unique from the minute you retrieved it > until you use it and for operations on that particular key. > > The cas Id is simply used for comparison, to check whether someone else > has changed the entry since you retrieved it. System.nanoTime() gives > you precisely that, anyone that modifies that entry in that machine will > definitely have a different nano time. Also, since nano time is based > off an arbitrary time, the chances that a different machine will produce > the same nano time when modifying that very same key are almost very > very small. Improbable but certainly not impossible, even in 1 machine. Why not just use what we use for 'versioning' internally - object references? System.identityHashCode(), for example?

Cheers -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

>> Yeah, u can get the cas id via gets command. >> >>> >>>> In the txt protocol, I implemented the cas id using System.nanoTime(). >>> >>> Is this cas id meant to be unique? If so, System.nanoTime() may not be ... >> >> Not over time. You want to be unique from the minute you retrieved it >> until you use it and for operations on that particular key. >> >> The cas Id is simply used for comparison, to check whether someone else >> has changed the entry since you retrieved it. System.nanoTime() gives >> you precisely that, anyone that modifies that entry in that machine will >> definitely have a different nano time. Also, since nano time is based >> off an arbitrary time, the chances that a different machine will produce >> the same nano time when modifying that very same key are almost very >> very small. > > Improbable but certainly not impossible, even in 1 machine. > > Why not just use what we use for 'versioning' internally - object references? System.identityHashCode(), for example? Remember that you have to deal with eviction/expiration too. You could get a case like this: - get a cas based on identity hash code - the entry is expired - a new entry is put, with diff values, which happens to have the same identify hash code. - doing a cas will succeed when it shouldn't. I don't think this can happen with System.nanoTime(). When looking into this, I checked java.util.UUID but it didn't work, for at least for the memcached txt protocol, where a 64bit value is required and UUID is 128bits. Maybe we could try to compress it somehow?

Actually, according to the memcached-txt protocol definition, this is a 64-bit integer, so no real chance of compacting that. You need to provide a long of some sort: http://github.com/memcached/memcached/blob/master/doc/protocol.txt

In a cluster, you can't just have an atomic long starting at 0 that you increase everytime a node is modified, because you could easily find yourself in a situation where two nodes modifying the same key might generate the same cas id.

From: Galder Zamarreno <galder(a)redhat.com&gt; Subject: Re: [infinispan-dev] HotRod?? [ISPN-29 and a "custom protocol"] To: infinispan-dev(a)lists.jboss.org Date: Friday, December 11, 2009, 2:30 AM On 12/10/2009 08:19 PM, Alex Kluge wrote: >>>> Yeah, u can get the cas id via gets command. >>>> >>>>> >>>>>> In the txt protocol, I implemented the cas id using System.nanoTime(). >>>>> >>>>> Is this cas id meant to be unique?  If so, System.nanoTime() may not be ... >>>> >>>> Not over time. You want to be unique from the minute you retrieved it >>>> until you use it and for operations on that particular key. >>>> >>>> The cas Id is simply used for comparison, to check whether someone else >>>> has changed the entry since you retrieved it. System.nanoTime() gives >>>> you precisely that, anyone that modifies that entry in that machine will >>>> definitely have a different nano time. Also, since nano time is based >>>> off an arbitrary time, the chances that a different machine will produce >>>> the same nano time when modifying that very same key are almost very >>>> very small. >>> >>> Improbable but certainly not impossible, even in 1 machine. >>> >>> Why not just use what we use for 'versioning' internally - object references?  System.identityHashCode(), for example? >> >> Remember that you have to deal with eviction/expiration too. You could >> get a case like this: >> - get a cas based on identity hash code >> - the entry is expired >> - a new entry is put, with diff values, which happens to have the same >> identify hash code. >> - doing a cas will succeed when it shouldn't. >> >> I don't think this can happen with System.nanoTime(). >> >> When looking into this, I checked java.util.UUID but it didn't work, for >> at least for the memcached txt protocol, where a 64bit value is required >> and UUID is 128bits. Maybe we could try to compress it somehow? > >> Actually, according to the memcached-txt protocol definition, this is a >> 64-bit integer, so no real chance of compacting that. You need to >> provide a long of some sort: >> >> http://github.com/memcached/memcached/blob/master/doc/protocol.txt > >   Is there any expected use of this value where it will be used outside of > the context of a specific key? If not, then it does not have to be > globally unique, and indeed, can simply be a modification count. > Easily computed, and easily fit within 64 bits even over a long lifetime > of a server. To do that, might as well use System.nanoTime(). It's much better because than modification count because each VM will start off a different offset. In a cluster, you can't just have an atomic long starting at 0 that you increase everytime a node is modified, because you could easily find yourself in a situation where two nodes modifying the same key might generate the same cas id. If you don't start at 0, that's pretty much what you get with System.nanoTime() and I think it's better to call that all the time, than taking the given long and adding one at the time. Someone might think that you could have a counter per cache entry but this definitely cannot start at 0, since after eviction, when the entry is gone, it'll go back to 0 and you can have issues like the one I explained above. Given the constraints of at least the memcached txt protocol, I think System.nanoTime() is the best option in that case. For hotrod, a 128-bit UUID would avoid issues mentioned here. We'd only need to make sure to find a performant solution. > >   For now though the real question is can we fit a quantity that will > fulfil the requirements for the field into the space alloted. I think > we can. > > >                                         Alex > > > > > > _______________________________________________

> infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

> In a cluster, you can't just have an atomic long starting at 0 that you > increase everytime a node is modified, because you could easily find > yourself in a situation where two nodes modifying the same key might > generate the same cas id. Hmmm, in a cluster no mater who writes the key/vale, they have to wind up modifying the same data, no? Otherwise the cluster becomes inconsistent. That is the point, where, on modification, you can increment the counter.

Eviction, however, raises more challenges. Clearly, when a value is evicted, you can't simply restart the counter when the data is rewritten. This would confuse the clients. However, I also get uncomfortable putting something time consuming into the main write path. I guess I am a bit jaded from when System.currentTimeMillis() took a long time. But then again: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6440250 Also note from this that sometimes nanoTime just returns currentTimeMillis*10^6.

Perhaps setting an initial value in a time consuming way, then a simple increment thereafter. The goal is to get consistent, usable behaviour, while maintaining performance.

We definitely shouldn't use completely independent CAS values for different protocols. Got to run, Alex --- On Fri, 12/11/09, Galder Zamarreno&lt;galder(a)redhat.com&gt; wrote: > From: Galder Zamarreno&lt;galder(a)redhat.com&gt; > Subject: Re: [infinispan-dev] HotRod?? [ISPN-29 and a "custom protocol"] > To: infinispan-dev(a)lists.jboss.org > Date: Friday, December 11, 2009, 2:30 AM > > > On 12/10/2009 08:19 PM, Alex Kluge wrote: >>>>> Yeah, u can get the cas id via gets > command. >>>>> >>>>>> >>>>>>> In the txt protocol, I implemented > the cas id using System.nanoTime(). >>>>>> >>>>>> Is this cas id meant to be > unique? If so, System.nanoTime() may not be ... >>>>> >>>>> Not over time. You want to be unique from > the minute you retrieved it >>>>> until you use it and for operations on > that particular key. >>>>> >>>>> The cas Id is simply used for comparison, > to check whether someone else >>>>> has changed the entry since you retrieved > it. System.nanoTime() gives >>>>> you precisely that, anyone that modifies > that entry in that machine will >>>>> definitely have a different nano time. > Also, since nano time is based >>>>> off an arbitrary time, the chances that a > different machine will produce >>>>> the same nano time when modifying that > very same key are almost very >>>>> very small. >>>> >>>> Improbable but certainly not impossible, even > in 1 machine. >>>> >>>> Why not just use what we use for 'versioning' > internally - object references? > System.identityHashCode(), for example? >>> >>> Remember that you have to deal with > eviction/expiration too. You could >>> get a case like this: >>> - get a cas based on identity hash code >>> - the entry is expired >>> - a new entry is put, with diff values, which > happens to have the same >>> identify hash code. >>> - doing a cas will succeed when it shouldn't. >>> >>> I don't think this can happen with > System.nanoTime(). >>> >>> When looking into this, I checked java.util.UUID > but it didn't work, for >>> at least for the memcached txt protocol, where a > 64bit value is required >>> and UUID is 128bits. Maybe we could try to > compress it somehow? >> >>> Actually, according to the memcached-txt protocol > definition, this is a >>> 64-bit integer, so no real chance of compacting > that. You need to >>> provide a long of some sort: >>> >>> http://github.com/memcached/memcached/blob/master/doc/protocol.txt >> >> Is there any expected use of this > value where it will be used outside of >> the context of a specific key? If not, then it does > not have to be >> globally unique, and indeed, can simply be a > modification count. >> Easily computed, and easily fit within 64 bits even > over a long lifetime >> of a server. > > To do that, might as well use System.nanoTime(). It's much > better > because than modification count because each VM will start > off a > different offset. > > In a cluster, you can't just have an atomic long starting > at 0 that you > increase everytime a node is modified, because you could > easily find > yourself in a situation where two nodes modifying the same > key might > generate the same cas id. If you don't start at 0, that's > pretty much > what you get with System.nanoTime() and I think it's better > to call that > all the time, than taking the given long and adding one at > the time. > > Someone might think that you could have a counter per cache > entry but > this definitely cannot start at 0, since after eviction, > when the entry > is gone, it'll go back to 0 and you can have issues like > the one I > explained above. > > Given the constraints of at least the memcached txt > protocol, I think > System.nanoTime() is the best option in that case. For > hotrod, a 128-bit > UUID would avoid issues mentioned here. We'd only need to > make sure to > find a performant solution. > >> >> For now though the real question is > can we fit a quantity that will >> fulfil the requirements for the field into the space > alloted. I think >> we can. >> >> >> > > Alex >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

1. Message IDs. Is this whap the client uses to match responses to requests sent, perhaps if sent in an async manner (NIO)? If so, is 4 bytes (a Java int) enough? Would we not want 8 bytes for this? Also, if it is unsigned, perhaps we could use a variable-length long?

*Async ops (putAsync, removeAsync...etc)

- keySet, entrySet, values: I don't plan to include them since they'd be very expensive both from a cache iteration perspective and from sending that stuff back to the client.

Also, it might be wised to implement operations like replaceIfEquals using with a cas long, like memcached does

Finally, memcached has this notion of getq where, as far as I can understand the spec, it's a quiet operation where the server holds up the return until you send a non quiet operation, i.e. a get or a noop.

From: Manik Surtani <manik(a)jboss.org&gt; Subject: Re: [infinispan-dev] HotRod?? [ISPN-29 and a "custom protocol"] To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; Date: Wednesday, December 9, 2009, 6:44 AM A few more comments: 1.  Message IDs.  Is this whap the client uses to match responses to requests sent, perhaps if sent in an async manner (NIO)?  If so, is 4 bytes (a Java int) enough?  Would we not want 8 bytes for this?  Also, if it is unsigned, perhaps we could use a variable-length long?     http://lucene.apache.org/java/2_9_1/fileformats.html#VInt Or maybe there is a better way to express this - UUIDs?  Or is that too long/verbose? 2.  Responses.  How do you express hostname/IP (intelligence = 1) and hashes (intelligence = 2)?  I presume the former is an IP represented as a String so that the client can open a connection to the node, and the latter is a 32-bit int since these are the hashes used by the ConsistentHash impl on the server nodes? 3.  Responses.  Number of owners - 4 bytes for this?  Surely this is wrong, since you only have 2 bytes for number of cluster members?  :)  Personally I think you could use 2 bytes for the former and a variable-length int/long for the latter (see above). Cheers Manik On 8 Dec 2009, at 19:36, Galder Zamarreno wrote: > Hi all, > > I've updated with a very early draft of the hot rod protocol: > http://www.jboss.org/community/wiki/HotRodProtocol > > As a reference, I've been using > http://code.google.com/p/memcached/wiki/MemcacheBinaryProtocol to see > what memcached did. Alex, as I emailed you yesterday, it'd be > interesting to see what you did too. > > First, several operations are still missing from the spec: remove, > removeIfPresent, *Async ops (putAsync, removeAsync...etc), putAll, > putForExternalRead, evict, version, name (get cache's name). > > Also, I still need to think how to add flags into each operation sent > across the network. > > There're some other operations that I'm consider whether to include or > not. Here's what I'm thinking at the moment: > > - keySet, entrySet, values: I don't plan to include them since they'd be > very expensive both from a cache iteration perspective and from sending > that stuff back to the client. > > - containsKey: I don't plan to include it, you can figure it out with > get command. > > - containsValue: No plan to include it since we don't support it. > > - size, isEmpty: Maybe include them as part of stats command like > memcached does? > > Also, it might be wised to implement operations like replaceIfEquals > using with a cas long, like memcached does > (http://code.google.com/p/memcached/wiki/MemcacheBinaryProtocol), rather > than sending the value cos that would be more network efficient. The old > value could be much more longer. > > Another thing to note here is that for expiry and maxIdle, I followed > the Java TimeUnit based definition. To make it easier for client > implementers, we could follow the memcached style, where seconds are > used always and if the number of seconds is bigger than 30 days, those > seconds represent UNIX time (number of seconds since 1st January 1970). > WDYT? > > Finally, memcached has this notion of getq where, as far as I can > understand the spec, it's a quiet operation where the server holds up > the return until you send a non quiet operation, i.e. a get or a noop. I > need to look further into it to understand how exactly works, but it'd > be interesting to know the thoughts of anyone who's dealt with this already. > > Cheers, > > On 12/03/2009 06:22 PM, Manik Surtani wrote: >> Great, thanks Alex! >> >> On 3 Dec 2009, at 05:00, Alex Kluge wrote: >> >>> Hi, >>> >>>>> I'll see what I can do about documenting it... >>>> >>>> Yeah lets do that - I would recommend using the JBoss.org >>>> wiki pages so that the designs can be distilled by all of us >>>> and then linked to from the other Infinispan design pages. >>> >>> I've started this at >>> >>>  http://www.jboss.org/community/wiki/RemoteCacheInteractions >>> >>> I will start with a description of what I have, which is a good multilevel cache implementation. Then get to a description of some of the additional motivations and requirements for a data grid, which is a bit more general, and use that to motivate changes in the protocol. >>> >>>                                                       Alex >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

A few more comments: 1. Message IDs. Is this whap the client uses to match responses to requests sent, perhaps if sent in an async manner (NIO)? If so, is 4 bytes (a Java int) enough? Would we not want 8 bytes for this? Also, if it is unsigned, perhaps we could use a variable-length long? http://lucene.apache.org/java/2_9_1/fileformats.html#VInt Or maybe there is a better way to express this - UUIDs? Or is that too long/verbose? 2. Responses. How do you express hostname/IP (intelligence = 1) and hashes (intelligence = 2)? I presume the former is an IP represented as a String so that the client can open a connection to the node, and the latter is a 32-bit int since these are the hashes used by the ConsistentHash impl on the server nodes?

3. Responses. Number of owners - 4 bytes for this? Surely this is wrong, since you only have 2 bytes for number of cluster members? :) Personally I think you could use 2 bytes for the former and a variable-length int/long for the latter (see above).

Cheers Manik On 8 Dec 2009, at 19:36, Galder Zamarreno wrote: > Hi all, > > I've updated with a very early draft of the hot rod protocol: > http://www.jboss.org/community/wiki/HotRodProtocol > > As a reference, I've been using > http://code.google.com/p/memcached/wiki/MemcacheBinaryProtocol to see > what memcached did. Alex, as I emailed you yesterday, it'd be > interesting to see what you did too. > > First, several operations are still missing from the spec: remove, > removeIfPresent, *Async ops (putAsync, removeAsync...etc), putAll, > putForExternalRead, evict, version, name (get cache's name). > > Also, I still need to think how to add flags into each operation sent > across the network. > > There're some other operations that I'm consider whether to include or > not. Here's what I'm thinking at the moment: > > - keySet, entrySet, values: I don't plan to include them since they'd be > very expensive both from a cache iteration perspective and from sending > that stuff back to the client. > > - containsKey: I don't plan to include it, you can figure it out with > get command. > > - containsValue: No plan to include it since we don't support it. > > - size, isEmpty: Maybe include them as part of stats command like > memcached does? > > Also, it might be wised to implement operations like replaceIfEquals > using with a cas long, like memcached does > (http://code.google.com/p/memcached/wiki/MemcacheBinaryProtocol), rather > than sending the value cos that would be more network efficient. The old > value could be much more longer. > > Another thing to note here is that for expiry and maxIdle, I followed > the Java TimeUnit based definition. To make it easier for client > implementers, we could follow the memcached style, where seconds are > used always and if the number of seconds is bigger than 30 days, those > seconds represent UNIX time (number of seconds since 1st January 1970). > WDYT? > > Finally, memcached has this notion of getq where, as far as I can > understand the spec, it's a quiet operation where the server holds up > the return until you send a non quiet operation, i.e. a get or a noop. I > need to look further into it to understand how exactly works, but it'd > be interesting to know the thoughts of anyone who's dealt with this already. > > Cheers, > > On 12/03/2009 06:22 PM, Manik Surtani wrote: >> Great, thanks Alex! >> >> On 3 Dec 2009, at 05:00, Alex Kluge wrote: >> >>> Hi, >>> >>>>> I'll see what I can do about documenting it... >>>> >>>> Yeah lets do that - I would recommend using the JBoss.org >>>> wiki pages so that the designs can be distilled by all of us >>>> and then linked to from the other Infinispan design pages. >>> >>> I've started this at >>> >>> http://www.jboss.org/community/wiki/RemoteCacheInteractions >>> >>> I will start with a description of what I have, which is a good multilevel cache implementation. Then get to a description of some of the additional motivations and requirements for a data grid, which is a bit more general, and use that to motivate changes in the protocol. >>> >>> Alex >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev