Hi all, Re: http://community.jboss.org/wiki/HotRodProtocol I've updated the wiki with the following stuff: - Renamed replaceIfEquals to replaceIfUnmodified - Added remove and removeIfUnmodified. - Added containsKey command. - Added getWithCas command so that cas value can be returned. I decided for a separate command rather than adding cas to get return because you don't always want cas to be returned. Having a separate command makes better use of network bandwith. - Added stats command. JMX attributes are basically accessible through this, including cache size. - Added error handling section and updated status codes. Note that Mircea added some interesting comments and I replied to them directly in the wiki. Still remaining to add: - Commands: putForExternalRead evict, clear, version, name and quite commands. - Passing flags. Regards, p.s. Updating this has been quite a struggle due to F12 + FF 3.5.5 crashing at least 5 times, plus parts of the wiki dissapearing after publishing them! -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 12/15/2009 01:13 PM, Manik Surtani wrote: > A few comments: > > - Why do you have OpCode in your response header? Surely this is redundant? If the client is sync, it knows what it sent. If it is async, it has a message ID. True, I fixed that. > - 'Not So Dumb' and 'Clever' response headers should be optional? Surely this stuff is only sent when there is a topology change? We also may need some extra info here - how does the back-end know to send this info? If a client hits different back-end nodes, and there is a topology change, how does Node A decide that it should not bother with topology info since the client already hit Node B after the topo change and has the new topology map? Perhaps a TopologyVersion (== JGroups View ID) should be sent back with any topo map, and the client would send it's current TopologyVersion with every request (non-dumb clients only)? Could be a vlong... Yeah, as you and Mircea suggest, clients sending the view id could help better manage size of responses. Something to note here is that not all cluster view changes necessarily involve a client side topology view change, cos if you have N infinispan nodes in the cluster, you could have M running hot rod server where M <= N. So, Hot rod will need to figure out when there's been a view change in the cluster that involves the addition or removal of a hot rod running Infinispan instance (similar thing to what happens when a clustered EJB is deployed, the target list for that clustered EJB gets updated).

So, bearing in mind this, could we just use the JGroups view id for this? AFAIK, it's 0 based long but shouldn't cause problems with complete cluster restarts. If the whole cluster gets restarted, existing connections will be closed and at that point, clients could revert back to trying to connect one of their known hot rod servers and pass -1 as view id which means that I have no view, so the responding server would send back the hot rod cluster view.

I'll add this to the wiki. > > Cheers > Manik > > > On 14 Dec 2009, at 20:08, Galder Zamarreno wrote: > >> Hi all, >> >> Re: http://community.jboss.org/wiki/HotRodProtocol >> >> I've updated the wiki with the following stuff: >> - Renamed replaceIfEquals to replaceIfUnmodified >> - Added remove and removeIfUnmodified. >> - Added containsKey command. >> - Added getWithCas command so that cas value can be returned. I decided >> for a separate command rather than adding cas to get return because you >> don't always want cas to be returned. Having a separate command makes >> better use of network bandwith. >> - Added stats command. JMX attributes are basically accessible through >> this, including cache size. >> - Added error handling section and updated status codes. >> >> Note that Mircea added some interesting comments and I replied to them >> directly in the wiki. >> >> Still remaining to add: >> - Commands: putForExternalRead evict, clear, version, name and quite >> commands. >> - Passing flags. >> >> Regards, >> >> p.s. Updating this has been quite a struggle due to F12 + FF 3.5.5 >> crashing at least 5 times, plus parts of the wiki dissapearing after >> publishing them! >> -- >> Galder Zamarreño >> Sr. Software Engineer >> Infinispan, JBoss Cache >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 12/16/2009 12:14 PM, Manik Surtani wrote: > > On 16 Dec 2009, at 09:26, Galder Zamarreno wrote: > >> >> >> On 12/15/2009 01:13 PM, Manik Surtani wrote: >>> A few comments: >>> >>> - Why do you have OpCode in your response header? Surely this is redundant? If the client is sync, it knows what it sent. If it is async, it has a message ID. >> >> True, I fixed that. >> >>> - 'Not So Dumb' and 'Clever' response headers should be optional? Surely this stuff is only sent when there is a topology change? We also may need some extra info here - how does the back-end know to send this info? If a client hits different back-end nodes, and there is a topology change, how does Node A decide that it should not bother with topology info since the client already hit Node B after the topo change and has the new topology map? Perhaps a TopologyVersion (== JGroups View ID) should be sent back with any topo map, and the client would send it's current TopologyVersion with every request (non-dumb clients only)? Could be a vlong... >> >> Yeah, as you and Mircea suggest, clients sending the view id could help >> better manage size of responses. >> >> Something to note here is that not all cluster view changes necessarily >> involve a client side topology view change, cos if you have N infinispan >> nodes in the cluster, you could have M running hot rod server where M<= >> N. So, Hot rod will need to figure out when there's been a view change >> in the cluster that involves the addition or removal of a hot rod >> running Infinispan instance (similar thing to what happens when a >> clustered EJB is deployed, the target list for that clustered EJB gets >> updated). > > Yes. > >> So, bearing in mind this, could we just use the JGroups view id for >> this? AFAIK, it's 0 based long but shouldn't cause problems with >> complete cluster restarts. If the whole cluster gets restarted, existing >> connections will be closed and at that point, clients could revert back >> to trying to connect one of their known hot rod servers and pass -1 as >> view id which means that I have no view, so the responding server would >> send back the hot rod cluster view. > > Provided the client knows the servers have all been restarted. > > I think a better approach is that the client sends its last known topology version (vClient). The server node it connects to compares vClient with vServer (the version that the servers are aware of). if vClient != vServer then resend topology map. I added that already this morning to the request header :) http://community.jboss.org/wiki/HotRodProtocol Not-so-dumb and clever clients send their latest viewId to the server and the server compares would compare it and if different, send the new topology. Note that in the response, I had added a "View change marker" field so that the client knows whether a view change follows or not. The reason I had done this is because I didn't think the client could simply compare the vServer with the vClient and decide whether more needs to be read or not.

You could have an scenario where a client starts up and sends 3 request in pararell, all with vClient=-1 because no replies have been received from hot rod server. When the server replied to those 3 req, it would have sent for example, vServer=4. The first response to be processed would have update the local vClient to 4, but the other two responses would still contain the vServer=4. So, they would somehow need to read, or discard the response. And after writing this and thinking it again, the "View change marker" is not be necessary. If vClient and vServer are equals, the client knows the total length of the body so it can read it and discard it directly. > >> I'll add this to the wiki. >> >>> >>> Cheers >>> Manik >>> >>> >>> On 14 Dec 2009, at 20:08, Galder Zamarreno wrote: >>> >>>> Hi all, >>>> >>>> Re: http://community.jboss.org/wiki/HotRodProtocol >>>> >>>> I've updated the wiki with the following stuff: >>>> - Renamed replaceIfEquals to replaceIfUnmodified >>>> - Added remove and removeIfUnmodified. >>>> - Added containsKey command. >>>> - Added getWithCas command so that cas value can be returned. I decided >>>> for a separate command rather than adding cas to get return because you >>>> don't always want cas to be returned. Having a separate command makes >>>> better use of network bandwith. >>>> - Added stats command. JMX attributes are basically accessible through >>>> this, including cache size. >>>> - Added error handling section and updated status codes. >>>> >>>> Note that Mircea added some interesting comments and I replied to them >>>> directly in the wiki. >>>> >>>> Still remaining to add: >>>> - Commands: putForExternalRead evict, clear, version, name and quite >>>> commands. >>>> - Passing flags. >>>> >>>> Regards, >>>> >>>> p.s. Updating this has been quite a struggle due to F12 + FF 3.5.5 >>>> crashing at least 5 times, plus parts of the wiki dissapearing after >>>> publishing them! >>>> -- >>>> Galder Zamarreño >>>> Sr. Software Engineer >>>> Infinispan, JBoss Cache >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> -- >>> Manik Surtani >>> manik(a)jboss.org >>> Lead, Infinispan >>> Lead, JBoss Cache >>> http://www.infinispan.org >>> http://www.jbosscache.org >>> >>> >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Galder Zamarreño >> Sr. Software Engineer >> Infinispan, JBoss Cache >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Why do you have OpCode in your response header? Surely this is redundant? If the client is sync, it knows what it sent. If it is async, it has a message ID. >>>> True, I fixed that.

From: Galder Zamarreno <galder(a)redhat.com&gt; Subject: Re: [infinispan-dev] Hot Rod - pt2 To: infinispan-dev(a)lists.jboss.org Date: Wednesday, December 16, 2009, 8:16 AM On 12/16/2009 02:53 PM, Manik Surtani wrote: > > On 16 Dec 2009, at 13:38, Galder Zamarreno wrote: > >> >> >> On 12/16/2009 12:14 PM, Manik Surtani wrote: >>> >>> On 16 Dec 2009, at 09:26, Galder Zamarreno wrote: >>> >>>> >>>> >>>> On 12/15/2009 01:13 PM, Manik Surtani wrote: >>>>> A few comments: >>>>> >>>>> - Why do you have OpCode in your response header?  Surely this is redundant?  If the client is sync, it knows what it sent.  If it is async, it has a message ID. >>>> >>>> True, I fixed that. >>>> >>>>> - 'Not So Dumb' and 'Clever' response headers should be optional?  Surely this stuff is only sent when there is a topology change?  We also may need some extra info here - how does the back-end know to send this info?  If a client hits different back-end nodes, and there is a topology change, how does Node A decide that it should not bother with topology info since the client already hit Node B after the topo change and has the new topology map?  Perhaps a TopologyVersion (== JGroups View ID) should be sent back with any topo map, and the client would send it's current TopologyVersion with every request (non-dumb clients only)?  Could be a vlong... >>>> >>>> Yeah, as you and Mircea suggest, clients sending the view id could help >>>> better manage size of responses. >>>> >>>> Something to note here is that not all cluster view changes necessarily >>>> involve a client side topology view change, cos if you have N infinispan >>>> nodes in the cluster, you could have M running hot rod server where M<= >>>> N. So, Hot rod will need to figure out when there's been a view change >>>> in the cluster that involves the addition or removal of a hot rod >>>> running Infinispan instance (similar thing to what happens when a >>>> clustered EJB is deployed, the target list for that clustered EJB gets >>>> updated). >>> >>> Yes. >>> >>>> So, bearing in mind this, could we just use the JGroups view id for >>>> this? AFAIK, it's 0 based long but shouldn't cause problems with >>>> complete cluster restarts. If the whole cluster gets restarted, existing >>>> connections will be closed and at that point, clients could revert back >>>> to trying to connect one of their known hot rod servers and pass -1 as >>>> view id which means that I have no view, so the responding server would >>>> send back the hot rod cluster view. >>> >>> Provided the client knows the servers have all been restarted. >>> >>> I think a better approach is that the client sends its last known topology version (vClient).  The server node it connects to compares vClient with vServer (the version that the servers are aware of).  if vClient != vServer then resend topology map. >> >> I added that already this morning to the request header :) >> http://community.jboss.org/wiki/HotRodProtocol >> >> Not-so-dumb and clever clients send their latest viewId to the server >> and the server compares would compare it and if different, send the new >> topology. >> >> Note that in the response, I had added a "View change marker" field so >> that the client knows whether a view change follows or not. The reason I >> had done this is because I didn't think the client could simply compare >> the vServer with the vClient and decide whether more needs to be read or >> not. > > No, the server decides whether the Topo map is out of date.  If so, it sends a new one.  The client just applies the new one if a new one is sent. Indeed, but this was about the client knowing that a new topo has been sent to it. I mean, how does the client know that server has actually sent back a new topology? I was using that field to indicate to the client: "hey, a new view follows the head after this field read". I can then read the vServer sent...etc. > >> You could have an scenario where a client starts up and sends 3 request >> in pararell, all with vClient=-1 because no replies have been received >> from hot rod server. When the server replied to those 3 req, it would >> have sent for example, vServer=4. The first response to be processed >> would have update the local vClient to 4, but the other two responses >> would still contain the vServer=4. So, they would somehow need to read, >> or discard the response. >> >> And after writing this and thinking it again, the "View change marker" >> is not be necessary. If vClient and vServer are equals, the client knows >> the total length of the body so it can read it and discard it directly. Hmmmm, but thinking through this again, the total body includes other stuff. So I do need that marker unless the header comes with a total number of bytes in header, in which case yeah, I can discard the remaining bytes from the header and ignore the topology information cos I have already applied it. >> >>> >>>> I'll add this to the wiki. >>>> >>>>> >>>>> Cheers >>>>> Manik >>>>> >>>>> >>>>> On 14 Dec 2009, at 20:08, Galder Zamarreno wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> Re: http://community.jboss.org/wiki/HotRodProtocol >>>>>> >>>>>> I've updated the wiki with the following stuff: >>>>>> - Renamed replaceIfEquals to replaceIfUnmodified >>>>>> - Added remove and removeIfUnmodified. >>>>>> - Added containsKey command. >>>>>> - Added getWithCas command so that cas value can be returned. I decided >>>>>> for a separate command rather than adding cas to get return because you >>>>>> don't always want cas to be returned. Having a separate command makes >>>>>> better use of network bandwith. >>>>>> - Added stats command. JMX attributes are basically accessible through >>>>>> this, including cache size. >>>>>> - Added error handling section and updated status codes. >>>>>> >>>>>> Note that Mircea added some interesting comments and I replied to them >>>>>> directly in the wiki. >>>>>> >>>>>> Still remaining to add: >>>>>> - Commands: putForExternalRead evict, clear, version, name and quite >>>>>> commands. >>>>>> - Passing flags. >>>>>> >>>>>> Regards, >>>>>> >>>>>> p.s. Updating this has been quite a struggle due to F12 + FF 3.5.5 >>>>>> crashing at least 5 times, plus parts of the wiki dissapearing after >>>>>> publishing them! >>>>>> -- >>>>>> Galder Zamarreño >>>>>> Sr. Software Engineer >>>>>> Infinispan, JBoss Cache >>>>>> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>>> -- >>>>> Manik Surtani >>>>> manik(a)jboss.org >>>>> Lead, Infinispan >>>>> Lead, JBoss Cache >>>>> http://www.infinispan.org >>>>> http://www.jbosscache.org >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> -- >>>> Galder Zamarreño >>>> Sr. Software Engineer >>>> Infinispan, JBoss Cache >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> -- >>> Manik Surtani >>> manik(a)jboss.org >>> Lead, Infinispan >>> Lead, JBoss Cache >>> http://www.infinispan.org >>> http://www.jbosscache.org >>> >>> >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Galder Zamarreño >> Sr. Software Engineer >> Infinispan, JBoss Cache >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 12/29/2009 07:35 PM, Alex Kluge wrote: >> Why do you have OpCode in your >> response header? Surely this is redundant? If >> the client is sync, it knows what it sent. If it is >> async, it has a message ID. >>>>>> True, I fixed that. > > I don't think it's a significant issue, and it is sort of redundant > in this case because you have the status response, however, it isn't > completely clear cut. When I implemented a similar protocol I kept > the response code. I suppose you mean that you kept the opt code? > It simplifies what you need to track within a > request/response cycle, and allows for easy extensibility. For > example, on error I return a different response type (a different > op code) which causes the response to be parsed by a different > message decoder. Why would you want the error to be parsed by a different decoder? > > I also made it easy to register custom requests and responses, and this > is facilitated by having distinct op codes for each. True. If you have a custom op in a custom request and response, this can be handled by a custom message decoder, assuming that the granularity of the message decoder would be based on the op code. This would indeed make it easier for the protocol to be extended. I haven't thought too deeply on the client implementation itself, so this will be clearer once I get around to doing that. For the time being, I'll leave it as it is and will revisit this decision at that point.

> > Alex > > > --- On Wed, 12/16/09, Galder Zamarreno&lt;galder(a)redhat.com&gt; wrote: > >> From: Galder Zamarreno&lt;galder(a)redhat.com&gt; >> Subject: Re: [infinispan-dev] Hot Rod - pt2 >> To: infinispan-dev(a)lists.jboss.org >> Date: Wednesday, December 16, 2009, 8:16 AM >> >> >> On 12/16/2009 02:53 PM, Manik Surtani wrote: >>> >>> On 16 Dec 2009, at 13:38, Galder Zamarreno wrote: >>> >>>> >>>> >>>> On 12/16/2009 12:14 PM, Manik Surtani wrote: >>>>> >>>>> On 16 Dec 2009, at 09:26, Galder Zamarreno >> wrote: >>>>> >>>>>> >>>>>> >>>>>> On 12/15/2009 01:13 PM, Manik Surtani >> wrote: >>>>>>> A few comments: >>>>>>> >>>>>>> - Why do you have OpCode in your >> response header? Surely this is redundant? If >> the client is sync, it knows what it sent. If it is >> async, it has a message ID. >>>>>> >>>>>> True, I fixed that. >>>>>> >>>>>>> - 'Not So Dumb' and 'Clever' response >> headers should be optional? Surely this stuff is only >> sent when there is a topology change? We also may need >> some extra info here - how does the back-end know to send >> this info? If a client hits different back-end nodes, >> and there is a topology change, how does Node A decide that >> it should not bother with topology info since the client >> already hit Node B after the topo change and has the new >> topology map? Perhaps a TopologyVersion (== JGroups >> View ID) should be sent back with any topo map, and the >> client would send it's current TopologyVersion with every >> request (non-dumb clients only)? Could be a vlong... >>>>>> >>>>>> Yeah, as you and Mircea suggest, clients >> sending the view id could help >>>>>> better manage size of responses. >>>>>> >>>>>> Something to note here is that not all >> cluster view changes necessarily >>>>>> involve a client side topology view >> change, cos if you have N infinispan >>>>>> nodes in the cluster, you could have M >> running hot rod server where M<= >>>>>> N. So, Hot rod will need to figure out >> when there's been a view change >>>>>> in the cluster that involves the addition >> or removal of a hot rod >>>>>> running Infinispan instance (similar thing >> to what happens when a >>>>>> clustered EJB is deployed, the target list >> for that clustered EJB gets >>>>>> updated). >>>>> >>>>> Yes. >>>>> >>>>>> So, bearing in mind this, could we just >> use the JGroups view id for >>>>>> this? AFAIK, it's 0 based long but >> shouldn't cause problems with >>>>>> complete cluster restarts. If the whole >> cluster gets restarted, existing >>>>>> connections will be closed and at that >> point, clients could revert back >>>>>> to trying to connect one of their known >> hot rod servers and pass -1 as >>>>>> view id which means that I have no view, >> so the responding server would >>>>>> send back the hot rod cluster view. >>>>> >>>>> Provided the client knows the servers have all >> been restarted. >>>>> >>>>> I think a better approach is that the client >> sends its last known topology version (vClient). The >> server node it connects to compares vClient with vServer >> (the version that the servers are aware of). if >> vClient != vServer then resend topology map. >>>> >>>> I added that already this morning to the request >> header :) >>>> http://community.jboss.org/wiki/HotRodProtocol >>>> >>>> Not-so-dumb and clever clients send their latest >> viewId to the server >>>> and the server compares would compare it and if >> different, send the new >>>> topology. >>>> >>>> Note that in the response, I had added a "View >> change marker" field so >>>> that the client knows whether a view change >> follows or not. The reason I >>>> had done this is because I didn't think the client >> could simply compare >>>> the vServer with the vClient and decide whether >> more needs to be read or >>>> not. >>> >>> No, the server decides whether the Topo map is out of >> date. If so, it sends a new one. The client just >> applies the new one if a new one is sent. >> >> Indeed, but this was about the client knowing that a new >> topo has been >> sent to it. I mean, how does the client know that server >> has actually >> sent back a new topology? I was using that field to >> indicate to the >> client: "hey, a new view follows the head after this field >> read". I can >> then read the vServer sent...etc. >> >>> >>>> You could have an scenario where a client starts >> up and sends 3 request >>>> in pararell, all with vClient=-1 because no >> replies have been received >>>> from hot rod server. When the server replied to >> those 3 req, it would >>>> have sent for example, vServer=4. The first >> response to be processed >>>> would have update the local vClient to 4, but the >> other two responses >>>> would still contain the vServer=4. So, they would >> somehow need to read, >>>> or discard the response. >>>> >>>> And after writing this and thinking it again, the >> "View change marker" >>>> is not be necessary. If vClient and vServer are >> equals, the client knows >>>> the total length of the body so it can read it and >> discard it directly. >> >> Hmmmm, but thinking through this again, the total body >> includes other >> stuff. So I do need that marker unless the header comes >> with a total >> number of bytes in header, in which case yeah, I can >> discard the >> remaining bytes from the header and ignore the topology >> information cos >> I have already applied it. >> >>>> >>>>> >>>>>> I'll add this to the wiki. >>>>>> >>>>>>> >>>>>>> Cheers >>>>>>> Manik >>>>>>> >>>>>>> >>>>>>> On 14 Dec 2009, at 20:08, Galder >> Zamarreno wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> Re: http://community.jboss.org/wiki/HotRodProtocol >>>>>>>> >>>>>>>> I've updated the wiki with the >> following stuff: >>>>>>>> - Renamed replaceIfEquals to >> replaceIfUnmodified >>>>>>>> - Added remove and >> removeIfUnmodified. >>>>>>>> - Added containsKey command. >>>>>>>> - Added getWithCas command so that >> cas value can be returned. I decided >>>>>>>> for a separate command rather than >> adding cas to get return because you >>>>>>>> don't always want cas to be >> returned. Having a separate command makes >>>>>>>> better use of network bandwith. >>>>>>>> - Added stats command. JMX >> attributes are basically accessible through >>>>>>>> this, including cache size. >>>>>>>> - Added error handling section and >> updated status codes. >>>>>>>> >>>>>>>> Note that Mircea added some >> interesting comments and I replied to them >>>>>>>> directly in the wiki. >>>>>>>> >>>>>>>> Still remaining to add: >>>>>>>> - Commands: putForExternalRead >> evict, clear, version, name and quite >>>>>>>> commands. >>>>>>>> - Passing flags. >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> p.s. Updating this has been quite >> a struggle due to F12 + FF 3.5.5 >>>>>>>> crashing at least 5 times, plus >> parts of the wiki dissapearing after >>>>>>>> publishing them! >>>>>>>> -- >>>>>>>> Galder Zamarreño >>>>>>>> Sr. Software Engineer >>>>>>>> Infinispan, JBoss Cache >>>>>>>> >> _______________________________________________ >>>>>>>> infinispan-dev mailing list >>>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>> >>>>>>> -- >>>>>>> Manik Surtani >>>>>>> manik(a)jboss.org >>>>>>> Lead, Infinispan >>>>>>> Lead, JBoss Cache >>>>>>> http://www.infinispan.org >>>>>>> http://www.jbosscache.org >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >> _______________________________________________ >>>>>>> infinispan-dev mailing list >>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> >>>>>> -- >>>>>> Galder Zamarreño >>>>>> Sr. Software Engineer >>>>>> Infinispan, JBoss Cache >>>>>> >> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>>> -- >>>>> Manik Surtani >>>>> manik(a)jboss.org >>>>> Lead, Infinispan >>>>> Lead, JBoss Cache >>>>> http://www.infinispan.org >>>>> http://www.jbosscache.org >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> -- >>>> Galder Zamarreño >>>> Sr. Software Engineer >>>> Infinispan, JBoss Cache >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> -- >>> Manik Surtani >>> manik(a)jboss.org >>> Lead, Infinispan >>> Lead, JBoss Cache >>> http://www.infinispan.org >>> http://www.jbosscache.org >>> >>> >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Galder Zamarreño >> Sr. Software Engineer >> Infinispan, JBoss Cache >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hi, > > When I implemented a similar protocol I kept > > the response code. > > I suppose you mean that you kept the opt code? Actually I went as far as to have different codes for a the request and response pairs of messages. It made debugging easier as well :)

> Why would you want the error to be parsed by a different decoder? The error responses are fundamentally different from a normal response, and contain the reason for the error. Generally, they contain the error message and the stack trace rendered to text. The get on the response future returns a response object, and the client can use instanceof to see exactly what type of response was returned. Most of the time this is done within the framework.

> > I haven't thought too deeply on the client implementation itself... Luckily, you have the advantage of having someone who has walked through this process, more than once. :) > p.s. Thanks for the feedback :) you're welcome. --- On Mon, 1/4/10, Galder Zamarreno&lt;galder(a)redhat.com&gt; wrote: > From: Galder Zamarreno&lt;galder(a)redhat.com&gt; > Subject: Re: [infinispan-dev] Hot Rod - pt2 > To: infinispan-dev(a)lists.jboss.org > Date: Monday, January 4, 2010, 5:25 AM > > > On 12/29/2009 07:35 PM, Alex Kluge wrote: >>> Why do you have OpCode in your >>> response header? Surely this is > redundant? If >>> the client is sync, it knows what it sent. > If it is >>> async, it has a message ID. >>>>>>> True, I fixed that. >> >> I don't think it's a significant > issue, and it is sort of redundant >> in this case because you have the status response, > however, it isn't >> completely clear cut. When I implemented a similar > protocol I kept >> the response code. > > I suppose you mean that you kept the opt code? > >> It simplifies what you need to track within a >> request/response cycle, and allows for easy > extensibility. For >> example, on error I return a different response type > (a different >> op code) which causes the response to be parsed by a > different >> message decoder. > > Why would you want the error to be parsed by a different > decoder? > >> >> I also made it easy to register custom requests and > responses, and this >> is facilitated by having distinct op codes for each. > > True. If you have a custom op in a custom request and > response, this can > be handled by a custom message decoder, assuming that the > granularity of > the message decoder would be based on the op code. This > would indeed > make it easier for the protocol to be extended. > > I haven't thought too deeply on the client implementation > itself, so > this will be clearer once I get around to doing that. For > the time > being, I'll leave it as it is and will revisit this > decision at that point. > >> >> > > > Alex >> >> >> --- On Wed, 12/16/09, Galder Zamarreno&lt;galder(a)redhat.com&gt; > wrote: >> >>> From: Galder Zamarreno&lt;galder(a)redhat.com&gt; >>> Subject: Re: [infinispan-dev] Hot Rod - pt2 >>> To: infinispan-dev(a)lists.jboss.org >>> Date: Wednesday, December 16, 2009, 8:16 AM >>> >>> >>> On 12/16/2009 02:53 PM, Manik Surtani wrote: >>>> >>>> On 16 Dec 2009, at 13:38, Galder Zamarreno > wrote: >>>> >>>>> >>>>> >>>>> On 12/16/2009 12:14 PM, Manik Surtani > wrote: >>>>>> >>>>>> On 16 Dec 2009, at 09:26, Galder > Zamarreno >>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On 12/15/2009 01:13 PM, Manik > Surtani >>> wrote: >>>>>>>> A few comments: >>>>>>>> >>>>>>>> - Why do you have OpCode in > your >>> response header? Surely this is > redundant? If >>> the client is sync, it knows what it sent. > If it is >>> async, it has a message ID. >>>>>>> >>>>>>> True, I fixed that. >>>>>>> >>>>>>>> - 'Not So Dumb' and 'Clever' > response >>> headers should be optional? Surely this > stuff is only >>> sent when there is a topology change? We > also may need >>> some extra info here - how does the back-end know > to send >>> this info? If a client hits different > back-end nodes, >>> and there is a topology change, how does Node A > decide that >>> it should not bother with topology info since the > client >>> already hit Node B after the topo change and has > the new >>> topology map? Perhaps a TopologyVersion (== > JGroups >>> View ID) should be sent back with any topo map, > and the >>> client would send it's current TopologyVersion > with every >>> request (non-dumb clients only)? Could be a > vlong... >>>>>>> >>>>>>> Yeah, as you and Mircea suggest, > clients >>> sending the view id could help >>>>>>> better manage size of responses. >>>>>>> >>>>>>> Something to note here is that not > all >>> cluster view changes necessarily >>>>>>> involve a client side topology > view >>> change, cos if you have N infinispan >>>>>>> nodes in the cluster, you could > have M >>> running hot rod server where M<= >>>>>>> N. So, Hot rod will need to figure > out >>> when there's been a view change >>>>>>> in the cluster that involves the > addition >>> or removal of a hot rod >>>>>>> running Infinispan instance > (similar thing >>> to what happens when a >>>>>>> clustered EJB is deployed, the > target list >>> for that clustered EJB gets >>>>>>> updated). >>>>>> >>>>>> Yes. >>>>>> >>>>>>> So, bearing in mind this, could we > just >>> use the JGroups view id for >>>>>>> this? AFAIK, it's 0 based long > but >>> shouldn't cause problems with >>>>>>> complete cluster restarts. If the > whole >>> cluster gets restarted, existing >>>>>>> connections will be closed and at > that >>> point, clients could revert back >>>>>>> to trying to connect one of their > known >>> hot rod servers and pass -1 as >>>>>>> view id which means that I have no > view, >>> so the responding server would >>>>>>> send back the hot rod cluster > view. >>>>>> >>>>>> Provided the client knows the servers > have all >>> been restarted. >>>>>> >>>>>> I think a better approach is that the > client >>> sends its last known topology version > (vClient). The >>> server node it connects to compares vClient with > vServer >>> (the version that the servers are aware of). > if >>> vClient != vServer then resend topology map. >>>>> >>>>> I added that already this morning to the > request >>> header :) >>>>> http://community.jboss.org/wiki/HotRodProtocol >>>>> >>>>> Not-so-dumb and clever clients send their > latest >>> viewId to the server >>>>> and the server compares would compare it > and if >>> different, send the new >>>>> topology. >>>>> >>>>> Note that in the response, I had added a > "View >>> change marker" field so >>>>> that the client knows whether a view > change >>> follows or not. The reason I >>>>> had done this is because I didn't think > the client >>> could simply compare >>>>> the vServer with the vClient and decide > whether >>> more needs to be read or >>>>> not. >>>> >>>> No, the server decides whether the Topo map is > out of >>> date. If so, it sends a new one. The > client just >>> applies the new one if a new one is sent. >>> >>> Indeed, but this was about the client knowing that > a new >>> topo has been >>> sent to it. I mean, how does the client know that > server >>> has actually >>> sent back a new topology? I was using that field > to >>> indicate to the >>> client: "hey, a new view follows the head after > this field >>> read". I can >>> then read the vServer sent...etc. >>> >>>> >>>>> You could have an scenario where a client > starts >>> up and sends 3 request >>>>> in pararell, all with vClient=-1 because > no >>> replies have been received >>>>> from hot rod server. When the server > replied to >>> those 3 req, it would >>>>> have sent for example, vServer=4. The > first >>> response to be processed >>>>> would have update the local vClient to 4, > but the >>> other two responses >>>>> would still contain the vServer=4. So, > they would >>> somehow need to read, >>>>> or discard the response. >>>>> >>>>> And after writing this and thinking it > again, the >>> "View change marker" >>>>> is not be necessary. If vClient and > vServer are >>> equals, the client knows >>>>> the total length of the body so it can > read it and >>> discard it directly. >>> >>> Hmmmm, but thinking through this again, the total > body >>> includes other >>> stuff. So I do need that marker unless the header > comes >>> with a total >>> number of bytes in header, in which case yeah, I > can >>> discard the >>> remaining bytes from the header and ignore the > topology >>> information cos >>> I have already applied it. >>> >>>>> >>>>>> >>>>>>> I'll add this to the wiki. >>>>>>> >>>>>>>> >>>>>>>> Cheers >>>>>>>> Manik >>>>>>>> >>>>>>>> >>>>>>>> On 14 Dec 2009, at 20:08, > Galder >>> Zamarreno wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> Re: http://community.jboss.org/wiki/HotRodProtocol >>>>>>>>> >>>>>>>>> I've updated the wiki with > the >>> following stuff: >>>>>>>>> - Renamed replaceIfEquals > to >>> replaceIfUnmodified >>>>>>>>> - Added remove and >>> removeIfUnmodified. >>>>>>>>> - Added containsKey > command. >>>>>>>>> - Added getWithCas command > so that >>> cas value can be returned. I decided >>>>>>>>> for a separate command > rather than >>> adding cas to get return because you >>>>>>>>> don't always want cas to > be >>> returned. Having a separate command makes >>>>>>>>> better use of network > bandwith. >>>>>>>>> - Added stats command. > JMX >>> attributes are basically accessible through >>>>>>>>> this, including cache > size. >>>>>>>>> - Added error handling > section and >>> updated status codes. >>>>>>>>> >>>>>>>>> Note that Mircea added > some >>> interesting comments and I replied to them >>>>>>>>> directly in the wiki. >>>>>>>>> >>>>>>>>> Still remaining to add: >>>>>>>>> - Commands: > putForExternalRead >>> evict, clear, version, name and quite >>>>>>>>> commands. >>>>>>>>> - Passing flags. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> p.s. Updating this has > been quite >>> a struggle due to F12 + FF 3.5.5 >>>>>>>>> crashing at least 5 times, > plus >>> parts of the wiki dissapearing after >>>>>>>>> publishing them! >>>>>>>>> -- >>>>>>>>> Galder Zamarreño >>>>>>>>> Sr. Software Engineer >>>>>>>>> Infinispan, JBoss Cache >>>>>>>>> >>> _______________________________________________ >>>>>>>>> infinispan-dev mailing > list >>>>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>>> >>>>>>>> -- >>>>>>>> Manik Surtani >>>>>>>> manik(a)jboss.org >>>>>>>> Lead, Infinispan >>>>>>>> Lead, JBoss Cache >>>>>>>> http://www.infinispan.org >>>>>>>> http://www.jbosscache.org >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>> _______________________________________________ >>>>>>>> infinispan-dev mailing list >>>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>> >>>>>>> -- >>>>>>> Galder Zamarreño >>>>>>> Sr. Software Engineer >>>>>>> Infinispan, JBoss Cache >>>>>>> >>> _______________________________________________ >>>>>>> infinispan-dev mailing list >>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> >>>>>> -- >>>>>> Manik Surtani >>>>>> manik(a)jboss.org >>>>>> Lead, Infinispan >>>>>> Lead, JBoss Cache >>>>>> http://www.infinispan.org >>>>>> http://www.jbosscache.org >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>>> -- >>>>> Galder Zamarreño >>>>> Sr. Software Engineer >>>>> Infinispan, JBoss Cache >>>>> > _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> -- >>>> Manik Surtani >>>> manik(a)jboss.org >>>> Lead, Infinispan >>>> Lead, JBoss Cache >>>> http://www.infinispan.org >>>> http://www.jbosscache.org >>>> >>>> >>>> >>>> >>>> >>>> > _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> -- >>> Galder Zamarreño >>> Sr. Software Engineer >>> Infinispan, JBoss Cache >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On Dec 16, 2009, at 9:26 AM, Galder Zamarreno wrote: > > > On 12/15/2009 01:13 PM, Manik Surtani wrote: >> A few comments: >> >> - Why do you have OpCode in your response header? Surely this is redundant? If the client is sync, it knows what it sent. If it is async, it has a message ID. > > True, I fixed that. > >> - 'Not So Dumb' and 'Clever' response headers should be optional? Surely this stuff is only sent when there is a topology change? We also may need some extra info here - how does the back-end know to send this info? If a client hits different back-end nodes, and there is a topology change, how does Node A decide that it should not bother with topology info since the client already hit Node B after the topo change and has the new topology map? Perhaps a TopologyVersion (== JGroups View ID) should be sent back with any topo map, and the client would send it's current TopologyVersion with every request (non-dumb clients only)? Could be a vlong... > > Yeah, as you and Mircea suggest, clients sending the view id could help > better manage size of responses. > > Something to note here is that not all cluster view changes necessarily > involve a client side topology view change, cos if you have N infinispan > nodes in the cluster, you could have M running hot rod server where M <= > N. So, Hot rod will need to figure out when there's been a view change > in the cluster that involves the addition or removal of a hot rod > running Infinispan instance (similar thing to what happens when a > clustered EJB is deployed, the target list for that clustered EJB gets > updated). yep, that's a good point. Every Hotrod server has to know exactly what other hotrod server are there running. I assume that because one way or the other each hotrod node will have to piggyback the list of active hotrod servers to the clients.

Can't the hotrod server track the topology changes (topology of hot rod servers) and increase the view id number at that time only?

Actually I think it can also rely on the jgroups view id, but only update the hotrod-view-id when jgroups cluster change overlaps with hotrod cluster change (i.e. a infinispan instance that has a hotrod serv started is added or removed).

> So, bearing in mind this, could we just use the JGroups view id for > this? AFAIK, it's 0 based long but shouldn't cause problems with > complete cluster restarts. If the whole cluster gets restarted, existing > connections will be closed and at that point, clients could revert back > to trying to connect one of their known hot rod servers and pass -1 wouldn't they pass the old view Id at this time rather than -1? Guess the client will realize that the Socket is closed, and at that time will send an "I don't have view" request, i.e. -1?

> as > view id which means that I have no view, so the responding server would > send back the hot rod cluster view. > > I'll add this to the wiki. > >> >> Cheers >> Manik >> >> >> On 14 Dec 2009, at 20:08, Galder Zamarreno wrote: >> >>> Hi all, >>> >>> Re: http://community.jboss.org/wiki/HotRodProtocol >>> >>> I've updated the wiki with the following stuff: >>> - Renamed replaceIfEquals to replaceIfUnmodified >>> - Added remove and removeIfUnmodified. >>> - Added containsKey command. >>> - Added getWithCas command so that cas value can be returned. I decided >>> for a separate command rather than adding cas to get return because you >>> don't always want cas to be returned. Having a separate command makes >>> better use of network bandwith. >>> - Added stats command. JMX attributes are basically accessible through >>> this, including cache size. >>> - Added error handling section and updated status codes. >>> >>> Note that Mircea added some interesting comments and I replied to them >>> directly in the wiki. >>> >>> Still remaining to add: >>> - Commands: putForExternalRead evict, clear, version, name and quite >>> commands. >>> - Passing flags. >>> >>> Regards, >>> >>> p.s. Updating this has been quite a struggle due to F12 + FF 3.5.5 >>> crashing at least 5 times, plus parts of the wiki dissapearing after >>> publishing them! >>> -- >>> Galder Zamarreño >>> Sr. Software Engineer >>> Infinispan, JBoss Cache >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On Dec 16, 2009, at 12:08 PM, Manik Surtani wrote: > > On 16 Dec 2009, at 11:54, Mircea Markus wrote: > >> >> On Dec 16, 2009, at 9:26 AM, Galder Zamarreno wrote: >> >>> >>> >>> On 12/15/2009 01:13 PM, Manik Surtani wrote: >>>> A few comments: >>>> >>>> - Why do you have OpCode in your response header? Surely this is redundant? If the client is sync, it knows what it sent. If it is async, it has a message ID. >>> >>> True, I fixed that. >>> >>>> - 'Not So Dumb' and 'Clever' response headers should be optional? Surely this stuff is only sent when there is a topology change? We also may need some extra info here - how does the back-end know to send this info? If a client hits different back-end nodes, and there is a topology change, how does Node A decide that it should not bother with topology info since the client already hit Node B after the topo change and has the new topology map? Perhaps a TopologyVersion (== JGroups View ID) should be sent back with any topo map, and the client would send it's current TopologyVersion with every request (non-dumb clients only)? Could be a vlong... >>> >>> Yeah, as you and Mircea suggest, clients sending the view id could help >>> better manage size of responses. >>> >>> Something to note here is that not all cluster view changes necessarily >>> involve a client side topology view change, cos if you have N infinispan >>> nodes in the cluster, you could have M running hot rod server where M <= >>> N. So, Hot rod will need to figure out when there's been a view change >>> in the cluster that involves the addition or removal of a hot rod >>> running Infinispan instance (similar thing to what happens when a >>> clustered EJB is deployed, the target list for that clustered EJB gets >>> updated). >> yep, that's a good point. Every Hotrod server has to know exactly what other hotrod server are there running. I assume that because one way or the other each hotrod node will have to piggyback the list of active hotrod servers to the clients. > > Yes, nodes that are not running HotRod are removed from the list prior to transmission. > >> Can't the hotrod server track the topology changes (topology of hot rod servers) and increase the view id number at that time only? > > That would be reimplementing a lot of logic that is already in JGroups. Maintaining view IDs is non-trivial if (a) any node can update this and (b) this needs to be kept coherent even as nodes leave/join (including any node maintaining this state). some logic will need to be there to make the difference between an jgroups node leaving and a an jgroups node where hotrod is deployed leaving. Each HR server will need to know the other HR servers running (different from jgroups level view), so we already have some view management logic.

> >> Actually I think it can also rely on the jgroups view id, but only update the hotrod-view-id when jgroups cluster change overlaps with hotrod cluster change (i.e. a infinispan instance that has a hotrod serv started is added or removed). > > Precisely. > >>> So, bearing in mind this, could we just use the JGroups view id for >>> this? AFAIK, it's 0 based long but shouldn't cause problems with >>> complete cluster restarts. If the whole cluster gets restarted, existing >>> connections will be closed and at that point, clients could revert back >>> to trying to connect one of their known hot rod servers and pass -1 >> wouldn't they pass the old view Id at this time rather than -1? Guess the client will realize that the Socket is closed, and at that time will send an "I don't have view" request, i.e. -1? > > No, socket being closed doesn't mean your view is out of date - it just means you fail over to the next server. A socket could close for reasons other than a server leaving the group. > >>> as >>> view id which means that I have no view, so the responding server would >>> send back the hot rod cluster view. >>> >>> I'll add this to the wiki. >> >>> >>>> >>>> Cheers >>>> Manik >>>> >>>> >>>> On 14 Dec 2009, at 20:08, Galder Zamarreno wrote: >>>> >>>>> Hi all, >>>>> >>>>> Re: http://community.jboss.org/wiki/HotRodProtocol >>>>> >>>>> I've updated the wiki with the following stuff: >>>>> - Renamed replaceIfEquals to replaceIfUnmodified >>>>> - Added remove and removeIfUnmodified. >>>>> - Added containsKey command. >>>>> - Added getWithCas command so that cas value can be returned. I decided >>>>> for a separate command rather than adding cas to get return because you >>>>> don't always want cas to be returned. Having a separate command makes >>>>> better use of network bandwith. >>>>> - Added stats command. JMX attributes are basically accessible through >>>>> this, including cache size. >>>>> - Added error handling section and updated status codes. >>>>> >>>>> Note that Mircea added some interesting comments and I replied to them >>>>> directly in the wiki. >>>>> >>>>> Still remaining to add: >>>>> - Commands: putForExternalRead evict, clear, version, name and quite >>>>> commands. >>>>> - Passing flags. >>>>> >>>>> Regards, >>>>> >>>>> p.s. Updating this has been quite a struggle due to F12 + FF 3.5.5 >>>>> crashing at least 5 times, plus parts of the wiki dissapearing after >>>>> publishing them! >>>>> -- >>>>> Galder Zamarreño >>>>> Sr. Software Engineer >>>>> Infinispan, JBoss Cache >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> -- >>>> Manik Surtani >>>> manik(a)jboss.org >>>> Lead, Infinispan >>>> Lead, JBoss Cache >>>> http://www.infinispan.org >>>> http://www.jbosscache.org >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> -- >>> Galder Zamarreño >>> Sr. Software Engineer >>> Infinispan, JBoss Cache >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 16 Dec 2009, at 12:13, Mircea Markus wrote: > > On Dec 16, 2009, at 12:08 PM, Manik Surtani wrote: > >> >> On 16 Dec 2009, at 11:54, Mircea Markus wrote: >> >>> >>> On Dec 16, 2009, at 9:26 AM, Galder Zamarreno wrote: >>> >>>> >>>> >>>> On 12/15/2009 01:13 PM, Manik Surtani wrote: >>>>> A few comments: >>>>> >>>>> - Why do you have OpCode in your response header? Surely this is redundant? If the client is sync, it knows what it sent. If it is async, it has a message ID. >>>> >>>> True, I fixed that. >>>> >>>>> - 'Not So Dumb' and 'Clever' response headers should be optional? Surely this stuff is only sent when there is a topology change? We also may need some extra info here - how does the back-end know to send this info? If a client hits different back-end nodes, and there is a topology change, how does Node A decide that it should not bother with topology info since the client already hit Node B after the topo change and has the new topology map? Perhaps a TopologyVersion (== JGroups View ID) should be sent back with any topo map, and the client would send it's current TopologyVersion with every request (non-dumb clients only)? Could be a vlong... >>>> >>>> Yeah, as you and Mircea suggest, clients sending the view id could help >>>> better manage size of responses. >>>> >>>> Something to note here is that not all cluster view changes necessarily >>>> involve a client side topology view change, cos if you have N infinispan >>>> nodes in the cluster, you could have M running hot rod server where M<= >>>> N. So, Hot rod will need to figure out when there's been a view change >>>> in the cluster that involves the addition or removal of a hot rod >>>> running Infinispan instance (similar thing to what happens when a >>>> clustered EJB is deployed, the target list for that clustered EJB gets >>>> updated). >>> yep, that's a good point. Every Hotrod server has to know exactly what other hotrod server are there running. I assume that because one way or the other each hotrod node will have to piggyback the list of active hotrod servers to the clients. >> >> Yes, nodes that are not running HotRod are removed from the list prior to transmission. >> >>> Can't the hotrod server track the topology changes (topology of hot rod servers) and increase the view id number at that time only? >> >> That would be reimplementing a lot of logic that is already in JGroups. Maintaining view IDs is non-trivial if (a) any node can update this and (b) this needs to be kept coherent even as nodes leave/join (including any node maintaining this state). > some logic will need to be there to make the difference between an jgroups node leaving and a an jgroups node where hotrod is deployed leaving. Each HR server will need to know the other HR servers running (different from jgroups level view), so we already have some view management logic. I expect this to be declarative (in XML). Then you just filter the JGroups view against the nodes you have in the list of HR servers to determine what to send back.

>> >>> Actually I think it can also rely on the jgroups view id, but only update the hotrod-view-id when jgroups cluster change overlaps with hotrod cluster change (i.e. a infinispan instance that has a hotrod serv started is added or removed). >> >> Precisely. >> >>>> So, bearing in mind this, could we just use the JGroups view id for >>>> this? AFAIK, it's 0 based long but shouldn't cause problems with >>>> complete cluster restarts. If the whole cluster gets restarted, existing >>>> connections will be closed and at that point, clients could revert back >>>> to trying to connect one of their known hot rod servers and pass -1 >>> wouldn't they pass the old view Id at this time rather than -1? Guess the client will realize that the Socket is closed, and at that time will send an "I don't have view" request, i.e. -1? >> >> No, socket being closed doesn't mean your view is out of date - it just means you fail over to the next server. A socket could close for reasons other than a server leaving the group. >> >>>> as >>>> view id which means that I have no view, so the responding server would >>>> send back the hot rod cluster view. >>>> >>>> I'll add this to the wiki. >>> >>>> >>>>> >>>>> Cheers >>>>> Manik >>>>> >>>>> >>>>> On 14 Dec 2009, at 20:08, Galder Zamarreno wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> Re: http://community.jboss.org/wiki/HotRodProtocol >>>>>> >>>>>> I've updated the wiki with the following stuff: >>>>>> - Renamed replaceIfEquals to replaceIfUnmodified >>>>>> - Added remove and removeIfUnmodified. >>>>>> - Added containsKey command. >>>>>> - Added getWithCas command so that cas value can be returned. I decided >>>>>> for a separate command rather than adding cas to get return because you >>>>>> don't always want cas to be returned. Having a separate command makes >>>>>> better use of network bandwith. >>>>>> - Added stats command. JMX attributes are basically accessible through >>>>>> this, including cache size. >>>>>> - Added error handling section and updated status codes. >>>>>> >>>>>> Note that Mircea added some interesting comments and I replied to them >>>>>> directly in the wiki. >>>>>> >>>>>> Still remaining to add: >>>>>> - Commands: putForExternalRead evict, clear, version, name and quite >>>>>> commands. >>>>>> - Passing flags. >>>>>> >>>>>> Regards, >>>>>> >>>>>> p.s. Updating this has been quite a struggle due to F12 + FF 3.5.5 >>>>>> crashing at least 5 times, plus parts of the wiki dissapearing after >>>>>> publishing them! >>>>>> -- >>>>>> Galder Zamarreño >>>>>> Sr. Software Engineer >>>>>> Infinispan, JBoss Cache >>>>>> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>>> -- >>>>> Manik Surtani >>>>> manik(a)jboss.org >>>>> Lead, Infinispan >>>>> Lead, JBoss Cache >>>>> http://www.infinispan.org >>>>> http://www.jbosscache.org >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> -- >>>> Galder Zamarreño >>>> Sr. Software Engineer >>>> Infinispan, JBoss Cache >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 16 Dec 2009, at 14:04, Galder Zamarreno wrote: >> I expect this to be declarative (in XML). Then you just filter the JGroups view against the nodes you have in the list of HR servers to determine what to send back. >> > I was expecting to this in similar to way to how clustered EJBs do. When > you you deploy a clustered EJB, it sends a sync message around the > cluster indicating to the other servers that there's a new endpoint for > that clustered EJB and hence, they update a server side list of HATarget > which is what is send back to the clients in piggyback. > > I was thinking that when a HR server deploys, I could do a similar > thing. You could then use the existing JGroups view id and associate it > with the current HR server target list. > > This however assumes that the server side HR servers are actually joined > in a single cluster, which might not necessarily suit all situations. > Maybe Mircea's suggestion makes sense - a separate, internal Cache instance that uses REPL_ASYNC or something, to propagate a HR server list. -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 17 Dec 2009, at 11:58, Bela Ban wrote: > Do you guys need something from me, e.g. a unique ID (over time and > space) for ViewId ? > > Note that ViewId consists of an Address (UUID) and an ID, so this could > be returned as 3 longs... > Isn't a ViewID just a long? ViewID.getId() ? > Manik Surtani wrote: > >> On 16 Dec 2009, at 14:04, Galder Zamarreno wrote: >> >> >> >>>> I expect this to be declarative (in XML). Then you just filter the JGroups view against the nodes you have in the list of HR servers to determine what to send back. >>>> >>>> >>> I was expecting to this in similar to way to how clustered EJBs do. When >>> you you deploy a clustered EJB, it sends a sync message around the >>> cluster indicating to the other servers that there's a new endpoint for >>> that clustered EJB and hence, they update a server side list of HATarget >>> which is what is send back to the clients in piggyback. >>> >>> I was thinking that when a HR server deploys, I could do a similar >>> thing. You could then use the existing JGroups view id and associate it >>> with the current HR server target list. >>> >>> This however assumes that the server side HR servers are actually joined >>> in a single cluster, which might not necessarily suit all situations. >>> >>> >> Maybe Mircea's suggestion makes sense - a separate, internal Cache instance that uses REPL_ASYNC or something, to propagate a HR server list. >> >> >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> > -- > Bela Ban > Lead JGroups / Clustering Team > JBoss > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 12/17/2009 02:45 PM, Bela Ban wrote: > Correct, but that ID by itself is NOT unique over space& time. I don't > know though whether you ned this uniqueness... > Hmmmm, I don't think the consequences of it not being unique are huge. Let's say a client has vClient=10 that consists of nodeA and nodeB and then the server generates a vServer=10 with nodeA, nodeC and nodeD. The server won't send back the topology but the client will be working sub-optimally until a new view is generated since it will be sending everything to the sole remaining node in his view, nodeA. If the new view had included only nodeE and nodeF (assumming nodeA and nodeB have been brought down), the client would need to reconnect to a known server somehow and would start at -1, so it'd get a new topology in first response.

> Hmmmm, those two sentences seem to contradict each other. When there's a full cluster restart, and your sockets gets "closed by peer" and you can't contact any of the remaining servers, next time I think it's better if you send -1 rather than trying to send the old view. Otherwise, you might have issues like this: - Client had vClient=2. - Cluster restarts and after two hot rod cluster nodes started up, vServer is 2. - Client figured out that the cluster restarted and restablishes the contact and sends vClient=2. - The server does not send an updated view. Now, this would be a problem for example if while the server was down, you had changed the TCP port where the hot rod server was listening, or there was some other change that might have affected the calculation of the cluster node hashcode. If the client sends -1, you avoid this issues, it's like a local reset.