Hi all, Re: https://community.jboss.org/docs/DOC-17571 Over the past week and a bit I've been working on a rough prototype for remote event handling in Hot Rod that covers the server side (I've not done any work on the Hot Rod client).In the link above you can find my design notes. I wanted to get some feedback on the minimum requirements explained and I wanted to discuss the need of the optional requirements, in particular the 1st of the optional requirements. The idea is that at a logical level, it'd be interesting to know what the origin of a modification for a couple of reasons: - It allows clients be able to know whether the modification is originated locally from a logical point of view. If it sounds to abstract, think about near caches (see preso in https://www.jboss.org/dms/judcon/presentations/London2011/day1track2sessi...) and imagine a local cache (near cache) configured with a remote cache store (a Java hot rod client with N channels open at the same time). Remote listeners could decide to act differently if the modification was originated locally or not, i.e. if it's not local then remove the key from cache, if local, it means the modification comes from this remote cache store and so I have the latest data in memory. This is a very nice optimisation for at least this use case. - This can be extended further. If all channels opened with can be associated with a logical origin, we could optimise sending back events. For example, imagine a remote cache store (it has 1 remote cache manager) that has N channels open with the server. There's no need for all N channels to receive a notification for a cache removal. As long as one of the channels gets it event, it's good enough to act to on the local cache. As you can see, what I'm heading towards is that for each remote cache manager started to be id'd uniquely, and this id to be shipped with all Hot Rod operations. It could be possible to limit the operations that carry such an id, but this could complicate the protocol. Thoughts? Also, any thoughts on the need for the 4th optional requirement? For near caches, remote events are important, but you could limit the need for it with an aggressive eviction policy in the near cache to cover against lost events. Cheers, -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

> > * Not sure I get the "option #1" in your doc? If you cared about locally originating events (and want to behave differently), you'd just register that listener using the embedded API and not the remote one? Hmmm, not sure I understand your question…

I thought I had made myself clear enough with the wiki and explanation on the email. Let me try again: Imagine a near cache scenario: 1. Client A interacts with a near cache (i.e. embedded Infinispan with a remote cache store) and stores V1 in key=k 2. Client B interacts with near cache and retrieves key=k. The req goes to server and returns V1 3. Client B goes and updates key=k to V2 4. Client A receives a notification for key=k that it has been updated and it decides to delete it from the near cache. 5. Client B receives a notification for key=k that it has been updated and it decides to delete it from the near cache. Step 5. is suboptiomal because the update originiates at Client B. The idea of the "origin" is that the server could potentially be able to tell client B that the notification is the result of an operation that started 'locally' and so client B could read that and decide to not delete it from the near cache. Client A when it receieves the notification it realises that the notification is not originated locally and can decide to delete the key from the near cache.

>> >> Client A when it receieves the notification it realises that the notification is not originated locally and can decide to delete the key from the near cache. > > Client A and Client B are two threads in the same VM as the embedded cache with a remote cache store? Maybe, but not necessarily. It could very easily happen that Client A and Client B are running on the same machine but in different VMs.

On Feb 22, 2012, at 2:57 PM, Manik Surtani wrote: > > On 22 Feb 2012, at 13:53, Galder Zamarreño wrote: > >>>> >>>> Client A when it receieves the notification it realises that the notification is not originated locally and can decide to delete the key from the near cache. >>> >>> Client A and Client B are two threads in the same VM as the embedded cache with a remote cache store? >> >> Maybe, but not necessarily. It could very easily happen that Client A and Client B are running on the same machine but in different VMs. > > Ok, just trying to understand your use case. So we assume Client A and Client B are different JVMs, with different Infinispan instances, but talking to the same Hot Rod servers via the RemoteCacheStore? Yeah, for example, Client A and Client B could be two AS7 instances clustered that are running on the same machine, possibly part of a bigger cluster. Granted, if you have a cluster set betwen these two AS7 instances already, you could just cluster the near caches and avoid the need for notifications. However, I don't think the use case I hinted above seems that far fetched.

> > -- > Manik Surtani > manik(a)jboss.org > twitter.com/maniksurtani > > Lead, Infinispan > http://www.infinispan.org > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache

On 21 Feb 2012, at 16:00, Galder Zamarreño wrote: >> >> For the implementation, I'd be interested in what you have in mind, especially from a performance perspective. I'm adding Clebert and Mike in cc, since some of the stuff they do is related to such event bus/notification/pub-sub models and they may have insights to add. > > So far I've found 2 factors to be important here when it comes to performance: > 1. Avoid remote eventing to have a major impact on consumption of cache operations. > 2. Avoid swamping clients with too many notifications. > > For 1, I had thought about implementing notification in a sync=false listener. Yes, that would be necessary. It should be async on the server side. > For 2, imagine a client that starts a remote cache manager, signs up for notifications in cache C, and has 50 threads interacting with cache C concurrently (so, 50 channels are open with the server). I don't want the server to send back 50 events for each interested cache operation that happens on the server side. 1 notification should be enough. This is one of the reasons I want "option #1". Yes, the server definitely needs to be smart enough to identify multiple connections from the same client, and this also needs to be distributed.

E.g., if client C is connected to 2 server nodes S1 and S2, we don't want both S1 and S2 to send back the same notification,

Also, what are your thoughts around batching notifications?

Cheers Manik -- Manik Surtani manik(a)jboss.org twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 22 Feb 2012, at 10:05, Galder Zamarreño wrote: >> >>> For 2, imagine a client that starts a remote cache manager, signs up for notifications in cache C, and has 50 threads interacting with cache C concurrently (so, 50 channels are open with the server). I don't want the server to send back 50 events for each interested cache operation that happens on the server side. 1 notification should be enough. This is one of the reasons I want "option #1". >> >> Yes, the server definitely needs to be smart enough to identify multiple connections from the same client, and this also needs to be distributed. > > +1, but the question is, how do you define "same client"? This is what I was getting to with "origin" earlier (a way to identify cache managers). You can't assume that a client IP to differentiate between different clients cos you could have multiple Hot Rod clients running independently in a machine. > > If you have any other ideas, I'm happy to hear Each client could be assigned a UUID when it first connects… and subsequent messages could include this UUID in a header. Hmm, could get expensive though.

> >> E.g., if client C is connected to 2 server nodes S1 and S2, we don't want both S1 and S2 to send back the same notification, > > +1 again, this can very easily done by using the isLocal() call in listeners. I was only planning to send notifications from the node where the operation is local, which gets around this issue. +1. >> Also, what are your thoughts around batching notifications? > > This could be handy to avoid overloading clients as well, but wasn't in my initial plans. > > What might be important at this stage is if we consider batching of notifications to be important, whether we'd want to embedd it into the protocol, so that an event notification response could return 1 to N notifications in a single message. This would be more optimal and should not result in a huge message since values will not be sent over, only keys if anything. > > Otherwise, batching of notifications could be implemented at a later stage using a similar method to the replication queue. We could even consider using disruptor instead of a blocking queue… Lets start without batching then, at least for a first pass. Cheers Manik -- Manik Surtani manik(a)jboss.org twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On Feb 22, 2012, at 5:26 PM, Galder Zamarreño wrote: > > On Feb 22, 2012, at 1:00 PM, Manik Surtani wrote: > >> >> On 22 Feb 2012, at 10:05, Galder Zamarreño wrote: >> >>>> >>>>> For 2, imagine a client that starts a remote cache manager, signs up for notifications in cache C, and has 50 threads interacting with cache C concurrently (so, 50 channels are open with the server). I don't want the server to send back 50 events for each interested cache operation that happens on the server side. 1 notification should be enough. This is one of the reasons I want "option #1". >>>> >>>> Yes, the server definitely needs to be smart enough to identify multiple connections from the same client, and this also needs to be distributed. >>> >>> +1, but the question is, how do you define "same client"? This is what I was getting to with "origin" earlier (a way to identify cache managers). You can't assume that a client IP to differentiate between different clients cos you could have multiple Hot Rod clients running independently in a machine. >>> >>> If you have any other ideas, I'm happy to hear >> >> Each client could be assigned a UUID when it first connects… and subsequent messages could include this UUID in a header.  Hmm, could get expensive though. > > Hmmm, not sure that could work. It's the client that knows whether two channels belong to the same client (i.e. two channels generated by same remote cache manager for example). > > So, I'm inclined to think that such ID should be generated by the client itself. If you wanna avoid sending it with each request, you'd have to assume that there's a first-request where that info comes, and assume that for the rest of the time the channel is open, that won't change. This is a fair assumption but complicates clients cos they need to differentiated between the first and any subsequent requests. > > I'm investigating other possibilities. Btw, I'm reading http://sigops.org/sosp/sosp11/current/2011-Cascais/10-adya-online.pdf which talks about Google's Thialfi client notification service and they have a similar mechanism. "When present, the optional source parameter identifies the client that made the change. (This ID is provided by the application client at startup and is referred to as its application ID.) As an optimization, Thialfi omits delivery of the notification to this client, since the client already knows about the change." So, they go for the option of providing this logical ID on startup and then be able to avoid sending the notification to the originating node. I think we could add this app or logica id as part of the add/register listener call and avoid sending it with all operations. Tbh, only those nodes that have register a listener care about knowing the event was generated locally or not, and only those care about not receiving notifications in all their channels. So, I think this could work.

Thanks for putting this together, it looks good. A few comments: * The Hot Rod Java Client API should probably look more like JSR 107's listener API rather than Infinispan's annotation-based one. In future (Infinispan 6?) we'll deprecate our core annotation based API in favour of JSR 107's one. * Not sure I get the "option #1" in your doc? If you cared about locally originating events (and want to behave differently), you'd just register that listener using the embedded API and not the remote one? * Not sure I understand option #3. Is this to allow attaching a listener to a key while a key is added? * For option #4, no - not for now anyway. Too much complexity. For the implementation, I'd be interested in what you have in mind, especially from a performance perspective. I'm adding Clebert and Mike in cc, since some of the stuff they do is related to such event bus/notification/pub-sub models and they may have insights to add. Cheers Manik On 20 Feb 2012, at 14:29, Galder Zamarreño wrote: > Hi all, > > Re: https://community.jboss.org/docs/DOC-17571 > > Over the past week and a bit I've been working on a rough prototype for remote event handling in Hot Rod that covers the server side (I've not done any work on the Hot Rod client).In the link above you can find my design notes. > > I wanted to get some feedback on the minimum requirements explained and I wanted to discuss the need of the optional requirements, in particular the 1st of the optional requirements. > > The idea is that at a logical level, it'd be interesting to know what the origin of a modification for a couple of reasons: > > - It allows clients be able to know whether the modification is originated locally from a logical point of view. If it sounds to abstract, think about near caches (see preso in https://www.jboss.org/dms/judcon/presentations/London2011/day1track2sessi...) and imagine a local cache (near cache) configured with a remote cache store (a Java hot rod client with N channels open at the same time). Remote listeners could decide to act differently if the modification was originated locally or not, i.e. if it's not local then remove the key from cache, if local, it means the modification comes from this remote cache store and so I have the latest data in memory. This is a very nice optimisation for at least this use case. > > - This can be extended further. If all channels opened with can be associated with a logical origin, we could optimise sending back events. For example, imagine a remote cache store (it has 1 remote cache manager) that has N channels open with the server. There's no need for all N channels to receive a notification for a cache removal. As long as one of the channels gets it event, it's good enough to act to on the local cache. > > As you can see, what I'm heading towards is that for each remote cache manager started to be id'd uniquely, and this id to be shipped with all Hot Rod operations. It could be possible to limit the operations that carry such an id, but this could complicate the protocol. > > Thoughts? > > Also, any thoughts on the need for the 4th optional requirement? For near caches, remote events are important, but you could limit the need for it with an aggressive eviction policy in the near cache to cover against lost events. > > Cheers, > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org