Re: [jboss-cluster-dev] MyRpcDispatcher

Adding Infinispan dev list. -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache ----- galder(a)redhat.com wrote: > As a FYI: > > I'm currently testing this and it's interesting to see that the order > of 1) and 2) as below matters: > > 1) disp1=new MyRpcDispatcher((short)100, c1, null, null, new > Server("foo1()")); > disp2=new MyRpcDispatcher((short)0, c1, null, null, new > Server("foo2()")); > disp3=new MyRpcDispatcher((short)150, c1, null, null, new > Server("foo3()")); > > 2) MyUpHandler handler=new MyUpHandler(); > c1.setUpHandler(handler); > > The MyRpcDispatcher constructor relies on MessageDispatcher ctror that > does: > > prot_adapter=new ProtocolAdapter(); > ... > channel.setUpHandler(prot_adapter); > > And after all those ctros have executed, we override the handler in > 2). If you execute 2) then 1), all messages end up in the last > uphandler set, which means that all calls go to foo3(). The reason I'm > looking into this is because from a consumer perspective, I'm trying > to understand what Hot Rod server would need to give to Infinispan or > whoever managers the channel. > > Note that after 2) comes: > > 3) handler.put((short)100, disp1.getProtocolAdapter()); > handler.put((short)0, disp2.getProtocolAdapter()); > handler.put((short)150, disp3.getProtocolAdapter()); > > Bearing all this mind, I see Hot Rod server providing its own > Rpcdispatcher in 1) and then 3). 3) might not need to rely on Hot Rod > server entire if out MyRpcDispatcher, whoever executes 3) can figure > out the scope number. In other words 3) could be make to look like > this assuming that disp1, disp2 and disp3 are defined as > MyRpcDispatcher rather than RpcDispatcher > > handler.put(disp1.scope, disp1.getProtocolAdapter()); > handler.put(disp2.scope, disp2.getProtocolAdapter()); > handler.put(disp3.scope, disp3.getProtocolAdapter()); > > My investigation continues... > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > > ----- "Bela Ban" <bban(a)redhat.com&gt; wrote: > > > OK, why don't you take a look ? JGRP-1177 has sample code attached > > showing how to use this. > > > > I'll respond in more detail tomorrow > > > > galder(a)redhat.com wrote: > > > Indeed, I think I need this as well for the Hot Rod use case. Hot > > Rod servers need to know which other Hot Rod servers are running in > > the cluster so that they can forward the corresponding information > > back to clients for load balancing, failover...etc. > > > > > > I don't want to polute the core cache channel code having to deal > > with Hot Rod rpcs. > > > > > > Also, a separate channel over a shared transport does not make > sense > > since the Hot Rod is tightly linked to the core cache channel that > > provides data replication capabilities amongst other things. IOW, > if > > Hot Rod used a separate channel for this, you could have scenarios > > where a Hot Rod servers thinks that HotRodServer1, HotRodServer2, > and > > HotRodServer3 are the hot rod servers available in the cluster, but > > underneath the core cache channel would only be formed of > CoreCache1 > > and CoreCache2, with CoreCache3 isolated. This would be bad since a > > client could send requests to HotRodServer3 expecting it to > replicate > > to 1 or 2 but that would not happen. If the same view was used, > > clients would realise that HotRodServer3 is not part of the cluster > > any more. > > > > > > I'll check the current proposed solution for > > https://jira.jboss.org/jira/browse/JGRP-1177 tomorrow and comment > > further. > > > > > > Bela, when do u expect to have something that I can test? > > > > > > Cheers, > > > > > > p.s. I'm adding cluster-dev list to discussion since this is a > true > > inter cluster project concern. > > > -- > > > Galder Zamarreño > > > Sr. Software Engineer > > > Infinispan, JBoss Cache > > > > > > ----- "Brian Stansberry" <brian.stansberry(a)redhat.com&gt; wrote: > > > > > > > > >> Paul, > > >> > > >> Was just chatting with Galder and it seems Infinispan internally > > could > > >> > > >> also have a use for this. (Their HotRodServers would want to > make > > RPCs > > >> > > >> but he doesn't want to corrupt the core infinispan usage with > > that.) > > >> > > >> Can you pull together your thoughts on this, respond to Bela, > bring > > in > > >> > > >> Galder and get this pushed through? Galder's also the most > > >> knowledgable > > >> guy on the Hibernate/ISPN integration, which is the other big > use > > case > > >> > > >> for this. > > >> > > >> Thanks, > > >> > > >> Brian > > >> > > >> On 04/06/2010 03:48 PM, Brian Stansberry wrote: > > >> > > >>> Below is the transcript from our chat. > > >>> > > >>> Only other thoughts I had on this were: > > >>> > > >>> 1) If MyUpHandler exposed a setter for defaultHandler instead > of > > >>> > > >> using > > >> > > >>> scope 0 that would be good. > > >>> > > >>> 2) If JGroups shipped an UpHandler impl that sent a Message > with > > a > > >>> marker object like NoHandlerForRpc, whoever's responsible for > > >>> > > >> setting > > >> > > >>> MyUpHandler on the channel could also set that default handler. > > As > > >>> > > >> a > > >> > > >>> convenience an overloaded constructor on MyUpHandler could do > > this > > >>> > > >> for > > >> > > >>> you based on a boolean param. > > >>> > > >>> 3) If JGroups shipped a RspFilter impl that returned false to > > >>> isAcceptable(Object response, Address sender) if response > > >>> > > >> instanceof > > >> > > >>> NoHandlerForRpc, then code that was concerned about getting > > >>> NoHandlerForRpc could pass that to RpcDispatcher call remote > > >>> > > >> methods. > > >> > > >>> 4) The impl of 3) above should also accept an optional delegate > > >>> > > >> whose > > >> > > >>> methods would be invoked if the response isn't an instance of > > >>> NoHandlerForRpc. That could be used in case whatever was making > > the > > >>> > > >> RPC > > >> > > >>> also wanted to filter responses some other way. > > >>> > > >>> The above plus what we discussed below I think pretty nicely > > >>> encapsulates the guts of this inside JGroups code. The users of > > >>> > > >> this > > >> > > >>> stuff would then have to: > > >>> > > >>> A) instantiate the channel (if not already done) > > >>> > > >>> B) instantiate and set the MyUpHandler and set up the default > > >>> > > >> handler > > >> > > >>> (if not already done) > > >>> > > >>> C) Register their RPC dispatcher > > >>> > > >>> D) Pass in the channel to Infinispan, which can be clueless > about > > >>> > > >> all of > > >> > > >>> this if they so choose. > > >>> > > >>> E) When making any RPCs, pass in the RspFilter impl described > > >>> > > >> above. > > >> > > >>> If the Channel already existed (e.g. a second service sharing a > > >>> CacheManager), then the user service would need to figure out a > > way > > >>> > > >> to > > >> > > >>> get ahold of it (does Infinispan expose it)? and then do C and > E > > >>> > > >> above. > > >> > > >>> I think the latter scenario should discouraged though. > Different > > >>> services sharing the same channel is always problematic. Bela > > >>> > > >> convinced > > >> > > >>> me that by default the different AS services should use > different > > >>> CacheManager's/Channels, so all of the above would just be > needed > > if > > >>> > > >> we > > >> > > >>> want to support CacheManager sharing between > > >>> > > >> HTTP/SFSB/Hib/HAPartition. > > >> > > >>> (02:07:31 PM) besYIM: this class extends ReceiverAdapter, but > > AFAICT > > >>> > > >> it > > >> > > >>> doesn't need to; it doesn't do anything w/ that API > > >>> (02:08:21 PM) paul_m_ferraro: I hadn't even noticed that > > >>> (02:09:11 PM) besYIM: good; so that means I probably didn't > miss > > >>> anything :) > > >>> (02:10:15 PM) besYIM: the part I don't see is where MyUpHandler > > >>> put/remove get called > > >>> (02:10:30 PM) besYIM: ah, in start > > >>> (02:10:37 PM) paul_m_ferraro: yep > > >>> (02:12:28 PM) besYIM: this would need to be used from 1) AS 2) > > >>> > > >> Hibernate > > >> > > >>> 2LC provider 3) Infinispan > > >>> (02:13:14 PM) paul_m_ferraro: why not put in jgroups directly? > > >>> (02:13:56 PM) besYIM: yeah, i expect so > > >>> (02:13:58 PM) paul_m_ferraro: in org.jgroups.blocks? > > >>> (02:14:02 PM) besYIM: yep > > >>> (02:14:17 PM) besYIM: i was thinking about what clients would > > need > > >>> > > >> to be > > >> > > >>> adapted to use it > > >>> (02:15:33 PM) besYIM: i.e. what ISPN would have to change to > deal > > >>> > > >> with > > >> > > >>> the fact this *may* be there, but typically isn't > > >>> (02:17:15 PM) paul_m_ferraro: oh - I see you point > > >>> (02:17:19 PM) paul_m_ferraro: your > > >>> (02:17:22 PM) besYIM: i'm trying to figure where that > > >>> MyRpcDispatcher.createRequestCorrelator() method gets invoked > > >>> (02:17:40 PM) besYIM: since that method smells like a nice hook > > to > > >>> > > >> hide > > >> > > >>> all this :) > > >>> (02:18:21 PM) paul_m_ferraro: infinispan uses their own > > >>> > > >> RpcDispatcher > > >> > > >>> extension, right? > > >>> (02:18:33 PM) besYIM: yeah, i believe so > > >>> (02:19:07 PM) paul_m_ferraro: so the overridden UpHandler would > > >>> > > >> just > > >> > > >>> fall through to the default behavior if no scope header exists > > >>> (02:20:23 PM) paul_m_ferraro: the question is, how to override > > >>> infinispan so that it too can be multiplexed by scope > > >>> (02:20:31 PM) paul_m_ferraro: no? > > >>> (02:20:43 PM) besYIM: yep > > >>> (02:21:33 PM) besYIM: the createRequestCorrelatorMethod seems > to > > be > > >>> > > >> the way > > >> > > >>> (02:21:42 PM) besYIM: well, maybe > > >>> (02:22:20 PM) besYIM: if it had knowledge as whether the > > UpHandler > > >>> > > >> was > > >> > > >>> MyUpHandler it could change the type of RequestCorrelator it > > >>> > > >> creates > > >> > > >>> (02:22:37 PM) besYIM: std UpHandler, just create the std one > > >>> (02:22:47 PM) besYIM: special one, create MyRequestCorrelator > > >>> (02:25:49 PM) paul_m_ferraro: so where would Infinispan get > it's > > >>> > > >> scope > > >> > > >>> from? > > >>> (02:26:14 PM) besYIM: "well known scopes" wiki page? > > >>> (02:26:45 PM) besYIM: i suppose adding it to the ISPN config > > would > > >>> > > >> be > > >> > > >>> better > > >>> (02:28:28 PM) besYIM: ah, i found the call to > > >>> > > >> createRequestCorrelator -- > > >> > > >>> in MessageDispatcher.start() > > >>> (02:31:16 PM) besYIM: none of the params passed to that method > > >>> > > >> provide > > >> > > >>> any info on the type of the UpHandler, but > > MessageDispatcher.channel > > >>> > > >> is > > >> > > >>> protected, so the method impl could just look at the Channel > > >>> (02:32:52 PM) paul_m_ferraro: I should think the > MyRpcDispatcher > > >>> > > >> should > > >> > > >>> also register itself w/the MyUpHandler, if its channel used it > > >>> (02:33:02 PM) besYIM: yep > > >>> (02:33:44 PM) besYIM: this could all be part of the std > > >>> > > >> RpcDispatcher > > >> > > >>> (02:34:43 PM) paul_m_ferraro: yep - and can use a default > scope, > > if > > >>> > > >> none > > >> > > >>> specified > > >>> (02:35:40 PM) besYIM: yeah, so infinispan could deal with > > >>> > > >> configuring > > >> > > >>> scope or not > > >>> (02:40:38 PM) besYIM: the other concern i had was lifecycle > > issues > > >>> > > >> -- > > >> > > >>> i.e. transient issues with a channel not disconnected but a > > handler > > >>> removed e.g. during service undeployment > > >>> (02:41:34 PM) besYIM: or deployment > > >>> (02:42:52 PM) paul_m_ferraro: so, for example, a message > arrives > > >>> > > >> w/scope > > >> > > >>> X but scope X is no longer registered w/the UpHandler? > > >>> (02:43:16 PM) besYIM: or hasn't yet been registered > > >>> (02:43:52 PM) besYIM: like lets assume to make it hard that > HTTP > > >>> session, SFSB, Hib 2LC will all use the same CacheManager and > > thus > > >>> > > >> same > > >> > > >>> Channel > > >>> (02:44:44 PM) besYIM: nodes A and B are fully started; node C > is > > >>> > > >> coming > > >> > > >>> on line. It deploys a distributable webapp, so cache is > started, > > >>> > > >> Channel > > >> > > >>> connected, etc > > >>> (02:45:10 PM) besYIM: but there is no handler for the SFSB and > > Hib > > >>> > > >> 2LC > > >> > > >>> stuff yet because those aren't deployed yet > > >>> (02:46:11 PM) besYIM: HAPartition deals with this via > > >>> > > >> NoHandlerForRPC > > >> > > >>> (02:51:02 PM) paul_m_ferraro: Well, this is the responsibility > of > > >>> whomever created the scoped UpHandler > > >>> (02:51:57 PM) paul_m_ferraro: bela's impl throws a > > >>> IllegalArgumentException - but it should be something more > > specific > > >>> (02:52:01 PM) besYIM: yeah > > >>> (02:52:31 PM) besYIM: but if it throws an exception I believe > > that > > >>> > > >> will > > >> > > >>> propagate back to the MyRequestCorrelator who sent the RPC and > > then > > >>> > > >> get > > >> > > >>> thrown > > >>> (02:52:32 PM) paul_m_ferraro: and perhaps infinispan should > allow > > a > > >>> pluggable handler for this? > > >>> (02:53:02 PM) paul_m_ferraro: so those cache users that want to > > >>> > > >> ignore > > >> > > >>> it can, or retry it, etc. > > >>> (02:53:13 PM) besYIM: I don't think Infinispan will have a > > problem > > >>> (02:53:24 PM) paul_m_ferraro: retry the operation, if possible > > >>> (02:53:49 PM) besYIM: Infinispan is the one who actually calls > > >>> Channel.connect, so they are always registered first > > >>> (02:54:24 PM) besYIM: it's the specialized handlers we'd add > for > > >>> > > >> SFSB > > >> > > >>> ownership, or the Hibernate eviction thing that would be > missing > > >>> (02:59:17 PM) besYIM: i was looking at the org.jgroups.Request > > >>> > > >> class > > >> > > >>> (02:59:58 PM) besYIM: it's receiveResponse method is a > > centralized > > >>> > > >> place > > >> > > >>> that could deal with something like a NoHandlerForRPC > > >>> (03:00:10 PM) besYIM: i.e. discard it but mark the response > > >>> > > >> received > > >> > > >>> (03:02:41 PM) besYIM: a simple impl of > > org.jgroups.blocks.RspFilter > > >>> could also work -- add such a thing to JGroups and then AS, > EJB3, > > >>> Hibernate could use it for the few RPCs they make > > >>> (03:06:45 PM) paul_m_ferraro: I see > > >>> (03:06:57 PM) paul_m_ferraro: damn - its 4 > > >>> (03:07:07 PM) paul_m_ferraro: today's my day to pick up marcus > > from > > >>> preschool > > >>> (03:07:19 PM) besYIM: that's always fun :). enjoy! > > >>> > > >> -- > > >> Brian Stansberry > > >> Lead, AS Clustering > > >> JBoss by Red Hat > > >> > > > > -- > > Bela Ban > > Lead JGroups / Clustering Team > > JBoss