Re: [infinispan-dev] [jboss-cluster-dev] MyRpcDispatcher

I've modified Bela's existing sample code to address the issues Brian and I discussed below. Here's a summary of the changes: * Refactored pieces into separate classes with nicer names * ScopedRpcDispatcher registers its scope with a ScopedUpHandler * Added ScopeUpHandler deregistration on ScopedRpcDispatcher.stop(). * ScopedUpHandler accepts an optional default handler, which will be triggered for non-message events or messages with no scope header * ScopedUpHandler returns NoHandlerForScope object if message is received for an unknown scope (e.g. its handler was not yet registered or was already deregistered) * Automatically adds RspFilter to RequestOptions that rejects NoHandlerForScope responses, decorating any existing filter, if necessary. * Added ScopedMessageDispatcher analogue So, usage would look like: Channel c = new JChannel(...); ScopedUpHandler h = new ScopedUpHandlerImpl(); ScopedRpcDispatcher d1 = new ScopedRpcDispatcher((short)1, h, c, null, null, target1); ScopedRpcDispatcher d2 = new ScopedRpcDispatcher((short)2, h, c, null, null, target2); // This must be set after all rpc dispatchers are created c.setUpHandler(h); c.connect(...); This will also work in conjunction with a standard rpc dispatcher: Channel c = new JChannel(...); RpcDispatcher d = new RpcDispatcher(c, null, null, target); ScopedUpHandler h = new ScopedUpHandlerImpl(d.getProtocolAdapter()); ScopedRpcDispatcher d1 = new ScopedRpcDispatcher((short)1, h, c, null, null, target1); ScopedRpcDispatcher d2 = new ScopedRpcDispatcher((short)2, h, c, null, null, target2); c.setUpHandler(h); c.connect(...); I've attached the changes in the form of a patch to JGRP-1177. Comments are welcome. Paul On Fri, 2010-04-09 at 06:55 -0400, Galder Zamarreno wrote: > Adding Infinispan dev list. > > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > > ----- galder(a)redhat.com wrote: > > >> As a FYI: >> >> I'm currently testing this and it's interesting to see that the order >> of 1) and 2) as below matters: >> >> 1) disp1=new MyRpcDispatcher((short)100, c1, null, null, new >> Server("foo1()")); >> disp2=new MyRpcDispatcher((short)0, c1, null, null, new >> Server("foo2()")); >> disp3=new MyRpcDispatcher((short)150, c1, null, null, new >> Server("foo3()")); >> >> 2) MyUpHandler handler=new MyUpHandler(); >> c1.setUpHandler(handler); >> >> The MyRpcDispatcher constructor relies on MessageDispatcher ctror that >> does: >> >> prot_adapter=new ProtocolAdapter(); >> ... >> channel.setUpHandler(prot_adapter); >> >> And after all those ctros have executed, we override the handler in >> 2). If you execute 2) then 1), all messages end up in the last >> uphandler set, which means that all calls go to foo3(). The reason I'm >> looking into this is because from a consumer perspective, I'm trying >> to understand what Hot Rod server would need to give to Infinispan or >> whoever managers the channel. >> >> Note that after 2) comes: >> >> 3) handler.put((short)100, disp1.getProtocolAdapter()); >> handler.put((short)0, disp2.getProtocolAdapter()); >> handler.put((short)150, disp3.getProtocolAdapter()); >> >> Bearing all this mind, I see Hot Rod server providing its own >> Rpcdispatcher in 1) and then 3). 3) might not need to rely on Hot Rod >> server entire if out MyRpcDispatcher, whoever executes 3) can figure >> out the scope number. In other words 3) could be make to look like >> this assuming that disp1, disp2 and disp3 are defined as >> MyRpcDispatcher rather than RpcDispatcher >> >> handler.put(disp1.scope, disp1.getProtocolAdapter()); >> handler.put(disp2.scope, disp2.getProtocolAdapter()); >> handler.put(disp3.scope, disp3.getProtocolAdapter()); >> >> My investigation continues... >> -- >> Galder Zamarreño >> Sr. Software Engineer >> Infinispan, JBoss Cache >> >> ----- "Bela Ban" <bban(a)redhat.com&gt; wrote: >> >> >>> OK, why don't you take a look ? JGRP-1177 has sample code attached >>> showing how to use this. >>> >>> I'll respond in more detail tomorrow >>> >>> galder(a)redhat.com wrote: >>> >>>> Indeed, I think I need this as well for the Hot Rod use case. Hot >>>> >>> Rod servers need to know which other Hot Rod servers are running in >>> the cluster so that they can forward the corresponding information >>> back to clients for load balancing, failover...etc. >>> >>>> I don't want to polute the core cache channel code having to deal >>>> >>> with Hot Rod rpcs. >>> >>>> Also, a separate channel over a shared transport does not make >>>> >> sense >> >>> since the Hot Rod is tightly linked to the core cache channel that >>> provides data replication capabilities amongst other things. IOW, >>> >> if >> >>> Hot Rod used a separate channel for this, you could have scenarios >>> where a Hot Rod servers thinks that HotRodServer1, HotRodServer2, >>> >> and >> >>> HotRodServer3 are the hot rod servers available in the cluster, but >>> underneath the core cache channel would only be formed of >>> >> CoreCache1 >> >>> and CoreCache2, with CoreCache3 isolated. This would be bad since a >>> client could send requests to HotRodServer3 expecting it to >>> >> replicate >> >>> to 1 or 2 but that would not happen. If the same view was used, >>> clients would realise that HotRodServer3 is not part of the cluster >>> any more. >>> >>>> I'll check the current proposed solution for >>>> >>> https://jira.jboss.org/jira/browse/JGRP-1177 tomorrow and comment >>> further. >>> >>>> Bela, when do u expect to have something that I can test? >>>> >>>> Cheers, >>>> >>>> p.s. I'm adding cluster-dev list to discussion since this is a >>>> >> true >> >>> inter cluster project concern. >>> >>>> -- >>>> Galder Zamarreño >>>> Sr. Software Engineer >>>> Infinispan, JBoss Cache >>>> >>>> ----- "Brian Stansberry" <brian.stansberry(a)redhat.com&gt; wrote: >>>> >>>> >>>> >>>>> Paul, >>>>> >>>>> Was just chatting with Galder and it seems Infinispan internally >>>>> >>> could >>> >>>>> also have a use for this. (Their HotRodServers would want to >>>>> >> make >> >>> RPCs >>> >>>>> but he doesn't want to corrupt the core infinispan usage with >>>>> >>> that.) >>> >>>>> Can you pull together your thoughts on this, respond to Bela, >>>>> >> bring >> >>> in >>> >>>>> Galder and get this pushed through? Galder's also the most >>>>> knowledgable >>>>> guy on the Hibernate/ISPN integration, which is the other big >>>>> >> use >> >>> case >>> >>>>> for this. >>>>> >>>>> Thanks, >>>>> >>>>> Brian >>>>> >>>>> On 04/06/2010 03:48 PM, Brian Stansberry wrote: >>>>> >>>>> >>>>>> Below is the transcript from our chat. >>>>>> >>>>>> Only other thoughts I had on this were: >>>>>> >>>>>> 1) If MyUpHandler exposed a setter for defaultHandler instead >>>>>> >> of >> >>>>>> >>>>>> >>>>> using >>>>> >>>>> >>>>>> scope 0 that would be good. >>>>>> >>>>>> 2) If JGroups shipped an UpHandler impl that sent a Message >>>>>> >> with >> >>> a >>> >>>>>> marker object like NoHandlerForRpc, whoever's responsible for >>>>>> >>>>>> >>>>> setting >>>>> >>>>> >>>>>> MyUpHandler on the channel could also set that default handler. >>>>>> >>> As >>> >>>>>> >>>>>> >>>>> a >>>>> >>>>> >>>>>> convenience an overloaded constructor on MyUpHandler could do >>>>>> >>> this >>> >>>>>> >>>>>> >>>>> for >>>>> >>>>> >>>>>> you based on a boolean param. >>>>>> >>>>>> 3) If JGroups shipped a RspFilter impl that returned false to >>>>>> isAcceptable(Object response, Address sender) if response >>>>>> >>>>>> >>>>> instanceof >>>>> >>>>> >>>>>> NoHandlerForRpc, then code that was concerned about getting >>>>>> NoHandlerForRpc could pass that to RpcDispatcher call remote >>>>>> >>>>>> >>>>> methods. >>>>> >>>>> >>>>>> 4) The impl of 3) above should also accept an optional delegate >>>>>> >>>>>> >>>>> whose >>>>> >>>>> >>>>>> methods would be invoked if the response isn't an instance of >>>>>> NoHandlerForRpc. That could be used in case whatever was making >>>>>> >>> the >>> >>>>>> >>>>>> >>>>> RPC >>>>> >>>>> >>>>>> also wanted to filter responses some other way. >>>>>> >>>>>> The above plus what we discussed below I think pretty nicely >>>>>> encapsulates the guts of this inside JGroups code. The users of >>>>>> >>>>>> >>>>> this >>>>> >>>>> >>>>>> stuff would then have to: >>>>>> >>>>>> A) instantiate the channel (if not already done) >>>>>> >>>>>> B) instantiate and set the MyUpHandler and set up the default >>>>>> >>>>>> >>>>> handler >>>>> >>>>> >>>>>> (if not already done) >>>>>> >>>>>> C) Register their RPC dispatcher >>>>>> >>>>>> D) Pass in the channel to Infinispan, which can be clueless >>>>>> >> about >> >>>>>> >>>>>> >>>>> all of >>>>> >>>>> >>>>>> this if they so choose. >>>>>> >>>>>> E) When making any RPCs, pass in the RspFilter impl described >>>>>> >>>>>> >>>>> above. >>>>> >>>>> >>>>>> If the Channel already existed (e.g. a second service sharing a >>>>>> CacheManager), then the user service would need to figure out a >>>>>> >>> way >>> >>>>>> >>>>>> >>>>> to >>>>> >>>>> >>>>>> get ahold of it (does Infinispan expose it)? and then do C and >>>>>> >> E >> >>>>>> >>>>>> >>>>> above. >>>>> >>>>> >>>>>> I think the latter scenario should discouraged though. >>>>>> >> Different >> >>>>>> services sharing the same channel is always problematic. Bela >>>>>> >>>>>> >>>>> convinced >>>>> >>>>> >>>>>> me that by default the different AS services should use >>>>>> >> different >> >>>>>> CacheManager's/Channels, so all of the above would just be >>>>>> >> needed >> >>> if >>> >>>>>> >>>>>> >>>>> we >>>>> >>>>> >>>>>> want to support CacheManager sharing between >>>>>> >>>>>> >>>>> HTTP/SFSB/Hib/HAPartition. >>>>> >>>>> >>>>>> (02:07:31 PM) besYIM: this class extends ReceiverAdapter, but >>>>>> >>> AFAICT >>> >>>>>> >>>>>> >>>>> it >>>>> >>>>> >>>>>> doesn't need to; it doesn't do anything w/ that API >>>>>> (02:08:21 PM) paul_m_ferraro: I hadn't even noticed that >>>>>> (02:09:11 PM) besYIM: good; so that means I probably didn't >>>>>> >> miss >> >>>>>> anything :) >>>>>> (02:10:15 PM) besYIM: the part I don't see is where MyUpHandler >>>>>> put/remove get called >>>>>> (02:10:30 PM) besYIM: ah, in start >>>>>> (02:10:37 PM) paul_m_ferraro: yep >>>>>> (02:12:28 PM) besYIM: this would need to be used from 1) AS 2) >>>>>> >>>>>> >>>>> Hibernate >>>>> >>>>> >>>>>> 2LC provider 3) Infinispan >>>>>> (02:13:14 PM) paul_m_ferraro: why not put in jgroups directly? >>>>>> (02:13:56 PM) besYIM: yeah, i expect so >>>>>> (02:13:58 PM) paul_m_ferraro: in org.jgroups.blocks? >>>>>> (02:14:02 PM) besYIM: yep >>>>>> (02:14:17 PM) besYIM: i was thinking about what clients would >>>>>> >>> need >>> >>>>>> >>>>>> >>>>> to be >>>>> >>>>> >>>>>> adapted to use it >>>>>> (02:15:33 PM) besYIM: i.e. what ISPN would have to change to >>>>>> >> deal >> >>>>>> >>>>>> >>>>> with >>>>> >>>>> >>>>>> the fact this *may* be there, but typically isn't >>>>>> (02:17:15 PM) paul_m_ferraro: oh - I see you point >>>>>> (02:17:19 PM) paul_m_ferraro: your >>>>>> (02:17:22 PM) besYIM: i'm trying to figure where that >>>>>> MyRpcDispatcher.createRequestCorrelator() method gets invoked >>>>>> (02:17:40 PM) besYIM: since that method smells like a nice hook >>>>>> >>> to >>> >>>>>> >>>>>> >>>>> hide >>>>> >>>>> >>>>>> all this :) >>>>>> (02:18:21 PM) paul_m_ferraro: infinispan uses their own >>>>>> >>>>>> >>>>> RpcDispatcher >>>>> >>>>> >>>>>> extension, right? >>>>>> (02:18:33 PM) besYIM: yeah, i believe so >>>>>> (02:19:07 PM) paul_m_ferraro: so the overridden UpHandler would >>>>>> >>>>>> >>>>> just >>>>> >>>>> >>>>>> fall through to the default behavior if no scope header exists >>>>>> (02:20:23 PM) paul_m_ferraro: the question is, how to override >>>>>> infinispan so that it too can be multiplexed by scope >>>>>> (02:20:31 PM) paul_m_ferraro: no? >>>>>> (02:20:43 PM) besYIM: yep >>>>>> (02:21:33 PM) besYIM: the createRequestCorrelatorMethod seems >>>>>> >> to >> >>> be >>> >>>>>> >>>>>> >>>>> the way >>>>> >>>>> >>>>>> (02:21:42 PM) besYIM: well, maybe >>>>>> (02:22:20 PM) besYIM: if it had knowledge as whether the >>>>>> >>> UpHandler >>> >>>>>> >>>>>> >>>>> was >>>>> >>>>> >>>>>> MyUpHandler it could change the type of RequestCorrelator it >>>>>> >>>>>> >>>>> creates >>>>> >>>>> >>>>>> (02:22:37 PM) besYIM: std UpHandler, just create the std one >>>>>> (02:22:47 PM) besYIM: special one, create MyRequestCorrelator >>>>>> (02:25:49 PM) paul_m_ferraro: so where would Infinispan get >>>>>> >> it's >> >>>>>> >>>>>> >>>>> scope >>>>> >>>>> >>>>>> from? >>>>>> (02:26:14 PM) besYIM: "well known scopes" wiki page? >>>>>> (02:26:45 PM) besYIM: i suppose adding it to the ISPN config >>>>>> >>> would >>> >>>>>> >>>>>> >>>>> be >>>>> >>>>> >>>>>> better >>>>>> (02:28:28 PM) besYIM: ah, i found the call to >>>>>> >>>>>> >>>>> createRequestCorrelator -- >>>>> >>>>> >>>>>> in MessageDispatcher.start() >>>>>> (02:31:16 PM) besYIM: none of the params passed to that method >>>>>> >>>>>> >>>>> provide >>>>> >>>>> >>>>>> any info on the type of the UpHandler, but >>>>>> >>> MessageDispatcher.channel >>> >>>>>> >>>>>> >>>>> is >>>>> >>>>> >>>>>> protected, so the method impl could just look at the Channel >>>>>> (02:32:52 PM) paul_m_ferraro: I should think the >>>>>> >> MyRpcDispatcher >> >>>>>> >>>>>> >>>>> should >>>>> >>>>> >>>>>> also register itself w/the MyUpHandler, if its channel used it >>>>>> (02:33:02 PM) besYIM: yep >>>>>> (02:33:44 PM) besYIM: this could all be part of the std >>>>>> >>>>>> >>>>> RpcDispatcher >>>>> >>>>> >>>>>> (02:34:43 PM) paul_m_ferraro: yep - and can use a default >>>>>> >> scope, >> >>> if >>> >>>>>> >>>>>> >>>>> none >>>>> >>>>> >>>>>> specified >>>>>> (02:35:40 PM) besYIM: yeah, so infinispan could deal with >>>>>> >>>>>> >>>>> configuring >>>>> >>>>> >>>>>> scope or not >>>>>> (02:40:38 PM) besYIM: the other concern i had was lifecycle >>>>>> >>> issues >>> >>>>>> >>>>>> >>>>> -- >>>>> >>>>> >>>>>> i.e. transient issues with a channel not disconnected but a >>>>>> >>> handler >>> >>>>>> removed e.g. during service undeployment >>>>>> (02:41:34 PM) besYIM: or deployment >>>>>> (02:42:52 PM) paul_m_ferraro: so, for example, a message >>>>>> >> arrives >> >>>>>> >>>>>> >>>>> w/scope >>>>> >>>>> >>>>>> X but scope X is no longer registered w/the UpHandler? >>>>>> (02:43:16 PM) besYIM: or hasn't yet been registered >>>>>> (02:43:52 PM) besYIM: like lets assume to make it hard that >>>>>> >> HTTP >> >>>>>> session, SFSB, Hib 2LC will all use the same CacheManager and >>>>>> >>> thus >>> >>>>>> >>>>>> >>>>> same >>>>> >>>>> >>>>>> Channel >>>>>> (02:44:44 PM) besYIM: nodes A and B are fully started; node C >>>>>> >> is >> >>>>>> >>>>>> >>>>> coming >>>>> >>>>> >>>>>> on line. It deploys a distributable webapp, so cache is >>>>>> >> started, >> >>>>>> >>>>>> >>>>> Channel >>>>> >>>>> >>>>>> connected, etc >>>>>> (02:45:10 PM) besYIM: but there is no handler for the SFSB and >>>>>> >>> Hib >>> >>>>>> >>>>>> >>>>> 2LC >>>>> >>>>> >>>>>> stuff yet because those aren't deployed yet >>>>>> (02:46:11 PM) besYIM: HAPartition deals with this via >>>>>> >>>>>> >>>>> NoHandlerForRPC >>>>> >>>>> >>>>>> (02:51:02 PM) paul_m_ferraro: Well, this is the responsibility >>>>>> >> of >> >>>>>> whomever created the scoped UpHandler >>>>>> (02:51:57 PM) paul_m_ferraro: bela's impl throws a >>>>>> IllegalArgumentException - but it should be something more >>>>>> >>> specific >>> >>>>>> (02:52:01 PM) besYIM: yeah >>>>>> (02:52:31 PM) besYIM: but if it throws an exception I believe >>>>>> >>> that >>> >>>>>> >>>>>> >>>>> will >>>>> >>>>> >>>>>> propagate back to the MyRequestCorrelator who sent the RPC and >>>>>> >>> then >>> >>>>>> >>>>>> >>>>> get >>>>> >>>>> >>>>>> thrown >>>>>> (02:52:32 PM) paul_m_ferraro: and perhaps infinispan should >>>>>> >> allow >> >>> a >>> >>>>>> pluggable handler for this? >>>>>> (02:53:02 PM) paul_m_ferraro: so those cache users that want to >>>>>> >>>>>> >>>>> ignore >>>>> >>>>> >>>>>> it can, or retry it, etc. >>>>>> (02:53:13 PM) besYIM: I don't think Infinispan will have a >>>>>> >>> problem >>> >>>>>> (02:53:24 PM) paul_m_ferraro: retry the operation, if possible >>>>>> (02:53:49 PM) besYIM: Infinispan is the one who actually calls >>>>>> Channel.connect, so they are always registered first >>>>>> (02:54:24 PM) besYIM: it's the specialized handlers we'd add >>>>>> >> for >> >>>>>> >>>>>> >>>>> SFSB >>>>> >>>>> >>>>>> ownership, or the Hibernate eviction thing that would be >>>>>> >> missing >> >>>>>> (02:59:17 PM) besYIM: i was looking at the org.jgroups.Request >>>>>> >>>>>> >>>>> class >>>>> >>>>> >>>>>> (02:59:58 PM) besYIM: it's receiveResponse method is a >>>>>> >>> centralized >>> >>>>>> >>>>>> >>>>> place >>>>> >>>>> >>>>>> that could deal with something like a NoHandlerForRPC >>>>>> (03:00:10 PM) besYIM: i.e. discard it but mark the response >>>>>> >>>>>> >>>>> received >>>>> >>>>> >>>>>> (03:02:41 PM) besYIM: a simple impl of >>>>>> >>> org.jgroups.blocks.RspFilter >>> >>>>>> could also work -- add such a thing to JGroups and then AS, >>>>>> >> EJB3, >> >>>>>> Hibernate could use it for the few RPCs they make >>>>>> (03:06:45 PM) paul_m_ferraro: I see >>>>>> (03:06:57 PM) paul_m_ferraro: damn - its 4 >>>>>> (03:07:07 PM) paul_m_ferraro: today's my day to pick up marcus >>>>>> >>> from >>> >>>>>> preschool >>>>>> (03:07:19 PM) besYIM: that's always fun :). enjoy! >>>>>> >>>>>> >>>>> -- >>>>> Brian Stansberry >>>>> Lead, AS Clustering >>>>> JBoss by Red Hat >>>>> >>>>> >>> -- >>> Bela Ban >>> Lead JGroups / Clustering Team >>> JBoss >>> _______________________________________________ jboss-cluster-dev mailing list jboss-cluster-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-cluster-dev