[infinispan-dev] MyRpcDispatcher
Paul Ferraro
paul.ferraro at redhat.com
Fri Apr 9 15:59:51 EDT 2010
I've modified Bela's existing sample code to address the issues Brian
and I discussed below. Here's a summary of the changes:
* Refactored pieces into separate classes with nicer names
* ScopedRpcDispatcher registers its scope with a ScopedUpHandler
* Added ScopeUpHandler deregistration on ScopedRpcDispatcher.stop().
* ScopedUpHandler accepts an optional default handler, which will be
triggered for non-message events or messages with no scope header
* ScopedUpHandler returns NoHandlerForScope object if message is
received for an unknown scope (e.g. its handler was not yet registered
or was already deregistered)
* Automatically adds RspFilter to RequestOptions that rejects
NoHandlerForScope responses, decorating any existing filter, if
necessary.
* Added ScopedMessageDispatcher analogue
So, usage would look like:
Channel c = new JChannel(...);
ScopedUpHandler h = new ScopedUpHandlerImpl();
ScopedRpcDispatcher d1 = new ScopedRpcDispatcher((short)1, h, c, null, null, target1);
ScopedRpcDispatcher d2 = new ScopedRpcDispatcher((short)2, h, c, null, null, target2);
// This must be set after all rpc dispatchers are created
c.setUpHandler(h);
c.connect(...);
This will also work in conjunction with a standard rpc dispatcher:
Channel c = new JChannel(...);
RpcDispatcher d = new RpcDispatcher(c, null, null, target);
ScopedUpHandler h = new ScopedUpHandlerImpl(d.getProtocolAdapter());
ScopedRpcDispatcher d1 = new ScopedRpcDispatcher((short)1, h, c, null, null, target1);
ScopedRpcDispatcher d2 = new ScopedRpcDispatcher((short)2, h, c, null, null, target2);
c.setUpHandler(h);
c.connect(...);
I've attached the changes in the form of a patch to JGRP-1177.
Comments are welcome.
Paul
On Fri, 2010-04-09 at 06:55 -0400, Galder Zamarreno wrote:
> Adding Infinispan dev list.
>
> --
> Galder Zamarreño
> Sr. Software Engineer
> Infinispan, JBoss Cache
>
> ----- galder at redhat.com wrote:
>
> > As a FYI:
> >
> > I'm currently testing this and it's interesting to see that the order
> > of 1) and 2) as below matters:
> >
> > 1) disp1=new MyRpcDispatcher((short)100, c1, null, null, new
> > Server("foo1()"));
> > disp2=new MyRpcDispatcher((short)0, c1, null, null, new
> > Server("foo2()"));
> > disp3=new MyRpcDispatcher((short)150, c1, null, null, new
> > Server("foo3()"));
> >
> > 2) MyUpHandler handler=new MyUpHandler();
> > c1.setUpHandler(handler);
> >
> > The MyRpcDispatcher constructor relies on MessageDispatcher ctror that
> > does:
> >
> > prot_adapter=new ProtocolAdapter();
> > ...
> > channel.setUpHandler(prot_adapter);
> >
> > And after all those ctros have executed, we override the handler in
> > 2). If you execute 2) then 1), all messages end up in the last
> > uphandler set, which means that all calls go to foo3(). The reason I'm
> > looking into this is because from a consumer perspective, I'm trying
> > to understand what Hot Rod server would need to give to Infinispan or
> > whoever managers the channel.
> >
> > Note that after 2) comes:
> >
> > 3) handler.put((short)100, disp1.getProtocolAdapter());
> > handler.put((short)0, disp2.getProtocolAdapter());
> > handler.put((short)150, disp3.getProtocolAdapter());
> >
> > Bearing all this mind, I see Hot Rod server providing its own
> > Rpcdispatcher in 1) and then 3). 3) might not need to rely on Hot Rod
> > server entire if out MyRpcDispatcher, whoever executes 3) can figure
> > out the scope number. In other words 3) could be make to look like
> > this assuming that disp1, disp2 and disp3 are defined as
> > MyRpcDispatcher rather than RpcDispatcher
> >
> > handler.put(disp1.scope, disp1.getProtocolAdapter());
> > handler.put(disp2.scope, disp2.getProtocolAdapter());
> > handler.put(disp3.scope, disp3.getProtocolAdapter());
> >
> > My investigation continues...
> > --
> > Galder Zamarreño
> > Sr. Software Engineer
> > Infinispan, JBoss Cache
> >
> > ----- "Bela Ban" <bban at redhat.com> wrote:
> >
> > > OK, why don't you take a look ? JGRP-1177 has sample code attached
> > > showing how to use this.
> > >
> > > I'll respond in more detail tomorrow
> > >
> > > galder at redhat.com wrote:
> > > > Indeed, I think I need this as well for the Hot Rod use case. Hot
> > > Rod servers need to know which other Hot Rod servers are running in
> > > the cluster so that they can forward the corresponding information
> > > back to clients for load balancing, failover...etc.
> > > >
> > > > I don't want to polute the core cache channel code having to deal
> > > with Hot Rod rpcs.
> > > >
> > > > Also, a separate channel over a shared transport does not make
> > sense
> > > since the Hot Rod is tightly linked to the core cache channel that
> > > provides data replication capabilities amongst other things. IOW,
> > if
> > > Hot Rod used a separate channel for this, you could have scenarios
> > > where a Hot Rod servers thinks that HotRodServer1, HotRodServer2,
> > and
> > > HotRodServer3 are the hot rod servers available in the cluster, but
> > > underneath the core cache channel would only be formed of
> > CoreCache1
> > > and CoreCache2, with CoreCache3 isolated. This would be bad since a
> > > client could send requests to HotRodServer3 expecting it to
> > replicate
> > > to 1 or 2 but that would not happen. If the same view was used,
> > > clients would realise that HotRodServer3 is not part of the cluster
> > > any more.
> > > >
> > > > I'll check the current proposed solution for
> > > https://jira.jboss.org/jira/browse/JGRP-1177 tomorrow and comment
> > > further.
> > > >
> > > > Bela, when do u expect to have something that I can test?
> > > >
> > > > Cheers,
> > > >
> > > > p.s. I'm adding cluster-dev list to discussion since this is a
> > true
> > > inter cluster project concern.
> > > > --
> > > > Galder Zamarreño
> > > > Sr. Software Engineer
> > > > Infinispan, JBoss Cache
> > > >
> > > > ----- "Brian Stansberry" <brian.stansberry at redhat.com> wrote:
> > > >
> > > >
> > > >> Paul,
> > > >>
> > > >> Was just chatting with Galder and it seems Infinispan internally
> > > could
> > > >>
> > > >> also have a use for this. (Their HotRodServers would want to
> > make
> > > RPCs
> > > >>
> > > >> but he doesn't want to corrupt the core infinispan usage with
> > > that.)
> > > >>
> > > >> Can you pull together your thoughts on this, respond to Bela,
> > bring
> > > in
> > > >>
> > > >> Galder and get this pushed through? Galder's also the most
> > > >> knowledgable
> > > >> guy on the Hibernate/ISPN integration, which is the other big
> > use
> > > case
> > > >>
> > > >> for this.
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Brian
> > > >>
> > > >> On 04/06/2010 03:48 PM, Brian Stansberry wrote:
> > > >>
> > > >>> Below is the transcript from our chat.
> > > >>>
> > > >>> Only other thoughts I had on this were:
> > > >>>
> > > >>> 1) If MyUpHandler exposed a setter for defaultHandler instead
> > of
> > > >>>
> > > >> using
> > > >>
> > > >>> scope 0 that would be good.
> > > >>>
> > > >>> 2) If JGroups shipped an UpHandler impl that sent a Message
> > with
> > > a
> > > >>> marker object like NoHandlerForRpc, whoever's responsible for
> > > >>>
> > > >> setting
> > > >>
> > > >>> MyUpHandler on the channel could also set that default handler.
> > > As
> > > >>>
> > > >> a
> > > >>
> > > >>> convenience an overloaded constructor on MyUpHandler could do
> > > this
> > > >>>
> > > >> for
> > > >>
> > > >>> you based on a boolean param.
> > > >>>
> > > >>> 3) If JGroups shipped a RspFilter impl that returned false to
> > > >>> isAcceptable(Object response, Address sender) if response
> > > >>>
> > > >> instanceof
> > > >>
> > > >>> NoHandlerForRpc, then code that was concerned about getting
> > > >>> NoHandlerForRpc could pass that to RpcDispatcher call remote
> > > >>>
> > > >> methods.
> > > >>
> > > >>> 4) The impl of 3) above should also accept an optional delegate
> > > >>>
> > > >> whose
> > > >>
> > > >>> methods would be invoked if the response isn't an instance of
> > > >>> NoHandlerForRpc. That could be used in case whatever was making
> > > the
> > > >>>
> > > >> RPC
> > > >>
> > > >>> also wanted to filter responses some other way.
> > > >>>
> > > >>> The above plus what we discussed below I think pretty nicely
> > > >>> encapsulates the guts of this inside JGroups code. The users of
> > > >>>
> > > >> this
> > > >>
> > > >>> stuff would then have to:
> > > >>>
> > > >>> A) instantiate the channel (if not already done)
> > > >>>
> > > >>> B) instantiate and set the MyUpHandler and set up the default
> > > >>>
> > > >> handler
> > > >>
> > > >>> (if not already done)
> > > >>>
> > > >>> C) Register their RPC dispatcher
> > > >>>
> > > >>> D) Pass in the channel to Infinispan, which can be clueless
> > about
> > > >>>
> > > >> all of
> > > >>
> > > >>> this if they so choose.
> > > >>>
> > > >>> E) When making any RPCs, pass in the RspFilter impl described
> > > >>>
> > > >> above.
> > > >>
> > > >>> If the Channel already existed (e.g. a second service sharing a
> > > >>> CacheManager), then the user service would need to figure out a
> > > way
> > > >>>
> > > >> to
> > > >>
> > > >>> get ahold of it (does Infinispan expose it)? and then do C and
> > E
> > > >>>
> > > >> above.
> > > >>
> > > >>> I think the latter scenario should discouraged though.
> > Different
> > > >>> services sharing the same channel is always problematic. Bela
> > > >>>
> > > >> convinced
> > > >>
> > > >>> me that by default the different AS services should use
> > different
> > > >>> CacheManager's/Channels, so all of the above would just be
> > needed
> > > if
> > > >>>
> > > >> we
> > > >>
> > > >>> want to support CacheManager sharing between
> > > >>>
> > > >> HTTP/SFSB/Hib/HAPartition.
> > > >>
> > > >>> (02:07:31 PM) besYIM: this class extends ReceiverAdapter, but
> > > AFAICT
> > > >>>
> > > >> it
> > > >>
> > > >>> doesn't need to; it doesn't do anything w/ that API
> > > >>> (02:08:21 PM) paul_m_ferraro: I hadn't even noticed that
> > > >>> (02:09:11 PM) besYIM: good; so that means I probably didn't
> > miss
> > > >>> anything :)
> > > >>> (02:10:15 PM) besYIM: the part I don't see is where MyUpHandler
> > > >>> put/remove get called
> > > >>> (02:10:30 PM) besYIM: ah, in start
> > > >>> (02:10:37 PM) paul_m_ferraro: yep
> > > >>> (02:12:28 PM) besYIM: this would need to be used from 1) AS 2)
> > > >>>
> > > >> Hibernate
> > > >>
> > > >>> 2LC provider 3) Infinispan
> > > >>> (02:13:14 PM) paul_m_ferraro: why not put in jgroups directly?
> > > >>> (02:13:56 PM) besYIM: yeah, i expect so
> > > >>> (02:13:58 PM) paul_m_ferraro: in org.jgroups.blocks?
> > > >>> (02:14:02 PM) besYIM: yep
> > > >>> (02:14:17 PM) besYIM: i was thinking about what clients would
> > > need
> > > >>>
> > > >> to be
> > > >>
> > > >>> adapted to use it
> > > >>> (02:15:33 PM) besYIM: i.e. what ISPN would have to change to
> > deal
> > > >>>
> > > >> with
> > > >>
> > > >>> the fact this *may* be there, but typically isn't
> > > >>> (02:17:15 PM) paul_m_ferraro: oh - I see you point
> > > >>> (02:17:19 PM) paul_m_ferraro: your
> > > >>> (02:17:22 PM) besYIM: i'm trying to figure where that
> > > >>> MyRpcDispatcher.createRequestCorrelator() method gets invoked
> > > >>> (02:17:40 PM) besYIM: since that method smells like a nice hook
> > > to
> > > >>>
> > > >> hide
> > > >>
> > > >>> all this :)
> > > >>> (02:18:21 PM) paul_m_ferraro: infinispan uses their own
> > > >>>
> > > >> RpcDispatcher
> > > >>
> > > >>> extension, right?
> > > >>> (02:18:33 PM) besYIM: yeah, i believe so
> > > >>> (02:19:07 PM) paul_m_ferraro: so the overridden UpHandler would
> > > >>>
> > > >> just
> > > >>
> > > >>> fall through to the default behavior if no scope header exists
> > > >>> (02:20:23 PM) paul_m_ferraro: the question is, how to override
> > > >>> infinispan so that it too can be multiplexed by scope
> > > >>> (02:20:31 PM) paul_m_ferraro: no?
> > > >>> (02:20:43 PM) besYIM: yep
> > > >>> (02:21:33 PM) besYIM: the createRequestCorrelatorMethod seems
> > to
> > > be
> > > >>>
> > > >> the way
> > > >>
> > > >>> (02:21:42 PM) besYIM: well, maybe
> > > >>> (02:22:20 PM) besYIM: if it had knowledge as whether the
> > > UpHandler
> > > >>>
> > > >> was
> > > >>
> > > >>> MyUpHandler it could change the type of RequestCorrelator it
> > > >>>
> > > >> creates
> > > >>
> > > >>> (02:22:37 PM) besYIM: std UpHandler, just create the std one
> > > >>> (02:22:47 PM) besYIM: special one, create MyRequestCorrelator
> > > >>> (02:25:49 PM) paul_m_ferraro: so where would Infinispan get
> > it's
> > > >>>
> > > >> scope
> > > >>
> > > >>> from?
> > > >>> (02:26:14 PM) besYIM: "well known scopes" wiki page?
> > > >>> (02:26:45 PM) besYIM: i suppose adding it to the ISPN config
> > > would
> > > >>>
> > > >> be
> > > >>
> > > >>> better
> > > >>> (02:28:28 PM) besYIM: ah, i found the call to
> > > >>>
> > > >> createRequestCorrelator --
> > > >>
> > > >>> in MessageDispatcher.start()
> > > >>> (02:31:16 PM) besYIM: none of the params passed to that method
> > > >>>
> > > >> provide
> > > >>
> > > >>> any info on the type of the UpHandler, but
> > > MessageDispatcher.channel
> > > >>>
> > > >> is
> > > >>
> > > >>> protected, so the method impl could just look at the Channel
> > > >>> (02:32:52 PM) paul_m_ferraro: I should think the
> > MyRpcDispatcher
> > > >>>
> > > >> should
> > > >>
> > > >>> also register itself w/the MyUpHandler, if its channel used it
> > > >>> (02:33:02 PM) besYIM: yep
> > > >>> (02:33:44 PM) besYIM: this could all be part of the std
> > > >>>
> > > >> RpcDispatcher
> > > >>
> > > >>> (02:34:43 PM) paul_m_ferraro: yep - and can use a default
> > scope,
> > > if
> > > >>>
> > > >> none
> > > >>
> > > >>> specified
> > > >>> (02:35:40 PM) besYIM: yeah, so infinispan could deal with
> > > >>>
> > > >> configuring
> > > >>
> > > >>> scope or not
> > > >>> (02:40:38 PM) besYIM: the other concern i had was lifecycle
> > > issues
> > > >>>
> > > >> --
> > > >>
> > > >>> i.e. transient issues with a channel not disconnected but a
> > > handler
> > > >>> removed e.g. during service undeployment
> > > >>> (02:41:34 PM) besYIM: or deployment
> > > >>> (02:42:52 PM) paul_m_ferraro: so, for example, a message
> > arrives
> > > >>>
> > > >> w/scope
> > > >>
> > > >>> X but scope X is no longer registered w/the UpHandler?
> > > >>> (02:43:16 PM) besYIM: or hasn't yet been registered
> > > >>> (02:43:52 PM) besYIM: like lets assume to make it hard that
> > HTTP
> > > >>> session, SFSB, Hib 2LC will all use the same CacheManager and
> > > thus
> > > >>>
> > > >> same
> > > >>
> > > >>> Channel
> > > >>> (02:44:44 PM) besYIM: nodes A and B are fully started; node C
> > is
> > > >>>
> > > >> coming
> > > >>
> > > >>> on line. It deploys a distributable webapp, so cache is
> > started,
> > > >>>
> > > >> Channel
> > > >>
> > > >>> connected, etc
> > > >>> (02:45:10 PM) besYIM: but there is no handler for the SFSB and
> > > Hib
> > > >>>
> > > >> 2LC
> > > >>
> > > >>> stuff yet because those aren't deployed yet
> > > >>> (02:46:11 PM) besYIM: HAPartition deals with this via
> > > >>>
> > > >> NoHandlerForRPC
> > > >>
> > > >>> (02:51:02 PM) paul_m_ferraro: Well, this is the responsibility
> > of
> > > >>> whomever created the scoped UpHandler
> > > >>> (02:51:57 PM) paul_m_ferraro: bela's impl throws a
> > > >>> IllegalArgumentException - but it should be something more
> > > specific
> > > >>> (02:52:01 PM) besYIM: yeah
> > > >>> (02:52:31 PM) besYIM: but if it throws an exception I believe
> > > that
> > > >>>
> > > >> will
> > > >>
> > > >>> propagate back to the MyRequestCorrelator who sent the RPC and
> > > then
> > > >>>
> > > >> get
> > > >>
> > > >>> thrown
> > > >>> (02:52:32 PM) paul_m_ferraro: and perhaps infinispan should
> > allow
> > > a
> > > >>> pluggable handler for this?
> > > >>> (02:53:02 PM) paul_m_ferraro: so those cache users that want to
> > > >>>
> > > >> ignore
> > > >>
> > > >>> it can, or retry it, etc.
> > > >>> (02:53:13 PM) besYIM: I don't think Infinispan will have a
> > > problem
> > > >>> (02:53:24 PM) paul_m_ferraro: retry the operation, if possible
> > > >>> (02:53:49 PM) besYIM: Infinispan is the one who actually calls
> > > >>> Channel.connect, so they are always registered first
> > > >>> (02:54:24 PM) besYIM: it's the specialized handlers we'd add
> > for
> > > >>>
> > > >> SFSB
> > > >>
> > > >>> ownership, or the Hibernate eviction thing that would be
> > missing
> > > >>> (02:59:17 PM) besYIM: i was looking at the org.jgroups.Request
> > > >>>
> > > >> class
> > > >>
> > > >>> (02:59:58 PM) besYIM: it's receiveResponse method is a
> > > centralized
> > > >>>
> > > >> place
> > > >>
> > > >>> that could deal with something like a NoHandlerForRPC
> > > >>> (03:00:10 PM) besYIM: i.e. discard it but mark the response
> > > >>>
> > > >> received
> > > >>
> > > >>> (03:02:41 PM) besYIM: a simple impl of
> > > org.jgroups.blocks.RspFilter
> > > >>> could also work -- add such a thing to JGroups and then AS,
> > EJB3,
> > > >>> Hibernate could use it for the few RPCs they make
> > > >>> (03:06:45 PM) paul_m_ferraro: I see
> > > >>> (03:06:57 PM) paul_m_ferraro: damn - its 4
> > > >>> (03:07:07 PM) paul_m_ferraro: today's my day to pick up marcus
> > > from
> > > >>> preschool
> > > >>> (03:07:19 PM) besYIM: that's always fun :). enjoy!
> > > >>>
> > > >> --
> > > >> Brian Stansberry
> > > >> Lead, AS Clustering
> > > >> JBoss by Red Hat
> > > >>
> > >
> > > --
> > > Bela Ban
> > > Lead JGroups / Clustering Team
> > > JBoss
More information about the infinispan-dev
mailing list