[infinispan-dev] Clustering standalone Infinispan w/ WF running Infinispan

Thu Oct 2 09:21:07 EDT 2014

I have successfully created a "hybrid" cluster between an application 
using Infinispan in embedded mode and an Infinispan server by doing the 
following on the embedded side:

- use a JGroups Channel wrapped in a MuxHandler
- use a custom class resolver which simulates (or rather... hacks) the 
behaviour of the ModularClassResolver when not using modules

You can find the code at my personal GitHub repo:

https://github.com/tristantarrant/infinispan-playground/tree/master/src/main/java/net/dataforte/infinispan/hybrid

suggestions and improvements are welcome.

Tristan

On 30/09/14 10:01, Stelios Koussouris wrote:
> Hi,
>
> To give a bit of context on this. We are doing a POC where the customer wishes to utilize JDG to speed up their application. We need (due to some customer requirements) to cluster
> EMBEDDED JDG (infinispan library mode) with REMOTE JDG (Infinispan Server) nodes. The infinispan jars should be the same as they are only libraries and they
> are on the same version. However, during "clustering" of the caches we started seeing errors which looked like there were due to the fact that the clustering of the caches contained different
> info between the 2 types of cache instantiation (embedded vs server).
>
> The result was to for a suggestion to create our own MuxChannel (I don't know if we have any other alternatives at this stage to cluster embedded with server infinispan caches) but at the moment we are facing https://gist.github.com/skoussou/5edc5689446b67f85ae8
>
> Regards,
>
> Stylianos Kousouris
> Red Hat Middleware Consultant
>
> ----- Original Message -----
> From: "Tristan Tarrant" <ttarrant at redhat.com>
> To: "infinispan -Dev List" <infinispan-dev at lists.jboss.org>, "Kurt T Stam" <kurt.stam at jboss.com>
> Cc: "Stelios Koussouris" <stkousso at redhat.com>, "Richard Achmatowicz" <rachmato at redhat.com>
> Sent: Tuesday, 30 September, 2014 8:02:27 AM
> Subject: Re: [infinispan-dev] Clustering standalone Infinispan w/ WF running Infinispan
>
> I don't know what Kurt is doing, but Stelios is attempting to cluster an
> application using embedded Infinispan deployed within WF together with
> an Infinispan Server instance.
> The application is managing its own caches, and therefore it is not
> interacting with the underlying Infinispan and JGroups subsystems in WF.
> Infinispan Server uses its Infinispan and JGroups subsystems (which are
> forked from WF's) and therefore are using MuxChannels.
>
> I told Stelios to use a MuxChannel-wrapped Channel in his application
> and it solved part of the issue (he was initially importing the one
> included in the WF's jgroups subsystem, but now he's using his local
> copy), but now he has run into further problems and I believe what Paul
> & Dennis have written might be correct.
>
> The code that configures this is in
> EmbeddedCacheManagerConfigurationService:
>
> GlobalConfigurationBuilder builder = new GlobalConfigurationBuilder();
> ModuleLoader moduleLoader = this.dependencies.getModuleLoader();
> builder.serialization().classResolver(ModularClassResolver.getInstance(moduleLoader));
>
> I don't know how you'd get a ModuleLoader from within a WF deployment,
> but I'm sure it can be done.
>
> Tristan
>
> On 29/09/14 18:57, Paul Ferraro wrote:
>> You should not need to use a MuxChannel.  This would only be necessary if there are other EAP services sharing the channel.  Using a MuxChannel allows your standalone Infinispan instance to filter these irrelevant messages.  However, in JDG, there should be no other services other than Infinispan using the channel - hence the MuxChannel stuff is unnecessary.
>>
>> I think Dennis earlier response was spot on.  EAP/JDG configures it's cache managers using a ModularClassResolver (which includes a module name along with the class name when marshalling).  Your standalone Infinispan instances do not use this and therefore cannot make sense of the message body.
>>
>> Paul
>>
>> ----- Original Message -----
>>> From: "Kurt T Stam" <kurt.stam at jboss.com>
>>> To: "Stelios Koussouris" <stkousso at redhat.com>, "Radoslav Husar" <rhusar at redhat.com>
>>> Cc: "Galder Zamarreño" <galder at redhat.com>, "Paul Ferraro" <paul.ferraro at redhat.com>, "Richard Achmatowicz"
>>> <rachmato at redhat.com>, "infinispan -Dev List" <infinispan-dev at lists.jboss.org>
>>> Sent: Monday, September 29, 2014 11:39:59 AM
>>> Subject: Re: Clustering standalone Infinispan w/ WF running Infinispan
>>>
>>> Thanks for following up Stelios, I think Galder is traveling the next 2
>>> weeks.
>>>
>>> So - do we need fixes on both ends then so that the boot order does not
>>> matter? In which project(s) would we apply
>>> there changes? Or can they be applied in the end-user's code?
>>>
>>> Thx,
>>>
>>> --Kurt
>>>
>>>
>>>
>>> On 9/26/14, 11:19 AM, Stelios Koussouris wrote:
>>>> Hi,
>>>>
>>>> Rado: It is both ways. ie. if I start first the JDG Server I get the issue
>>>> on the library mode side when I start that one. If reverse the order of
>>>> startup I get it in the JDG Server side.
>>>>
>>>> Question:
>>>> -----------------------------------------------------------------------------------------------------------------------
>>>> ...IMO the channel needs to be wrapped as
>>>> org.jboss.as.clustering.jgroups.MuxChannel before passing to infinispan.
>>>> ...
>>>> -----------------------------------------------------------------------------------------------------------------------
>>>> For now that this is not being done. If I wanted to do it manually on the
>>>> library side where I can create the protocol programmatically we are
>>>> talking about something like this?
>>>>
>>>> ProtocolStackConfigurator configurator =
>>>> ConfiguratorFactory.getStackConfigurator("jgroups-udp.xml");
>>>> MuxChannel channel = new MuxChannel(configurator);
>>>> org.infinispan.remoting.transport.Transport transport = new
>>>> org.infinispan.remoting.transport.jgroups.JGroupsTransport(channel);
>>>>
>>>> ....
>>>> then replace the below
>>>> new
>>>> GlobalConfigurationBuilder().clusteredDefault().globalJmxStatistics().cacheManagerName("RDSCacheManager").allowDuplicateDomains(true).enable().transport().clusterName("UDM-CLUSTER").addProperty("configurationFile",
>>>> "jgroups-udp.xml")
>>>>     
>>>> WITH
>>>> new
>>>> GlobalConfigurationBuilder().clusteredDefault().globalJmxStatistics().cacheManagerName("RDSCacheManager").allowDuplicateDomains(true).enable().transport(Transport).clusterName("UDM-CLUSTER")
>>>>
>>>> Btw, someone mentioned that if I follow this method I need to to know the
>>>> assigned mux ids, but that is not quite clear what it means with regards
>>>> to the JGroupsTransport configuration
>>>>
>>>> Thanks,
>>>>
>>>> Stylianos Kousouris
>>>> Red Hat Middleware Consultant
>>>>
>>>> ----- Original Message -----
>>>> From: "Radoslav Husar" <rhusar at redhat.com>
>>>> To: "Galder Zamarreño" <galder at redhat.com>, "Paul Ferraro"
>>>> <paul.ferraro at redhat.com>
>>>> Cc: "Richard Achmatowicz" <rachmato at redhat.com>, "infinispan -Dev List"
>>>> <infinispan-dev at lists.jboss.org>, "Stelios Koussouris"
>>>> <stkousso at redhat.com>, "Kurt T Stam" <kurt.stam at jboss.com>
>>>> Sent: Friday, 26 September, 2014 3:47:16 PM
>>>> Subject: Re: Clustering standalone Infinispan w/ WF running Infinispan
>>>>
>>>>     From what Stelios is telling me the question is a little bit other way
>>>> round: he is using library mode infinispan and jgroups in EAP and
>>>> connecting to JDG. So the question is what JDG is doing with the stack,
>>>> not AS/WF as its infinispan/jgroups subsystem is not used.
>>>>
>>>> Unfortunately I don't have access to the JDG repo so I don't know what
>>>> changes have been made there but if you are using the same jgroups
>>>> logic, IMO the channel needs to be wrapped as
>>>> org.jboss.as.clustering.jgroups.MuxChannel before passing to infinispan.
>>>>
>>>> Rado
>>>>
>>>> On 26/09/14 15:03, Galder Zamarreño wrote:
>>>>> Hey Paul,
>>>>>
>>>>> In the last couple of days, a couple of people have encountered the
>>>>> exception in [1] when trying to cluster a standalone Infinispan app with
>>>>> its own JGroups configuration file with a AS/WF running Infinispan cache.
>>>>>
>>>>>     From my POV, 3 possible causes:
>>>>>
>>>>> 1. Dependency mismatches between AS/WF and the standalone app. Having done
>>>>> some quick study of Kurt’s case, apart from micro version changes, all
>>>>> looks good.
>>>>>
>>>>> 2. Mismatch in the Infinispan and/or JGroups configuration file.
>>>>>
>>>>> 3. AS/WF puts something on the clustered wire that standalone Infinispan
>>>>> does not expect. Are you still doing multiplexing? Could you be adding
>>>>> extra info to the wire?
>>>>>
>>>>> With this email, I’m trying to get some clarification from you if the
>>>>> issue could be due to 3rd option. If it’s either of the first two, it’s a
>>>>> matter of digging and finding the difference, but if it’s 3rd one, it’s
>>>>> more problematic.
>>>>>
>>>>> Any ideas?
>>>>>
>>>>> [1] https://gist.github.com/skoussou/92f062f2d0bd17168e01
>>>>> --
>>>>> Galder Zamarreño
>>>>> galder at redhat.com
>>>>> twitter.com/galderz
>>>>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>