Re: [jgroups-dev] Shared transport in AS
by Brian Stansberry
Bela Ban wrote:
>
>
> Brian Stansberry wrote:
>> Been playing with using the 2.6 branch version of the shared transport
>> in JBoss AS. Unfortunately, the dead-simple integration approach I was
>> thinking of is not valid. (See
>> http://jira.jboss.com/jira/browse/JBAS-5167). I've got other ideas I'm
>> playing with; I'm sure I can come up with something; won't get into
>> details of the problem here.
>
> Is this a JBoss or JGroups issue ?
>
Really more of a JBC issue. The way JBC works, you either give it a full
protocol stack config, or inject a JChannelFactory and a stack name. In
the AS we're not interested in the former. If you configure for the
latter, JBC internally gets its channel by calling
factory.createMultiplexerChannel(...). That's no good because we don't
want a multiplexer channel; we want a shared transport channel called
via factory.createChannel(stack_name).
The AS has the same problem, but I control the code so I can work around
it more easily. JBC has its own release cycle, dependency mgmt, etc.
The JBAS-5167 approach is no good -- it tries to use the MC to create
the channel (using factory.createChannel(stack_name)) and then inject it
into the target service. No good because that injection only happens
once. If you stop() and re- start() the service, the injected channel
gets closed; when the re- start() tries to call connect() on it again
that fails.
The AS already uses a custom JChannelFactory subclass. It already
overrides createMultiplexerChannel(). As a temporary workaround, I've
added logic so it checks the requested protocol stack's config to see if
singleton_name is set; if it is it delegates to
createChannel(stack_name), if not it calls
super.createMultiplexerChannel(). Basically changed the semantic of the
method to create a shared transport channel if possible, and a
multiplexer channel if not possible.
That seems to work for now, well enough to allow continued experimentation.
>> But, aside from a few failures related to
>> the flaw in JBAS-5167, it looks like the AS testsuite ran well w/ shared
>> transport channels used instead of multiplexed ones. :)
>>
>> There are a few things I saw that IMHO need improvement before we can
>> switch the AS to this:
>>
>> 1) TP.members -- used to send to all members via unicast by TCP (or by
>> UDP if multicast is disabled). AIUI this should probably be a
>> Map<String, List<Address>> so separate memberships can be maintained.
>
> Why ? If we have a shared channel C1 and apps A1 (cluster name="A1") A2
> (name="A2") and A3 (name="A3") on top of it, then all 3 apps have the
> *same* JGroups address, e.g. 192.168.2.5:5000. If we have another shared
> channel C2 in a separate JVM, and apps A1, A2, A3 on top, with address
> 192.168.2.5:6000, then C1.A1's view is
> {192.168.2.5:5000, 192.168.2.5:6000}, same for all apps.
>
> So when C1.A1 multicasts a message, then it will conceptually send it to
> 192.168.2.5:5000 ("A1") and 192.168.2.5:6000 ("A1"). So its view has 2
> members. A2 and A3 will not receive the multicast, as the cluster name
> ("A1") is used to demultiplex the message to the right protocol above
> the shared channel.
>
True, but I'm talking about multiple unicast, not multicast. And assume
on node 2 that app A1 is not deployed. So now you're expending resources
sending messages to peers who will just drop them. Can't be helped and
is not very costly with multicast, but tracking the members by group
name allows you to avoid it for multiple unicast.
>> 2) MPING. We'd discussed having MPING drop GET_MBRS_RSP messages that
>> are not associated with the channel's group name. This allows a single
>> MPING config (i.e. mcast_addr, mcast_port) in a stack to be instantiated
>> multiple times, for different channels. AFAICT, the way it is now, 2
>> channels created from the same stack will see each others' ping requests
>> and respond, with the irrelevant responses from the wrong channel being
>> treated as valid.
>
> No, the invalid ping request should get discarded, this is analogous to
> a *non-shared stack*:
> - draw -props ./mping.xml -groupname X
> - draw -props ./mping.xml -groupname Y
>
> Both instances use the *same* MPING config, but will discard their
> discovery requests as they belong to different clusters.
>
> So, in your example, you must have different stacks. Or use the same
> stack, but change MPING's mcast_addr and mcast_port.
>
> I don't want to change this as it is *exactly* the same behavior as in
> the non-shared stack !
>
OK, it sounds like you are saying this already works the way I want. I
admit I didn't test it. :( I just looked at the MPING code and didn't
see anything where it would discard a message from the wrong group (e.g.
in MPING.run(), MPING.up() or Discovery.up()).
>> That's a pretty small list. I really like this stuff; excellent work,
>> guys!!
>
> Thanks !
>
>> Something else that seems *much* less a show-stopper but is still a
>> concern:
>>
>> The shared transports are stored in a VM-singleton map --
>> ProtocolStack.shared_transports. Can these instead be stored as an
>> instance field in JChannelFactory?
>
> But then you wouldn't be able to share transports between JChannels, and
> this would force you to use JChannelFactory... I'd like folks to be able
> to simply create a new JChannel(). If I do this, the variable has to be
> on the ProtocolStack, can't be on JChannelFactory.
>
>> This makes the scope of sharing
>> controllable. Perhaps this could be a flag on the JChannelFactory; use
>> the VM-singleton map by default but use a factory-scoped map if a flag
>> is set.
>
> Can you send me a short program that shows what you want to do ? I
> assume you don't use JChannelFactory.createMultiplexerChannel() right ?
> Do you use JCF.createChannel() ?
>
I'll play with this a bit; if it looks like a serious problem I'll send
you something.
>> This bit me in one unit test where I deliberately create 2
>> JChannelFactory instances to simulate two cluster nodes. Each factory
>> creates two channels (were multiplexed, now shared TP). Then I do some
>> manipulations. This test fails because when I create the channels from
>> the 2nd factory, Configurator.startProtocolStack barfs:
>>
>> Caused by: java.lang.IllegalStateException: cluster 'tunnel'
>> is already connected to singleton transport: [tunnel,
>> dummy-1201128933100]
>> at
>> org.jgroups.stack.Configurator.startProtocolStack(Configurator.java:88)
>> at org.jgroups.stack.ProtocolStack.startStack(ProtocolStack.java:316)
>> at org.jgroups.JChannel.startStack(JChannel.java:1442)
>> ... 28 more
>>
>> With the channels stored in a static map, there's no clean way to
>> simulate two separate environments. I imagine this kind of scenario
>> would be common in test cases; I believe JBC has some similar kinds of
>> tests.
>
> I'll take a look
>
--
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry(a)redhat.com
16 years, 3 months
Shared transport in AS
by Brian Stansberry
Been playing with using the 2.6 branch version of the shared transport
in JBoss AS. Unfortunately, the dead-simple integration approach I was
thinking of is not valid. (See
http://jira.jboss.com/jira/browse/JBAS-5167). I've got other ideas I'm
playing with; I'm sure I can come up with something; won't get into
details of the problem here. But, aside from a few failures related to
the flaw in JBAS-5167, it looks like the AS testsuite ran well w/ shared
transport channels used instead of multiplexed ones. :)
There are a few things I saw that IMHO need improvement before we can
switch the AS to this:
1) TP.members -- used to send to all members via unicast by TCP (or by
UDP if multicast is disabled). AIUI this should probably be a
Map<String, List<Address>> so separate memberships can be maintained.
2) MPING. We'd discussed having MPING drop GET_MBRS_RSP messages that
are not associated with the channel's group name. This allows a single
MPING config (i.e. mcast_addr, mcast_port) in a stack to be instantiated
multiple times, for different channels. AFAICT, the way it is now, 2
channels created from the same stack will see each others' ping requests
and respond, with the irrelevant responses from the wrong channel being
treated as valid.
That's a pretty small list. I really like this stuff; excellent work, guys!!
Something else that seems *much* less a show-stopper but is still a concern:
The shared transports are stored in a VM-singleton map --
ProtocolStack.shared_transports. Can these instead be stored as an
instance field in JChannelFactory? This makes the scope of sharing
controllable. Perhaps this could be a flag on the JChannelFactory; use
the VM-singleton map by default but use a factory-scoped map if a flag
is set.
This bit me in one unit test where I deliberately create 2
JChannelFactory instances to simulate two cluster nodes. Each factory
creates two channels (were multiplexed, now shared TP). Then I do some
manipulations. This test fails because when I create the channels from
the 2nd factory, Configurator.startProtocolStack barfs:
Caused by: java.lang.IllegalStateException: cluster 'tunnel'
is already connected to singleton transport: [tunnel, dummy-1201128933100]
at org.jgroups.stack.Configurator.startProtocolStack(Configurator.java:88)
at org.jgroups.stack.ProtocolStack.startStack(ProtocolStack.java:316)
at org.jgroups.JChannel.startStack(JChannel.java:1442)
... 28 more
With the channels stored in a static map, there's no clean way to
simulate two separate environments. I imagine this kind of scenario
would be common in test cases; I believe JBC has some similar kinds of
tests.
--
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry(a)redhat.com
16 years, 3 months
Incompatible API Change in 2.1.0.CR3
by Brian Stansberry
There's been a change in the public API in 2.1.0.CR3: Option.clone() no
longer throws CloneNotSupportedException.
This may break apps upgrading from 2.0.0; see example below.
-------- Original Message --------
Subject: Re: ejb3-head-testsuite spike
Date: Fri, 18 Jan 2008 12:58:49 +0000
From: Kabir Khan <kabir.khan(a)jboss.com>
To: Dimitris Andreadis <dandread(a)redhat.com>
CC: Carlo de Wolf <carlo.dewolf(a)jboss.com>, Brian Stansberry
<brian.stansberry(a)redhat.com>, JBoss AS <jboss-as(a)redhat.com>,
Flavia Rainone <frainone(a)redhat.com>
References: <4790849E.5080801(a)redhat.com>
<snip/>
Also, yesterday following the jboss cache upgrade, in order to be able
to build I had to make the following changes to StatefulTreeCache.java:
@@ -679,31 +679,31 @@
}
}
}
-
+
private static Option getLocalOnlyOption()
{
- try
- {
+ //try
+ //{
return LOCAL_ONLY_OPTION.clone();
- }
- catch (CloneNotSupportedException e)
- {
- throw new RuntimeException(e);
- }
+ //}
+ //catch (CloneNotSupportedException e)
+ //{
+ // throw new RuntimeException(e);
+ //}
}
-
+
private static Option getGravitateOption()
{
- try
- {
+ //try
+ //{
return GRAVITATE_OPTION.clone();
- }
- catch (CloneNotSupportedException e)
- {
- throw new RuntimeException(e);
- }
+ //}
+ //catch (CloneNotSupportedException e)
+ //{
+ // throw new RuntimeException(e);
+ //}
}
-
+
--
Kabir Khan
JBoss AOP Lead
JBoss, a division of Red Hat
--
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry(a)redhat.com
16 years, 3 months
Injection Code Incompatible with AS and PCache Maven Run
by Jason T. Greene
The injection design makes assumptions about the ClassLoader it is
running in, which we can't do.
1. ClasspathScanner's constructor assumes that all ClassLoaders are
URLClassLoader, this is not always true, and leads to CCE when it is not.
2. ClasspathScanner.getURLPathFromClassLoader expects only simple file
based URLs (like those returned from URLClassLoader). This will not work
with the AS5 VFS, which uses a custom URL type ("vfs").
3. ClasspathScanner.scan assumes that all URLS are on a local
filesystem. This is also not the case. An AS can have http based
deployments for example.
These problems not only prevent compatibility with AS5, they also cause
problems with my maven testruns, since I have to have
useSystemClassLoader set to true, which uses a manifest for constructing
the classpath.
We should remove this scanning code in favor of either hardcoded
constants, or a hardcoded registration process.
This is urgent because it prevents the CR3 release from going out.
--
Jason T. Greene
JBoss, a division of Red Hat
16 years, 3 months
ComponentRegistry and TransactionManager
by Brian Stansberry
Interesting change from the ComponentRegistry has bitten me in the ass.
The resolution of the TransactionManager used to take place in
CacheImpl.create(), now it's happening via
DefaultCacheFactory.bootstrap() -- i.e. as part of cache construction.
This causes problems with the Hibernate integration, which injects the
TM it knows about into the RuntimeConfig of the already-existing cache,
before it calls create(). But that's too late -- the
TransactionManagerFactory has already logged an error about a missing TM.
A solution would be for Hibernate to get a Configuration first, inject
the TM, and then build the cache. But the CacheManager API doesn't
expose a good way to do that. An external caller can't modify the
registered configurations.
So:
1) What do you think about changing the ConfigurationRegistry API to
allow some sort of updating of registered configurations?
2) In general, do you think it's a problem that we are now resolving
things earlier in the process than we used to? Other users may also be
doing things programatically after cache construction but before
create/start.
--
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry(a)redhat.com
16 years, 3 months