[
https://issues.jboss.org/browse/WFLY-3711?page=com.atlassian.jira.plugin....
]
Richard Achmatowicz commented on WFLY-3711:
-------------------------------------------
Some background on how these topology updates from cluster members are generated:
{noformat}
- use of RegistryCollector and Registry is to keep track of which clusters a node belongs
to and membership changes of the clusters
- RegistryCollector
- the RegistryCollector holds a set of Registry objects and represents the *set* of
clusters that a node belongs to
- implemented by a service on the node, which holds the collection
- the RegistryCollector has listeners:
registryAdded
registryRemoved
- Registry
- each Registry represents a cluster that the node is a member of and holds a
Map<String,List<ClientMapping>>
- each Map.Entry represents a node in that cluster and its local set of ClientMapping
elements (outgoing connections)
- implemented by a distributed cache which holds a Map of node names to
List<ClientMapping>
- CacheRegistry (the implementation) has three listeners:
@ToplologyChanged -> only processed on coordinator; calculate the set of old
addresses; perform an invocation to remove them from all nodes; notify remove listeners on
this node only
@CacheEntryModified -> addedEntries(Collections.singletonMap(entry.getKey(),
entry.getValue())) / updatedEntries(Collections.singletonMap(entry.getKey(),
entry.getValue()))
@CacheEntryRemoved -> removedEntries(Collections.singletonMap(entry.getKey(),
entry.getValue()))
- cluster-related update events originate on the server side, due to changes to the caches
underlying RegistryCollector and Registry
- these events are interpreted and propagated to the client side where they are used to
update the ClientContexts
{noformat}
The operation of these callbacks can be seen from the logs:
{no format}
[nrla@lenovo surefire-reports]$ grep Registry
org.jboss.as.test.clustering.cluster.ejb.remote.RemoteFailoverTestCase-SYNC-tcp-output.txt
// node-0, node-1 start up and deploy clustered app
15:11:18,705 INFO [stdout] (MSC service thread 1-8) Registry collector service started on
node node-0
15:11:25,895 INFO [stdout] (MSC service thread 1-9) Registry collector service started on
node node-1
15:11:36,017 INFO [stdout] (MSC service thread 1-2) Registry for cluster ejb added on
node node-0
15:11:39,950 INFO [stdout] (MSC service thread 1-10) Registry for cluster ejb added on
node node-1
// node-0 undeploys clustered app
15:11:42,320 INFO [stdout] (MSC service thread 1-2) Registry for cluster ejb removed on
node node-0
15:11:42,320 INFO [stdout] (MSC service thread 1-2) Registry removed on node node-0 with
(key) entries: [node-0, node-1] (same callback as previous message)
15:11:42,326 INFO [stdout] (remote-thread-0) Registry entries removed on node node-1 for
nodes: [node-0]
// node-0 deploys clustered app
15:11:43,180 INFO [stdout] (remote-thread-0) Registry entries added on node node-1 for
nodes: [node-0]
15:11:43,182 INFO [stdout] (MSC service thread 1-12) Registry for cluster ejb added on
node node-0
15:11:43,183 INFO [stdout] (MSC service thread 1-12) Registry added on node node-0 with
(key) entries: [node-0, node-1] (same callback as previous message)
// node-1 shuts down
15:11:48,644 INFO [stdout] (MSC service thread 1-9) Registry for cluster ejb removed on
node node-1
15:11:48,645 INFO [stdout] (MSC service thread 1-9) Registry removed on node node-1 with
(key) entries: [node-0, node-1] (same callback as previous message)
15:11:48,645 INFO [stdout] (MSC service thread 1-11) Registry collector service stopped
on node node-1
15:11:48,648 INFO [stdout] (remote-thread-0) Registry entries removed on node node-0 for
nodes: [node-1]
// node-1 starts up
15:11:55,788 INFO [stdout] (MSC service thread 1-12) Registry collector service started
on node node-1
15:11:57,365 INFO [stdout] (remote-thread-0) Registry entries added on node node-0 for
nodes: [node-1]
15:11:57,369 INFO [stdout] (MSC service thread 1-16) Registry for cluster ejb added on
node node-1
// node-0, node-1 shut down
15:12:12,374 INFO [stdout] (MSC service thread 1-16) Registry for cluster ejb removed on
node node-0
15:12:12,375 INFO [stdout] (MSC service thread 1-16) Registry removed on node node-0
with (key) entries: [node-0, node-1] (same callback as previous message)
15:12:12,829 INFO [stdout] (MSC service thread 1-12) Registry for cluster ejb removed on
node node-1
15:12:12,829 INFO [stdout] (MSC service thread 1-12) Registry removed on node node-1
with (key) entries: [node-1]
15:12:12,829 INFO [stdout] (MSC service thread 1-12) Registry removed on node node-1
with (key) entries: [node-1]
15:12:12,830 INFO [stdout] (MSC service thread 1-12) Registry removed on node node-1
with (key) entries: [node-1]
15:12:12,830 INFO [stdout] (MSC service thread 1-12) Registry removed on node node-1
with (key) entries: [node-1]
{noformat}
Topology updates of EJBClient ClusterContexts not being processed
correctly after failover
------------------------------------------------------------------------------------------
Key: WFLY-3711
URL:
https://issues.jboss.org/browse/WFLY-3711
Project: WildFly
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Clustering
Affects Versions: 9.0.0.Beta1
Reporter: Richard Achmatowicz
Assignee: Richard Achmatowicz
ClusterContexts are used by EJBClient to keep track of the current set of nodes in a
cluster, so that if an EJBClient invocation fails on one node, it may failover to another
node in the same cluster. The ClusterContext is made up of ClusterNodeManagers which are
responsible for setting up the connections between the EJBClient and the nodes in the
cluster.
Cluster topology updates are sent to registered EJBClients whenever the cluster topology
changes (nodes join, nodes leave a cluster). Thse topology updates are processed on the
client side by ClusterTopologyUpdateHandler and are used to update the current contents of
the associated ClusterContext held on the client.
The current implementation of the handling of topology updates does not correctly handle
the addition/removal of ClusterNodeManagers from the cluster context - namely, rather than
check whether or not a new ClusterNodeManager really needs to be added,
ClusterNodeManagers are added for every node in the received topology update, leading to
many unnecessary EJBReceivers and their channels being created.
The logs show that up to 18 cluster node manager instances may be created, have their
EJBReceivers registered and channels to the remote node created, when only one node has
been added to the cluster.
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)