[JBoss JIRA] (ISPN-5883) Node can apply new topology after sending status response
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-5883?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-5883:
-------------------------------
Git Pull Request: https://github.com/infinispan/infinispan/pull/3782, https://github.com/infinispan/infinispan/pull/3877 (was: https://github.com/infinispan/infinispan/pull/3782)
> Node can apply new topology after sending status response
> ---------------------------------------------------------
>
> Key: ISPN-5883
> URL: https://issues.jboss.org/browse/ISPN-5883
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Core
> Affects Versions: 8.0.1.Final, 7.2.5.Final, 8.1.0.Alpha2
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: testsuite_stability
> Fix For: 8.1.0.Final
>
>
> {{LocalTopologyManagerImpl}} is responsible for sending the {{ClusterTopologyControlCommand(GET_STATUS)}} response, and when it sends the response it doesn't check the current view id against the new coordinator's view id. If the old coordinator already sent a topology update before the merge, that topology update might be processed after sending the status response. The new coordinator will send a topology update with a topology id of {{max(status response topology ids) + 1}}. The node will then process the topology update from the old coordinator, but it will ignore the topology update from the new coordinator with the same topology id.
> This is extra common in the partition handling tests, e.g. {{BasePessimisticTxPartitionAndMergeTest}} subclasses, because the test "injects" the JGroups view on each node serially, and often the 4th node sends the status response before it gets the new view.
> {noformat}
> 22:16:37,776 DEBUG (remote-thread-NodeD-p26-t6:[]) [LocalTopologyManagerImpl] Sending cluster status response for view 10
> // Topology from NodeC
> 22:16:37,778 DEBUG (transport-thread-NodeD-p28-t2:[]) [LocalTopologyManagerImpl] Updating local topology for cache pes-cache: CacheTopology{id=8, rebalanceId=3, currentCH=DefaultConsistentHash{ns=60, owners = (4)[NodeA-37631: 15+15, NodeB-47846: 15+15, NodeC-46467: 15+15, NodeD-30486: 15+15]}, pendingCH=null, unionCH=null, actualMembers=[NodeC-46467, NodeD-30486]}
> // Later, topology from NodeA
> 22:16:37,827 DEBUG (transport-thread-NodeD-p28-t1:[]) [LocalTopologyManagerImpl] Ignoring late consistent hash update for cache pes-cache, current topology is 8: CacheTopology{id=8, rebalanceId=3, currentCH=DefaultConsistentHash{ns=60, owners = (4)[NodeA-37631: 15+15, NodeB-47846: 15+15, NodeC-46467: 15+15, NodeD-30486: 15+15]}, pendingCH=null, unionCH=null, actualMembers=[NodeA-37631, NodeB-47846, NodeC-46467, NodeD-30486]}
> {noformat}
> As a solution, we can delay sending the status response until we have the same view as the coordinator (or a later one). We already check that the sender is the current coordinator before applying a topology update, so this will guarantee that the we don't apply other topology updates from the old coordinator. Since the status request is only sent after the new view was installed, this will not introduce any delays in the vast majority of cases.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 4 months
[JBoss JIRA] (ISPN-6012) Too many warnings about unsolicited state during cluster shutdown
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-6012?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-6012:
-------------------------------
Status: Open (was: New)
> Too many warnings about unsolicited state during cluster shutdown
> -----------------------------------------------------------------
>
> Key: ISPN-6012
> URL: https://issues.jboss.org/browse/ISPN-6012
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 8.0.2.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 8.1.0.Final
>
>
> When shutting down, {{StateConsumerImpl}} first clears the map of inbound transfers. Any {{StateResponseCommand}} received afterwards will trigger a {{Received unsolicited state from node %s for segment %d of cache %s}} warning. We should just ignore any {{StateResponseCommand}} received during shutdown, without any warning.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 4 months
[JBoss JIRA] (ISPN-6013) Bind awestruct to 0.0.0.0 in Docker image
by Emmanuel Bernard (JIRA)
Emmanuel Bernard created ISPN-6013:
--------------------------------------
Summary: Bind awestruct to 0.0.0.0 in Docker image
Key: ISPN-6013
URL: https://issues.jboss.org/browse/ISPN-6013
Project: Infinispan
Issue Type: Enhancement
Reporter: Emmanuel Bernard
The website docker image was no longer exposing Awestruct as 4242 because Awestruct was binding itself to 127.0.0.1. That does not work at least for Mac OS and docker-machine based settings.
Now in the docker image, Awestruct is bound to 0.0.0.0
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 4 months
[JBoss JIRA] (ISPN-6012) Too many warnings about unsolicited state during cluster shutdown
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-6012?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-6012:
-------------------------------
Description: When shutting down, {{StateConsumerImpl}} first clears the map of inbound transfers. Any {{StateResponseCommand}} received afterwards will trigger a "Received unsolicited state from node %s for segment %d of cache %s". We should just ignore any {{StateResponseCommand}} received during shutdown, without any warning. (was: When shutting down, {{StateConsumerImpl}} first clears the map of inbound transfers. Any {{StateResponseCommand}}s received afterwards will trigger a "Received unsolicited state from node %s for segment %d of cache %s". We should just ignore any {{StateResponseCommand}} received during shutdown, without any warning.)
> Too many warnings about unsolicited state during cluster shutdown
> -----------------------------------------------------------------
>
> Key: ISPN-6012
> URL: https://issues.jboss.org/browse/ISPN-6012
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 8.0.2.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 8.1.0.Final
>
>
> When shutting down, {{StateConsumerImpl}} first clears the map of inbound transfers. Any {{StateResponseCommand}} received afterwards will trigger a "Received unsolicited state from node %s for segment %d of cache %s". We should just ignore any {{StateResponseCommand}} received during shutdown, without any warning.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 4 months
[JBoss JIRA] (ISPN-6012) Too many warnings about unsolicited state during cluster shutdown
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-6012?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-6012:
-------------------------------
Description: When shutting down, {{StateConsumerImpl}} first clears the map of inbound transfers. Any {{StateResponseCommand}} received afterwards will trigger a {{Received unsolicited state from node %s for segment %d of cache %s}} warning. We should just ignore any {{StateResponseCommand}} received during shutdown, without any warning. (was: When shutting down, {{StateConsumerImpl}} first clears the map of inbound transfers. Any {{StateResponseCommand}} received afterwards will trigger a "Received unsolicited state from node %s for segment %d of cache %s". We should just ignore any {{StateResponseCommand}} received during shutdown, without any warning.)
> Too many warnings about unsolicited state during cluster shutdown
> -----------------------------------------------------------------
>
> Key: ISPN-6012
> URL: https://issues.jboss.org/browse/ISPN-6012
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 8.0.2.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 8.1.0.Final
>
>
> When shutting down, {{StateConsumerImpl}} first clears the map of inbound transfers. Any {{StateResponseCommand}} received afterwards will trigger a {{Received unsolicited state from node %s for segment %d of cache %s}} warning. We should just ignore any {{StateResponseCommand}} received during shutdown, without any warning.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 4 months
[JBoss JIRA] (ISPN-6012) Too many warnings about unsolicited state during cluster shutdown
by Dan Berindei (JIRA)
Dan Berindei created ISPN-6012:
----------------------------------
Summary: Too many warnings about unsolicited state during cluster shutdown
Key: ISPN-6012
URL: https://issues.jboss.org/browse/ISPN-6012
Project: Infinispan
Issue Type: Bug
Components: Core
Affects Versions: 8.0.2.Final
Reporter: Dan Berindei
Assignee: Dan Berindei
Fix For: 8.1.0.Final
When shutting down, {{StateConsumerImpl}} first clears the map of inbound transfers. Any {{StateResponseCommand}}s received afterwards will trigger a "Received unsolicited state from node %s for segment %d of cache %s". We should just ignore any {{StateResponseCommand}} received during shutdown, without any warning.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 4 months