[JBoss JIRA] (ISPN-9517) State transfer times out if initiated with yet to be verified suspected member and reincarnated member
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/ISPN-9517?page=com.atlassian.jira.plugin.... ]
Paul Ferraro updated ISPN-9517:
-------------------------------
Attachment: Test.java
> State transfer times out if initiated with yet to be verified suspected member and reincarnated member
> ------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9517
> URL: https://issues.jboss.org/browse/ISPN-9517
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 9.3.3.Final
> Reporter: Paul Ferraro
> Assignee: Paul Ferraro
> Attachments: Test.java, node-1.zip, node-2.zip
>
>
> Here's the scenario:
> 1. Cluster contains caches on 2 members, node-1 and node-2
> 2. node-2 is killed
> 3. node-2 is restarted (using same physical address)
> 4. State transfer initiates, view contains node-1, suspected node-2, and reincarnated node-2
> 5. State transfer times out
> Log of node-1 includes:
> {noformat}
> 12:09:51,882 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p14-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 3 from node-2
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_181]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
> Suppressed: org.infinispan.util.logging.TraceException
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:75)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:525)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:508)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:321)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:87)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:731)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_181]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> {noformat}
> I've attached trace logs from node-1 and node-2.
> Changing ClusterTopologyManagerImpl.confirmMembersAvailable() to use ResponseMode.SYNCHRONOUS_IGNORE_LEAVERS instead of ResponseMode.SYNCHRONOUS allows state transfer to complete successfully.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-9517) State transfer times out if initiated with yet to be verified suspected member and reincarnated member
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/ISPN-9517?page=com.atlassian.jira.plugin.... ]
Paul Ferraro updated ISPN-9517:
-------------------------------
Attachment: (was: Test.java)
> State transfer times out if initiated with yet to be verified suspected member and reincarnated member
> ------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9517
> URL: https://issues.jboss.org/browse/ISPN-9517
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 9.3.3.Final
> Reporter: Paul Ferraro
> Assignee: Paul Ferraro
> Attachments: Test.java, node-1.zip, node-2.zip
>
>
> Here's the scenario:
> 1. Cluster contains caches on 2 members, node-1 and node-2
> 2. node-2 is killed
> 3. node-2 is restarted (using same physical address)
> 4. State transfer initiates, view contains node-1, suspected node-2, and reincarnated node-2
> 5. State transfer times out
> Log of node-1 includes:
> {noformat}
> 12:09:51,882 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p14-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 3 from node-2
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_181]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
> Suppressed: org.infinispan.util.logging.TraceException
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:75)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:525)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:508)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:321)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:87)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:731)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_181]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> {noformat}
> I've attached trace logs from node-1 and node-2.
> Changing ClusterTopologyManagerImpl.confirmMembersAvailable() to use ResponseMode.SYNCHRONOUS_IGNORE_LEAVERS instead of ResponseMode.SYNCHRONOUS allows state transfer to complete successfully.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-9517) State transfer times out if initiated with yet to be verified suspected member and reincarnated member
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/ISPN-9517?page=com.atlassian.jira.plugin.... ]
Paul Ferraro updated ISPN-9517:
-------------------------------
Attachment: Test.java
> State transfer times out if initiated with yet to be verified suspected member and reincarnated member
> ------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9517
> URL: https://issues.jboss.org/browse/ISPN-9517
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 9.3.3.Final
> Reporter: Paul Ferraro
> Assignee: Paul Ferraro
> Attachments: Test.java, node-1.zip, node-2.zip
>
>
> Here's the scenario:
> 1. Cluster contains caches on 2 members, node-1 and node-2
> 2. node-2 is killed
> 3. node-2 is restarted (using same physical address)
> 4. State transfer initiates, view contains node-1, suspected node-2, and reincarnated node-2
> 5. State transfer times out
> Log of node-1 includes:
> {noformat}
> 12:09:51,882 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p14-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 3 from node-2
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_181]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
> Suppressed: org.infinispan.util.logging.TraceException
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:75)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:525)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:508)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:321)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:87)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:731)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_181]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> {noformat}
> I've attached trace logs from node-1 and node-2.
> Changing ClusterTopologyManagerImpl.confirmMembersAvailable() to use ResponseMode.SYNCHRONOUS_IGNORE_LEAVERS instead of ResponseMode.SYNCHRONOUS allows state transfer to complete successfully.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-9517) State transfer times out if initiated with yet to be verified suspected member and reincarnated member
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/ISPN-9517?page=com.atlassian.jira.plugin.... ]
Paul Ferraro updated ISPN-9517:
-------------------------------
Attachment: (was: Test.java)
> State transfer times out if initiated with yet to be verified suspected member and reincarnated member
> ------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9517
> URL: https://issues.jboss.org/browse/ISPN-9517
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 9.3.3.Final
> Reporter: Paul Ferraro
> Assignee: Paul Ferraro
> Attachments: Test.java, node-1.zip, node-2.zip
>
>
> Here's the scenario:
> 1. Cluster contains caches on 2 members, node-1 and node-2
> 2. node-2 is killed
> 3. node-2 is restarted (using same physical address)
> 4. State transfer initiates, view contains node-1, suspected node-2, and reincarnated node-2
> 5. State transfer times out
> Log of node-1 includes:
> {noformat}
> 12:09:51,882 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p14-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 3 from node-2
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_181]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
> Suppressed: org.infinispan.util.logging.TraceException
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:75)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:525)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:508)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:321)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:87)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:731)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_181]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> {noformat}
> I've attached trace logs from node-1 and node-2.
> Changing ClusterTopologyManagerImpl.confirmMembersAvailable() to use ResponseMode.SYNCHRONOUS_IGNORE_LEAVERS instead of ResponseMode.SYNCHRONOUS allows state transfer to complete successfully.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-9517) State transfer times out if initiated with yet to be verified suspected member and reincarnated member
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/ISPN-9517?page=com.atlassian.jira.plugin.... ]
Paul Ferraro updated ISPN-9517:
-------------------------------
Attachment: Test.java
> State transfer times out if initiated with yet to be verified suspected member and reincarnated member
> ------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9517
> URL: https://issues.jboss.org/browse/ISPN-9517
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 9.3.3.Final
> Reporter: Paul Ferraro
> Assignee: Paul Ferraro
> Attachments: Test.java, node-1.zip, node-2.zip
>
>
> Here's the scenario:
> 1. Cluster contains caches on 2 members, node-1 and node-2
> 2. node-2 is killed
> 3. node-2 is restarted (using same physical address)
> 4. State transfer initiates, view contains node-1, suspected node-2, and reincarnated node-2
> 5. State transfer times out
> Log of node-1 includes:
> {noformat}
> 12:09:51,882 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p14-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 3 from node-2
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_181]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
> Suppressed: org.infinispan.util.logging.TraceException
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:75)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:525)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:508)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:321)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:87)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:731)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_181]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> {noformat}
> I've attached trace logs from node-1 and node-2.
> Changing ClusterTopologyManagerImpl.confirmMembersAvailable() to use ResponseMode.SYNCHRONOUS_IGNORE_LEAVERS instead of ResponseMode.SYNCHRONOUS allows state transfer to complete successfully.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-9517) State transfer times out if initiated with yet to be verified suspected member and reincarnated member
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/ISPN-9517?page=com.atlassian.jira.plugin.... ]
Paul Ferraro commented on ISPN-9517:
------------------------------------
Reproducer updated to run until failure (or interruption).
> State transfer times out if initiated with yet to be verified suspected member and reincarnated member
> ------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9517
> URL: https://issues.jboss.org/browse/ISPN-9517
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 9.3.3.Final
> Reporter: Paul Ferraro
> Assignee: Paul Ferraro
> Attachments: Test.java, node-1.zip, node-2.zip
>
>
> Here's the scenario:
> 1. Cluster contains caches on 2 members, node-1 and node-2
> 2. node-2 is killed
> 3. node-2 is restarted (using same physical address)
> 4. State transfer initiates, view contains node-1, suspected node-2, and reincarnated node-2
> 5. State transfer times out
> Log of node-1 includes:
> {noformat}
> 12:09:51,882 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p14-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 3 from node-2
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_181]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
> Suppressed: org.infinispan.util.logging.TraceException
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:75)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:525)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:508)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:321)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:87)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:731)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_181]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> {noformat}
> I've attached trace logs from node-1 and node-2.
> Changing ClusterTopologyManagerImpl.confirmMembersAvailable() to use ResponseMode.SYNCHRONOUS_IGNORE_LEAVERS instead of ResponseMode.SYNCHRONOUS allows state transfer to complete successfully.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-9345) TimeutException involving the org.infinispan.CONFIG cache with IPv6
by Michal Stehlik (JIRA)
[ https://issues.jboss.org/browse/ISPN-9345?page=com.atlassian.jira.plugin.... ]
Michal Stehlik commented on ISPN-9345:
--------------------------------------
[~dan.berindei] I created new issue: [https://issues.jboss.org/browse/ISPN-9527|https://issues.jboss.org/browse...]
> TimeutException involving the org.infinispan.CONFIG cache with IPv6
> -------------------------------------------------------------------
>
> Key: ISPN-9345
> URL: https://issues.jboss.org/browse/ISPN-9345
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.3.0.Final
> Reporter: Gustavo Fernandes
> Assignee: Dan Berindei
>
> {noformat}
> Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache org.infinispan.CONFIG on jedha-64980
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-9527) Initial state transfer timed out for cache org.infinispan.CONFIG
by Michal Stehlik (JIRA)
Michal Stehlik created ISPN-9527:
------------------------------------
Summary: Initial state transfer timed out for cache org.infinispan.CONFIG
Key: ISPN-9527
URL: https://issues.jboss.org/browse/ISPN-9527
Project: Infinispan
Issue Type: Bug
Components: Cloud
Affects Versions: 9.3.0.Final
Environment: 15 nodes connecting to cloud in same time. ARM based processors, 1GHz processor, 1GB RAM. IPv4
Reporter: Michal Stehlik
Attachments: cluster_config_distr.xml, infinispan.log, jgroupsObjectModel.xml
During starting of caches, several nodes fails due following exception (IPv4):
{code:java}
Cloud access error! org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart() on object of type GlobalConfigurationManagerImpl
org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart() on object of type GlobalConfigurationManagerImpl
at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:271) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
...
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart() on object of type GlobalConfigurationManagerImpl
at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.security.Security.doPrivileged(Security.java:44) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.AbstractComponentRegistry.postStart(AbstractComponentRegistry.java:620) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:267) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
... 33 more
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.security.Security.doPrivileged(Security.java:49) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:244) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.getStateCache(GlobalConfigurationManagerImpl.java:97) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart(GlobalConfigurationManagerImpl.java:80) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_131]
at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.security.Security.doPrivileged(Security.java:44) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.AbstractComponentRegistry.postStart(AbstractComponentRegistry.java:620) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:267) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
... 33 more
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache org.infinispan.CONFIG on 0125f36a-2860-4107-810a-d087224c9105-39521
at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_131]
at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.security.Security.doPrivileged(Security.java:49) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:244) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.getStateCache(GlobalConfigurationManagerImpl.java:97) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart(GlobalConfigurationManagerImpl.java:80) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_131]
at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.security.Security.doPrivileged(Security.java:44) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.AbstractComponentRegistry.postStart(AbstractComponentRegistry.java:620) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:267) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
... 33 more
{code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-9517) State transfer times out if initiated with yet to be verified suspected member and reincarnated member
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/ISPN-9517?page=com.atlassian.jira.plugin.... ]
Paul Ferraro updated ISPN-9517:
-------------------------------
Attachment: (was: Test.java)
> State transfer times out if initiated with yet to be verified suspected member and reincarnated member
> ------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9517
> URL: https://issues.jboss.org/browse/ISPN-9517
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 9.3.3.Final
> Reporter: Paul Ferraro
> Assignee: Paul Ferraro
> Attachments: Test.java, node-1.zip, node-2.zip
>
>
> Here's the scenario:
> 1. Cluster contains caches on 2 members, node-1 and node-2
> 2. node-2 is killed
> 3. node-2 is restarted (using same physical address)
> 4. State transfer initiates, view contains node-1, suspected node-2, and reincarnated node-2
> 5. State transfer times out
> Log of node-1 includes:
> {noformat}
> 12:09:51,882 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p14-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 3 from node-2
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_181]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
> Suppressed: org.infinispan.util.logging.TraceException
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:75)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:525)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:508)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:321)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:87)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:731)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_181]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> {noformat}
> I've attached trace logs from node-1 and node-2.
> Changing ClusterTopologyManagerImpl.confirmMembersAvailable() to use ResponseMode.SYNCHRONOUS_IGNORE_LEAVERS instead of ResponseMode.SYNCHRONOUS allows state transfer to complete successfully.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months
[JBoss JIRA] (ISPN-9517) State transfer times out if initiated with yet to be verified suspected member and reincarnated member
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/ISPN-9517?page=com.atlassian.jira.plugin.... ]
Paul Ferraro updated ISPN-9517:
-------------------------------
Attachment: Test.java
> State transfer times out if initiated with yet to be verified suspected member and reincarnated member
> ------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9517
> URL: https://issues.jboss.org/browse/ISPN-9517
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 9.3.3.Final
> Reporter: Paul Ferraro
> Assignee: Paul Ferraro
> Attachments: Test.java, node-1.zip, node-2.zip
>
>
> Here's the scenario:
> 1. Cluster contains caches on 2 members, node-1 and node-2
> 2. node-2 is killed
> 3. node-2 is restarted (using same physical address)
> 4. State transfer initiates, view contains node-1, suspected node-2, and reincarnated node-2
> 5. State transfer times out
> Log of node-1 includes:
> {noformat}
> 12:09:51,882 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p14-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 3 from node-2
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_181]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
> Suppressed: org.infinispan.util.logging.TraceException
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:75)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:525)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:508)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:321)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:87)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:731)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:37)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:227)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_181]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> {noformat}
> I've attached trace logs from node-1 and node-2.
> Changing ClusterTopologyManagerImpl.confirmMembersAvailable() to use ResponseMode.SYNCHRONOUS_IGNORE_LEAVERS instead of ResponseMode.SYNCHRONOUS allows state transfer to complete successfully.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 6 months