[JBoss JIRA] (ISPN-5493) SiteProviderTopologyChangeTest.testXSiteSTDuringLeave random failures
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-5493?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-5493:
-------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> SiteProviderTopologyChangeTest.testXSiteSTDuringLeave random failures
> ---------------------------------------------------------------------
>
> Key: ISPN-5493
> URL: https://issues.jboss.org/browse/ISPN-5493
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Cross-Site Replication, Test Suite - Core
> Affects Versions: 7.2.1.Final
> Reporter: Dan Berindei
> Assignee: Pedro Ruivo
> Priority: Blocker
> Labels: testsuite_failure
> Fix For: 8.2.0.Alpha1
>
> Attachments: SiteProviderTopologyChangeTest.log.gz
>
>
> Looks like the node is killed before the {{XSiteStateTransferControlCommand(START_SEND)}} command was replicated to all the nodes in the source cluster, and the {{SuspectException}} stops the state push:
> {noformat}
> 23:33:22,834 DEBUG (testng-SiteProviderTopologyChangeTest:) [XSiteAdminOperations] Unable to pushState to 'NYC'.java.lang.Exception: java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: Suspected member: SiteProviderTopologyChangeTest-NodeBG-25313
> at org.infinispan.xsite.statetransfer.XSiteStateTransferManagerImpl.startPushState(XSiteStateTransferManagerImpl.java:151)
> at org.infinispan.xsite.XSiteAdminOperations.pushState(XSiteAdminOperations.java:238)
> at org.infinispan.xsite.statetransfer.failures.AbstractTopologyChangeTest.startStateTransfer(AbstractTopologyChangeTest.java:144)
> at org.infinispan.xsite.statetransfer.failures.SiteProviderTopologyChangeTest.doXSiteStateTransferDuringTopologyChange(SiteProviderTopologyChangeTest.java:241)
> at org.infinispan.xsite.statetransfer.failures.SiteProviderTopologyChangeTest.testXSiteSTDuringLeave(SiteProviderTopologyChangeTest.java:78)
> Caused by: java.util.concurrent.ExecutionException: org.infinispan.remoting.transport.jgroups.SuspectException: Suspected member: SiteProviderTopologyChangeTest-NodeBG-25313
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.infinispan.commons.util.concurrent.NotifyingFutureImpl.get(NotifyingFutureImpl.java:77)
> at org.infinispan.xsite.statetransfer.XSiteStateTransferManagerImpl.invokeRemotelyInLocalSite(XSiteStateTransferManagerImpl.java:376)
> at org.infinispan.xsite.statetransfer.XSiteStateTransferManagerImpl.controlStateTransferOnLocalSite(XSiteStateTransferManagerImpl.java:335)
> at org.infinispan.xsite.statetransfer.XSiteStateTransferManagerImpl.startPushState(XSiteStateTransferManagerImpl.java:142)
> ... 24 more
> Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: Suspected member: SiteProviderTopologyChangeTest-NodeBG-25313
> at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:74)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:586)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:287)
> at org.infinispan.remoting.rpc.RpcManagerImpl$2.call(RpcManagerImpl.java:382)
> at org.infinispan.remoting.rpc.RpcManagerImpl$2.call(RpcManagerImpl.java:378)
> ... 4 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 12 months
[JBoss JIRA] (ISPN-6043) TransactionTable should ignore view changes during shutdown
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-6043?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-6043:
----------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 8.2.0.Final
Resolution: Done
> TransactionTable should ignore view changes during shutdown
> -----------------------------------------------------------
>
> Key: ISPN-6043
> URL: https://issues.jboss.org/browse/ISPN-6043
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 8.1.0.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 8.2.0.Alpha1, 8.2.0.Final, 8.1.1.Final
>
>
> During shutdown, {{TransactionTable}} unregisters itself as a view change listener, but it can still receive view change notifications after it stopped the executor service. When that happens, it causes a {{RejectedExecutionException}} that is eventually logged by JGroups:
> {noformat}
> pbcast.GMS - JGRP000027: failed passing message up
> java.lang.RuntimeException: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.util.concurrent.RejectedExecutionException] while invoking method [public void org.infinispan.transaction.TransactionTable.onViewChange(org.infinispan.notifications.cachemanagerlistener.event.ViewChangedEvent)] on listener instance: org.infinispan.transaction.TransactionTable@3d5ab0ba
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:682)
> at org.jgroups.JChannel.up(JChannel.java:733)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029)
> at org.jgroups.protocols.RSVP.up(RSVP.java:201)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)
> at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:732)
> at org.jgroups.protocols.pbcast.ParticipantGmsImpl.handleViewChange(ParticipantGmsImpl.java:146)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:922)
> at org.jgroups.stack.Protocol.up(Protocol.java:412)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:474)
> at org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:982)
> at org.jgroups.protocols.pbcast.NAKACK2.removeAndPassUp(NAKACK2.java:912)
> at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:846)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:618)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD.up(FD.java:255)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:297)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:288)
> at org.jgroups.protocols.Discovery.up(Discovery.java:291)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1572)
> at org.jgroups.protocols.TP$MyHandler.run(TP.java:1791)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.util.concurrent.RejectedExecutionException] while invoking method [public void org.infinispan.transaction.TransactionTable.onViewChange(org.infinispan.notifications.cachemanagerlistener.event.ViewChangedEvent)] on listener instance: org.infinispan.transaction.TransactionTable@3d5ab0ba
> at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:287)
> at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:22)
> at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:305)
> at org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl.notifyViewChange(CacheManagerNotifierImpl.java:88)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport$NotifyViewChange.emitNotification(JGroupsTransport.java:638)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.viewAccepted(JGroupsTransport.java:708)
> at org.jgroups.blocks.MessageDispatcher.handleUpEvent(MessageDispatcher.java:602)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:679)
> ... 25 more
> Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1f5986a3 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@2e964769[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1696]
> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
> at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:325)
> at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:546)
> at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:646)
> at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:641)
> at org.infinispan.transaction.TransactionTable.onViewChange(TransactionTable.java:491)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:282)
> ... 32 more
> {noformat}
> The exception is harmless for the stopping cache, the problem is that the following view change listeners are also skipped. We should fix both {{TransactionTable}} to avoid throwing the exception, and {{CacheManagerNotifier}} to ignore any exceptions during view changes.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 12 months
[JBoss JIRA] (ISPN-6017) Deprecate ReplicationQueue
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-6017?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-6017:
----------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 8.2.0.Alpha1
8.2.0.Final
Resolution: Done
> Deprecate ReplicationQueue
> --------------------------
>
> Key: ISPN-6017
> URL: https://issues.jboss.org/browse/ISPN-6017
> Project: Infinispan
> Issue Type: Task
> Components: Core
> Affects Versions: 8.0.0.Final
> Reporter: Radoslav Husar
> Assignee: Radoslav Husar
> Fix For: 8.2.0.Alpha1, 8.2.0.Final
>
>
> Per infinispan-dev list:
> Pedro:
> Can we remove the ReplicationQueue? First, it does not have any benefit
> (JGroups already bundles the message and the Network can do it too) and
> second, it is not more efficient (when the message is delivered, we
> process the commands sequentially. So, if the first command blocks, the
> other commands are not processed until it finished). Third, it is
> complex if you have commands with multiple delivers orders (no order,
> FIFO, total)
> Dan:
> +1, the replication queue has the same purpose as the bundler in
> JGroups. And while the JGroups bundler has multiple options that keep
> evolving, our replication queue still has the same algorithm it had in
> 4.0.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 12 months