[infinispan-issues] [JBoss JIRA] (ISPN-7332) XSite trying to replicate to site after site has been shutdown

[Tristan Tarrant (JIRA)]

     [ https://issues.jboss.org/browse/ISPN-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tristan Tarrant reassigned ISPN-7332:
-------------------------------------

    Assignee: Pedro Ruivo


> XSite trying to replicate to site after site has been shutdown
> --------------------------------------------------------------
>
>                 Key: ISPN-7332
>                 URL: https://issues.jboss.org/browse/ISPN-7332
>             Project: Infinispan
>          Issue Type: Bug
>    Affects Versions: 9.0.0.Beta1
>            Reporter: Galder Zamarreño
>            Assignee: Pedro Ruivo
>         Attachments: xsite-bug.tgz
>
>
> In a 2-site scenario, with a server in each site, when the server in one of the sites goes down, and as a result the entire site is gone, the initial site might still try to replicate to the other site. Example:
> Sites: EARTH and MOON
> Servers: server-earth-one and server-moon-one respectively
> server-moon-one stops:
> {code}
> 2017-01-04 12:16:38,666 INFO  [org.jboss.as] (MSC service thread 1-4) WFLYSRV0050: 
> Infinispan Server 9.0.0.Beta1 (WildFly Core 2.2.0.Final) stopped in 102ms
> {code}
> server-earth-one realises that and sets the correct view:
> {code}
> 2017-01-04 12:16:38,649 TRACE [org.jgroups.protocols.relay.RELAY2] (jgroups-3,_master:server-earth-one:EARTH) 
> [Relayer _master:server-earth-one:EARTH] view: [_master:server-earth-one:EARTH|4] (1) [_master:server-earth-one:EARTH]
> {code}
> server-earth-one gets a put invocation 
> {code}
> 2017-01-04 12:16:38,709 TRACE [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRodServerHandler-8-1) 
> Invoked with command PutKeyValueCommand{key=org.infinispan.commons.marshall.WrappedByteArray at a3b01a15, 
> value=org.infinispan.commons.marshall.WrappedByteArray at b68d6067, flags=[IGNORE_RETURN_VALUES], putIfAbsent=false, 
> valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=NumericVersion{version=4294967297}}, 
> successful=true} and InvocationContext [org.infinispan.context.SingleKeyNonTxInvocationContext at 3d9b13cf]
> {code}
> But for some reason, server-earth-one still tries to send it to the MOON site:
> {code}
> 2017-01-04 12:16:38,713 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (HotRodServerHandler-8-1) 
> About to send to backups [MOON (sync, timeout=10000)], command SingleXSiteRpcCommand{command=PutKeyValueCommand{key=org.infinispan.commons.marshall.WrappedByteArray at a3b01a15, 
> value=org.infinispan.commons.marshall.WrappedByteArray at b68d6067, flags=[IGNORE_RETURN_VALUES], putIfAbsent=false, 
> valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=NumericVersion{version=4294967297}}, 
> successful=true}}
> {code}
> ^ That should not happen.
> Moreover, the JGroups layer detects there's no site already:
> {code}
> 2017-01-04 12:16:38,717 ERROR [org.jgroups.protocols.relay.RELAY2] (HotRodServerHandler-8-1) 
> master:server-earth-one: no route to MOON: dropping message
> {code}
> But timeout needs to occur for the put to complete:
> {code}
> 2017-01-04 12:16:48,721 WARN  [org.infinispan.xsite.BackupSenderImpl] (HotRodServerHandler-8-1) 
> ISPN000202: Problems backing up data for cache xsiteCache to site MOON: org.infinispan.util.concurrent.TimeoutException: 
> Timed out after 10 seconds waiting for a response from MOON (sync, timeout=10000)
> ...
> 2017-01-04 12:16:48,726 TRACE [org.infinispan.server.hotrod.HotRodEncoder] (HotRodServerWorker-7-1) 
> Encode msg EmptyResponse{version=25, messageId=21, cacheName='xsiteCache', clientIntel=3, operation=PUT, status=Success, topologyId=1}
> {code}
> I'm attaching full TRACE logs.



--
This message was sent by Atlassian JIRA
(v7.5.0#75005)