[infinispan-issues] [JBoss JIRA] (ISPN-5208) Avoid invalid topology

[Matej Čimbora (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052839#comment-13052839 ] 

Matej Čimbora commented on ISPN-5208:
-------------------------------------

Problem still persists, i.e. I'm getting exception as included in description

java.lang.ArrayIndexOutOfBoundsException: 0
	at org.infinispan.client.hotrod.impl.consistenthash.SegmentConsistentHash.getServer(SegmentConsistentHash.java:33)
...

I managed to create reproducer for this (attached).

1. In clustered.xml, modify numOwners attribute for 'default' cache to 1.
2. Start 2 servers with clustered.xml configuration. 
3. Start attached reproducer (client).
4. Kill one of the servers.
5. Start it again. Exception should appear in client log.

This probably relates to losing segments for a subset of keys.

> Avoid invalid topology
> ----------------------
>
>                 Key: ISPN-5208
>                 URL: https://issues.jboss.org/browse/ISPN-5208
>             Project: Infinispan
>          Issue Type: Enhancement
>          Components: Server
>            Reporter: Takayoshi Kimura
>            Assignee: Galder Zamarreño
>              Labels: jdg641
>             Fix For: 7.2.0.Final
>
>
> We've seen some invalid topology propagated to client and it causes ArrayIndexOutOfBoundsException:
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
>   	at org.infinispan.client.hotrod.impl.transport.tcp.RoundRobinBalancingStrategy.getServerByIndex(RoundRobinBalancingStrategy.java:68) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
>   	at org.infinispan.client.hotrod.impl.transport.tcp.RoundRobinBalancingStrategy.nextServer(RoundRobinBalancingStrategy.java:44) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
>   	at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.nextServer(TcpTransportFactory.java:220) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
>   	at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:194) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
>   	at org.infinispan.client.hotrod.impl.operations.FaultTolerantPingOperation.getTransport(FaultTolerantPingOperation.java:27) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
>   	at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:48) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
>   	at org.infinispan.client.hotrod.impl.RemoteCacheImpl.ping(RemoteCacheImpl.java:535) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
>   	at org.infinispan.client.hotrod.RemoteCacheManager.ping(RemoteCacheManager.java:635) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
>   	at org.infinispan.client.hotrod.RemoteCacheManager.createRemoteCache(RemoteCacheManager.java:616) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
>   	at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:527) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
>     at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:523) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> {noformat}
> {noformat}
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0
>     at org.infinispan.client.hotrod.impl.consistenthash.SegmentConsistentHash.getServer(SegmentConsistentHash.java:33)
>     at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:204)
>     at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.getTransport(AbstractKeyOperation.java:40)
>     at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:48)
>     at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:237)
>     at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:79)
>     at sample.Main.main(Main.java:16)
> {noformat}
> It happens on both Hot Rod 2 and 1.3 clients.
> It's really hard to reproduce this state and we don't have a consistent way to reproduce it. However when this happens there is always view change happening so it's related to view change.
> Judging from the stack trace, the client receives numOwners=0 or numSegments=0 topology from the server.
> Also we are unable to find to recover this situation. Rebooting random nodes don't help and keep getting this exceptions on client side.
> Until we can find the root cause, I think it's better to add a guard to avoid this kind invalid topology stored in the server side and propagated to the clients.



--
This message was sent by Atlassian JIRA
(v6.3.11#6341)