[infinispan-issues] [JBoss JIRA] (ISPN-1654) Topology view management in Hotrod Server is not reliable
Galder Zamarreño (Commented) (JIRA)
jira-events at lists.jboss.org
Mon Jan 2 11:10:09 EST 2012
[ https://issues.jboss.org/browse/ISPN-1654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653363#comment-12653363 ]
Galder Zamarreño commented on ISPN-1654:
----------------------------------------
@Jacek, thanks for the report. Do you have any logs of the servers so that we can try to narrow down the issue? In particular, I'd like to have a look at one log file from either .104 or .102 and another log from one of the other machines.
> Topology view management in Hotrod Server is not reliable
> ---------------------------------------------------------
>
> Key: ISPN-1654
> URL: https://issues.jboss.org/browse/ISPN-1654
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.1.0.CR1
> Reporter: Jacek Gerbszt
> Assignee: Manik Surtani
> Priority: Blocker
>
> There is a problem with management of cluster topology view in address cache. You can see that by using remote client - there is the only way I know to see what is inside the address cache.
> When I restart the whole cluster and make a call from remote client, I receive full cluster topology (25 nodes):
> {code}
> INFO 02 sty 11:24:38 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 - ISPN004006: New topology: [/10.0.36.150:11311, /10.0.36.134:11311, /10.0.36.102:11311, /10.0.36.110:11311, /10.0.36.142:11311, /10.0.36.140:11311, /10.0.36.132:11311, /10.0.36.120:11311, /10.0.36.116:11311, /10.0.36.104:11311, /10.0.36.118:11311, /10.0.36.136:11311, /10.0.36.128:11311, /10.0.36.108:11311, /10.0.36.144:11311, /10.0.36.126:11311, /10.0.36.138:11311, /10.0.36.114:11311, /10.0.36.148:11311, /10.0.36.130:11311, /10.0.36.106:11311, /10.0.36.122:11311, /10.0.36.124:11311, /10.0.36.146:11311, /10.0.36.112:11311]
> INFO 02 sty 11:24:38 [main] org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New server added(/10.0.36.150:11311), adding to the pool.
> ...
> INFO 02 sty 11:24:38 [main] org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New server added(/10.0.36.112:11311), adding to the pool.
> {code}
> Next I stop one node (10.0.36.106 in this case) and receive another topology, but not that what I expected:
> {code}
> INFO 02 sty 11:26:39 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 - ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311]
> INFO 02 sty 11:26:39 [main] org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New server added(/10.0.36.102:11311), adding to the pool.
> INFO 02 sty 11:26:39 [main] org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New server added(/10.0.36.104:11311), adding to the pool.
> INFO 02 sty 11:26:39 [main] org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server not in cluster anymore(/10.0.36.148:11311), removing from the pool.
> ...
> INFO 02 sty 11:26:39 [main] org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server not in cluster anymore(/10.0.36.140:11311), removing from the pool.
> {code}
> The client is seeing only two nodes: the coordinator - 10.0.36.102 and the regular node - 10.0.36.104.
> Now I start a not running node back. And that's the reported topology:
> {code}
> INFO 02 sty 11:29:29 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 - ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311, /10.0.36.106:11311]
> INFO 02 sty 11:29:29 [main] org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New server added(/10.0.36.102:11311), adding to the pool.
> INFO 02 sty 11:29:29 [main] org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New server added(/10.0.36.104:11311), adding to the pool.
> INFO 02 sty 11:29:29 [main] org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New server added(/10.0.36.106:11311), adding to the pool.
> {code}
> The topology is still not valid. Whatever I do, I never receive the full cluster view, until the restart of all nodes.
> But the worse happens after stopping a coordinator. The client receives an empty topology:
> {code}
> INFO 02 sty 12:01:15 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 - ISPN004006: New topology: []
> INFO 02 sty 12:01:15 [main] org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server not in cluster anymore(/10.0.36.104:11311), removing from the pool.
> INFO 02 sty 12:01:15 [main] org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server not in cluster anymore(/10.0.36.102:11311), removing from the pool.
> INFO 02 sty 12:01:15 [main] org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server not in cluster anymore(/10.0.36.106:11311), removing from the pool.
> {code}
> Subsequent calls end with exceptions:
> {code}
> java.lang.IllegalStateException: We should not reach here!
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:78)
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:216)
> at org.infinispan.CacheSupport.put(CacheSupport.java:52)
> ...
> {code}
> Unfortunately this not reliable behaviour of remote client stops me from using HotRod Server on production.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the infinispan-issues
mailing list