Michal Linhard created ISPN-1826: ------------------------------------ Summary: Hot Rod protocol fails to retrieve correct topology after node join Key: ISPN-1826 URL: https://issues.jboss.org/browse/ISPN-1826 Project: Infinispan Issue Type: Bug Affects Versions: 5.1.0.FINAL Reporter: Michal Linhard Assignee: Manik Surtani Another resilience test: 4 nodes node2 goes down and up again: http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j... Test timeline goes like this: 2012-02-01 04:39:21,993: 20 hotrod clients started 2012-02-01 04:42:21,995: node02 killed 2012-02-01 04:45:27,950: node02 up again Endpoints: 172.18.1.1:11222 - node01 172.18.1.3:11222 - node02 172.18.1.5:11222 - node03 172.18.1.7:11222 - node04 in the client side log http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j... http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j... some warnings about connection refused and then node02 being removed from the pool by Timer thread: {code} 2012-02-01 04:43:15,608 254109 INFO [org.infinispan.client.hotrod.impl.protocol.Codec11] (Timer-1:) ISPN004006: New topology: [/172.18.1.1:11222, /172.18.1.7:11222, /172.18.1.5:11222] 2012-02-01 04:43:15,609 254110 INFO [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Timer-1:) ISPN004014: New server added(/172.18.1.1:11222), adding to the pool. 2012-02-01 04:43:15,610 254111 INFO [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Timer-1:) ISPN004014: New server added(/172.18.1.7:11222), adding to the pool. 2012-02-01 04:43:15,611 254112 INFO [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Timer-1:) ISPN004014: New server added(/172.18.1.5:11222), adding to the pool. 2012-02-01 04:43:15,612 254113 INFO [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Timer-1:) ISPN004016: Server not in cluster anymore(/172.18.1.3:11222), removing from the pool. {code} Log of Client-19 {code} 2012-02-01 04:39:20,962 19437 DEBUG [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) Statically configured servers: [/172.18.1.7:11222, /172.18.1.5:11222, /172.18.1.3:11222, /172.18.1.1:11222] 2012-02-01 04:39:20,962 19437 DEBUG [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) Load balancer class: org.infinispan.client.hotrod.impl.transport.tcp.RoundRobinBalancingStrategy 2012-02-01 04:39:20,963 19438 DEBUG [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) Tcp no delay = true; client socket timeout = 120000 ms; connect timeout = 120000 ms 2012-02-01 04:39:20,996 19471 INFO [org.infinispan.client.hotrod.impl.protocol.Codec11] (Client-19:) ISPN004006: New topology: [/172.18.1.1:11222, /172.18.1.3:11222, /172.18.1.7:11222, /172.18.1.5:11222] 2012-02-01 04:39:20,997 19472 INFO [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.1:11222), adding to the pool. 2012-02-01 04:39:20,998 19473 INFO [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.3:11222), adding to the pool. 2012-02-01 04:39:20,999 19474 INFO [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.7:11222), adding to the pool. 2012-02-01 04:39:21,000 19475 INFO [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.5:11222), adding to the pool. 2012-02-01 04:45:26,202 384677 INFO [org.infinispan.client.hotrod.impl.protocol.Codec11] (Client-19:) ISPN004006: New topology: [/172.18.1.1:11222, /172.18.1.7:11222, /172.18.1.5:11222] 2012-02-01 04:45:26,206 384681 INFO [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.1:11222), adding to the pool. 2012-02-01 04:45:26,207 384682 INFO [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.7:11222), adding to the pool. 2012-02-01 04:45:26,208 384683 INFO [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (Client-19:) ISPN004014: New server added(/172.18.1.5:11222), adding to the pool. {code} it seems like they get new topology update but without newly started node02 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira