6:49 a.m.
Topology view management in Hotrod Server is not reliable
---------------------------------------------------------
Key: ISPN-1654
URL: https://issues.jboss.org/browse/ISPN-1654
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 5.1.0.CR1
Reporter: Jacek Gerbszt
Assignee: Manik Surtani
Priority: Blocker
There is a problem with management of cluster topology view in address cache. You can see
that by using remote client - there is the only way I know to see what is inside the
address cache.
When I restart the whole cluster and make a call from remote client, I receive full
cluster topology (25 nodes):
{code}
INFO 02 sty 11:24:38 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.150:11311, /10.0.36.134:11311, /10.0.36.102:11311,
/10.0.36.110:11311, /10.0.36.142:11311, /10.0.36.140:11311, /10.0.36.132:11311,
/10.0.36.120:11311, /10.0.36.116:11311, /10.0.36.104:11311, /10.0.36.118:11311,
/10.0.36.136:11311, /10.0.36.128:11311, /10.0.36.108:11311, /10.0.36.144:11311,
/10.0.36.126:11311, /10.0.36.138:11311, /10.0.36.114:11311, /10.0.36.148:11311,
/10.0.36.130:11311, /10.0.36.106:11311, /10.0.36.122:11311, /10.0.36.124:11311,
/10.0.36.146:11311, /10.0.36.112:11311]
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.150:11311), adding to the pool.
...
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.112:11311), adding to the pool.
{code}
Next I stop one node (10.0.36.106 in this case) and receive another topology, but not that
what I expected:
{code}
INFO 02 sty 11:26:39 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311]
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.148:11311), removing from the pool.
...
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.140:11311), removing from the pool.
{code}
The client is seeing only two nodes: the coordinator - 10.0.36.102 and the regular node -
10.0.36.104.
Now I start a not running node back. And that's the reported topology:
{code}
INFO 02 sty 11:29:29 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311, /10.0.36.106:11311]
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.106:11311), adding to the pool.
{code}
The topology is still not valid. Whatever I do, I never receive the full cluster view,
until the restart of all nodes.
But the worse happens after stopping a coordinator. The client receives an empty
topology:
{code}
INFO 02 sty 12:01:15 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: []
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.104:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.102:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.106:11311), removing from the pool.
{code}
Subsequent calls end with exceptions:
{code}
java.lang.IllegalStateException: We should not reach here!
at
org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:78)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:216)
at org.infinispan.CacheSupport.put(CacheSupport.java:52)
...
{code}
Unfortunately this not reliable behaviour of remote client stops me from using HotRod
Server on production.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:10 a.m.
[
https://issues.jboss.org/browse/ISPN-1654?page=com.atlassian.jira.plugin....
]
Galder Zamarreño commented on ISPN-1654:
----------------------------------------
@Jacek, thanks for the report. Do you have any logs of the servers so that we can try to
narrow down the issue? In particular, I'd like to have a look at one log file from
either .104 or .102 and another log from one of the other machines.
Topology view management in Hotrod Server is not reliable
---------------------------------------------------------
Key: ISPN-1654
URL: https://issues.jboss.org/browse/ISPN-1654
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 5.1.0.CR1
Reporter: Jacek Gerbszt
Assignee: Manik Surtani
Priority: Blocker
There is a problem with management of cluster topology view in address cache. You can see
that by using remote client - there is the only way I know to see what is inside the
address cache.
When I restart the whole cluster and make a call from remote client, I receive full
cluster topology (25 nodes):
{code}
INFO 02 sty 11:24:38 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.150:11311, /10.0.36.134:11311, /10.0.36.102:11311,
/10.0.36.110:11311, /10.0.36.142:11311, /10.0.36.140:11311, /10.0.36.132:11311,
/10.0.36.120:11311, /10.0.36.116:11311, /10.0.36.104:11311, /10.0.36.118:11311,
/10.0.36.136:11311, /10.0.36.128:11311, /10.0.36.108:11311, /10.0.36.144:11311,
/10.0.36.126:11311, /10.0.36.138:11311, /10.0.36.114:11311, /10.0.36.148:11311,
/10.0.36.130:11311, /10.0.36.106:11311, /10.0.36.122:11311, /10.0.36.124:11311,
/10.0.36.146:11311, /10.0.36.112:11311]
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.150:11311), adding to the pool.
...
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.112:11311), adding to the pool.
{code}
Next I stop one node (10.0.36.106 in this case) and receive another topology, but not
that what I expected:
{code}
INFO 02 sty 11:26:39 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311]
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.148:11311), removing from the pool.
...
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.140:11311), removing from the pool.
{code}
The client is seeing only two nodes: the coordinator - 10.0.36.102 and the regular node -
10.0.36.104.
Now I start a not running node back. And that's the reported topology:
{code}
INFO 02 sty 11:29:29 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311, /10.0.36.106:11311]
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.106:11311), adding to the pool.
{code}
The topology is still not valid. Whatever I do, I never receive the full cluster view,
until the restart of all nodes.
But the worse happens after stopping a coordinator. The client receives an empty
topology:
{code}
INFO 02 sty 12:01:15 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: []
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.104:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.102:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.106:11311), removing from the pool.
{code}
Subsequent calls end with exceptions:
{code}
java.lang.IllegalStateException: We should not reach here!
at
org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:78)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:216)
at org.infinispan.CacheSupport.put(CacheSupport.java:52)
...
{code}
Unfortunately this not reliable behaviour of remote client stops me from using HotRod
Server on production.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:12 a.m.
[
https://issues.jboss.org/browse/ISPN-1654?page=com.atlassian.jira.plugin....
]
Galder Zamarreño reassigned ISPN-1654:
--------------------------------------
Assignee: Galder Zamarreño (was: Manik Surtani)
Topology view management in Hotrod Server is not reliable
---------------------------------------------------------
Key: ISPN-1654
URL: https://issues.jboss.org/browse/ISPN-1654
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 5.1.0.CR1
Reporter: Jacek Gerbszt
Assignee: Galder Zamarreño
Priority: Blocker
There is a problem with management of cluster topology view in address cache. You can see
that by using remote client - there is the only way I know to see what is inside the
address cache.
When I restart the whole cluster and make a call from remote client, I receive full
cluster topology (25 nodes):
{code}
INFO 02 sty 11:24:38 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.150:11311, /10.0.36.134:11311, /10.0.36.102:11311,
/10.0.36.110:11311, /10.0.36.142:11311, /10.0.36.140:11311, /10.0.36.132:11311,
/10.0.36.120:11311, /10.0.36.116:11311, /10.0.36.104:11311, /10.0.36.118:11311,
/10.0.36.136:11311, /10.0.36.128:11311, /10.0.36.108:11311, /10.0.36.144:11311,
/10.0.36.126:11311, /10.0.36.138:11311, /10.0.36.114:11311, /10.0.36.148:11311,
/10.0.36.130:11311, /10.0.36.106:11311, /10.0.36.122:11311, /10.0.36.124:11311,
/10.0.36.146:11311, /10.0.36.112:11311]
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.150:11311), adding to the pool.
...
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.112:11311), adding to the pool.
{code}
Next I stop one node (10.0.36.106 in this case) and receive another topology, but not
that what I expected:
{code}
INFO 02 sty 11:26:39 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311]
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.148:11311), removing from the pool.
...
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.140:11311), removing from the pool.
{code}
The client is seeing only two nodes: the coordinator - 10.0.36.102 and the regular node -
10.0.36.104.
Now I start a not running node back. And that's the reported topology:
{code}
INFO 02 sty 11:29:29 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311, /10.0.36.106:11311]
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.106:11311), adding to the pool.
{code}
The topology is still not valid. Whatever I do, I never receive the full cluster view,
until the restart of all nodes.
But the worse happens after stopping a coordinator. The client receives an empty
topology:
{code}
INFO 02 sty 12:01:15 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: []
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.104:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.102:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.106:11311), removing from the pool.
{code}
Subsequent calls end with exceptions:
{code}
java.lang.IllegalStateException: We should not reach here!
at
org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:78)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:216)
at org.infinispan.CacheSupport.put(CacheSupport.java:52)
...
{code}
Unfortunately this not reliable behaviour of remote client stops me from using HotRod
Server on production.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:41 a.m.
[
https://issues.jboss.org/browse/ISPN-1654?page=com.atlassian.jira.plugin....
]
Galder Zamarreño updated ISPN-1654:
-----------------------------------
Fix Version/s: 5.1.0.CR3
Topology view management in Hotrod Server is not reliable
---------------------------------------------------------
Key: ISPN-1654
URL: https://issues.jboss.org/browse/ISPN-1654
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 5.1.0.CR1
Reporter: Jacek Gerbszt
Assignee: Galder Zamarreño
Priority: Blocker
Fix For: 5.1.0.CR3
There is a problem with management of cluster topology view in address cache. You can see
that by using remote client - there is the only way I know to see what is inside the
address cache.
When I restart the whole cluster and make a call from remote client, I receive full
cluster topology (25 nodes):
{code}
INFO 02 sty 11:24:38 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.150:11311, /10.0.36.134:11311, /10.0.36.102:11311,
/10.0.36.110:11311, /10.0.36.142:11311, /10.0.36.140:11311, /10.0.36.132:11311,
/10.0.36.120:11311, /10.0.36.116:11311, /10.0.36.104:11311, /10.0.36.118:11311,
/10.0.36.136:11311, /10.0.36.128:11311, /10.0.36.108:11311, /10.0.36.144:11311,
/10.0.36.126:11311, /10.0.36.138:11311, /10.0.36.114:11311, /10.0.36.148:11311,
/10.0.36.130:11311, /10.0.36.106:11311, /10.0.36.122:11311, /10.0.36.124:11311,
/10.0.36.146:11311, /10.0.36.112:11311]
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.150:11311), adding to the pool.
...
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.112:11311), adding to the pool.
{code}
Next I stop one node (10.0.36.106 in this case) and receive another topology, but not
that what I expected:
{code}
INFO 02 sty 11:26:39 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311]
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.148:11311), removing from the pool.
...
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.140:11311), removing from the pool.
{code}
The client is seeing only two nodes: the coordinator - 10.0.36.102 and the regular node -
10.0.36.104.
Now I start a not running node back. And that's the reported topology:
{code}
INFO 02 sty 11:29:29 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311, /10.0.36.106:11311]
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.106:11311), adding to the pool.
{code}
The topology is still not valid. Whatever I do, I never receive the full cluster view,
until the restart of all nodes.
But the worse happens after stopping a coordinator. The client receives an empty
topology:
{code}
INFO 02 sty 12:01:15 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: []
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.104:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.102:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.106:11311), removing from the pool.
{code}
Subsequent calls end with exceptions:
{code}
java.lang.IllegalStateException: We should not reach here!
at
org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:78)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:216)
at org.infinispan.CacheSupport.put(CacheSupport.java:52)
...
{code}
Unfortunately this not reliable behaviour of remote client stops me from using HotRod
Server on production.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:41 a.m.
[
https://issues.jboss.org/browse/ISPN-1654?page=com.atlassian.jira.plugin....
]
Galder Zamarreño commented on ISPN-1654:
----------------------------------------
Btw, I'd suggest you give it a go to 5.1.0.CR2 too.
Topology view management in Hotrod Server is not reliable
---------------------------------------------------------
Key: ISPN-1654
URL: https://issues.jboss.org/browse/ISPN-1654
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 5.1.0.CR1
Reporter: Jacek Gerbszt
Assignee: Galder Zamarreño
Priority: Blocker
Fix For: 5.1.0.CR3
There is a problem with management of cluster topology view in address cache. You can see
that by using remote client - there is the only way I know to see what is inside the
address cache.
When I restart the whole cluster and make a call from remote client, I receive full
cluster topology (25 nodes):
{code}
INFO 02 sty 11:24:38 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.150:11311, /10.0.36.134:11311, /10.0.36.102:11311,
/10.0.36.110:11311, /10.0.36.142:11311, /10.0.36.140:11311, /10.0.36.132:11311,
/10.0.36.120:11311, /10.0.36.116:11311, /10.0.36.104:11311, /10.0.36.118:11311,
/10.0.36.136:11311, /10.0.36.128:11311, /10.0.36.108:11311, /10.0.36.144:11311,
/10.0.36.126:11311, /10.0.36.138:11311, /10.0.36.114:11311, /10.0.36.148:11311,
/10.0.36.130:11311, /10.0.36.106:11311, /10.0.36.122:11311, /10.0.36.124:11311,
/10.0.36.146:11311, /10.0.36.112:11311]
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.150:11311), adding to the pool.
...
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.112:11311), adding to the pool.
{code}
Next I stop one node (10.0.36.106 in this case) and receive another topology, but not
that what I expected:
{code}
INFO 02 sty 11:26:39 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311]
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.148:11311), removing from the pool.
...
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.140:11311), removing from the pool.
{code}
The client is seeing only two nodes: the coordinator - 10.0.36.102 and the regular node -
10.0.36.104.
Now I start a not running node back. And that's the reported topology:
{code}
INFO 02 sty 11:29:29 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311, /10.0.36.106:11311]
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.106:11311), adding to the pool.
{code}
The topology is still not valid. Whatever I do, I never receive the full cluster view,
until the restart of all nodes.
But the worse happens after stopping a coordinator. The client receives an empty
topology:
{code}
INFO 02 sty 12:01:15 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: []
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.104:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.102:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.106:11311), removing from the pool.
{code}
Subsequent calls end with exceptions:
{code}
java.lang.IllegalStateException: We should not reach here!
at
org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:78)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:216)
at org.infinispan.CacheSupport.put(CacheSupport.java:52)
...
{code}
Unfortunately this not reliable behaviour of remote client stops me from using HotRod
Server on production.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:21 a.m.
[
https://issues.jboss.org/browse/ISPN-1654?page=com.atlassian.jira.plugin....
]
Jacek Gerbszt updated ISPN-1654:
--------------------------------
Attachment: client.log.gz
front-21.log.gz
front-24.log.gz
front-21 - .102
front-22 - .104
front-23 - .106
front-24 = .108
Topology view management in Hotrod Server is not reliable
---------------------------------------------------------
Key: ISPN-1654
URL: https://issues.jboss.org/browse/ISPN-1654
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 5.1.0.CR1
Reporter: Jacek Gerbszt
Assignee: Galder Zamarreño
Priority: Blocker
Fix For: 5.1.0.CR3
Attachments: client.log.gz, front-21.log.gz, front-24.log.gz
There is a problem with management of cluster topology view in address cache. You can see
that by using remote client - there is the only way I know to see what is inside the
address cache.
When I restart the whole cluster and make a call from remote client, I receive full
cluster topology (25 nodes):
{code}
INFO 02 sty 11:24:38 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.150:11311, /10.0.36.134:11311, /10.0.36.102:11311,
/10.0.36.110:11311, /10.0.36.142:11311, /10.0.36.140:11311, /10.0.36.132:11311,
/10.0.36.120:11311, /10.0.36.116:11311, /10.0.36.104:11311, /10.0.36.118:11311,
/10.0.36.136:11311, /10.0.36.128:11311, /10.0.36.108:11311, /10.0.36.144:11311,
/10.0.36.126:11311, /10.0.36.138:11311, /10.0.36.114:11311, /10.0.36.148:11311,
/10.0.36.130:11311, /10.0.36.106:11311, /10.0.36.122:11311, /10.0.36.124:11311,
/10.0.36.146:11311, /10.0.36.112:11311]
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.150:11311), adding to the pool.
...
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.112:11311), adding to the pool.
{code}
Next I stop one node (10.0.36.106 in this case) and receive another topology, but not
that what I expected:
{code}
INFO 02 sty 11:26:39 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311]
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.148:11311), removing from the pool.
...
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.140:11311), removing from the pool.
{code}
The client is seeing only two nodes: the coordinator - 10.0.36.102 and the regular node -
10.0.36.104.
Now I start a not running node back. And that's the reported topology:
{code}
INFO 02 sty 11:29:29 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311, /10.0.36.106:11311]
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.106:11311), adding to the pool.
{code}
The topology is still not valid. Whatever I do, I never receive the full cluster view,
until the restart of all nodes.
But the worse happens after stopping a coordinator. The client receives an empty
topology:
{code}
INFO 02 sty 12:01:15 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: []
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.104:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.102:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.106:11311), removing from the pool.
{code}
Subsequent calls end with exceptions:
{code}
java.lang.IllegalStateException: We should not reach here!
at
org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:78)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:216)
at org.infinispan.CacheSupport.put(CacheSupport.java:52)
...
{code}
Unfortunately this not reliable behaviour of remote client stops me from using HotRod
Server on production.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:27 a.m.
[
https://issues.jboss.org/browse/ISPN-1654?page=com.atlassian.jira.plugin....
]
Jacek Gerbszt commented on ISPN-1654:
-------------------------------------
I've tried to install CR2, but it didn't worked for me. I don't know why, but
I couldn't manage to join the cluster. It needs more investigation.
Topology view management in Hotrod Server is not reliable
---------------------------------------------------------
Key: ISPN-1654
URL: https://issues.jboss.org/browse/ISPN-1654
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 5.1.0.CR1
Reporter: Jacek Gerbszt
Assignee: Galder Zamarreño
Priority: Blocker
Fix For: 5.1.0.CR3
Attachments: client.log.gz, front-21.log.gz, front-24.log.gz
There is a problem with management of cluster topology view in address cache. You can see
that by using remote client - there is the only way I know to see what is inside the
address cache.
When I restart the whole cluster and make a call from remote client, I receive full
cluster topology (25 nodes):
{code}
INFO 02 sty 11:24:38 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.150:11311, /10.0.36.134:11311, /10.0.36.102:11311,
/10.0.36.110:11311, /10.0.36.142:11311, /10.0.36.140:11311, /10.0.36.132:11311,
/10.0.36.120:11311, /10.0.36.116:11311, /10.0.36.104:11311, /10.0.36.118:11311,
/10.0.36.136:11311, /10.0.36.128:11311, /10.0.36.108:11311, /10.0.36.144:11311,
/10.0.36.126:11311, /10.0.36.138:11311, /10.0.36.114:11311, /10.0.36.148:11311,
/10.0.36.130:11311, /10.0.36.106:11311, /10.0.36.122:11311, /10.0.36.124:11311,
/10.0.36.146:11311, /10.0.36.112:11311]
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.150:11311), adding to the pool.
...
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.112:11311), adding to the pool.
{code}
Next I stop one node (10.0.36.106 in this case) and receive another topology, but not
that what I expected:
{code}
INFO 02 sty 11:26:39 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311]
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.148:11311), removing from the pool.
...
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.140:11311), removing from the pool.
{code}
The client is seeing only two nodes: the coordinator - 10.0.36.102 and the regular node -
10.0.36.104.
Now I start a not running node back. And that's the reported topology:
{code}
INFO 02 sty 11:29:29 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311, /10.0.36.106:11311]
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.106:11311), adding to the pool.
{code}
The topology is still not valid. Whatever I do, I never receive the full cluster view,
until the restart of all nodes.
But the worse happens after stopping a coordinator. The client receives an empty
topology:
{code}
INFO 02 sty 12:01:15 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: []
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.104:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.102:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.106:11311), removing from the pool.
{code}
Subsequent calls end with exceptions:
{code}
java.lang.IllegalStateException: We should not reach here!
at
org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:78)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:216)
at org.infinispan.CacheSupport.put(CacheSupport.java:52)
...
{code}
Unfortunately this not reliable behaviour of remote client stops me from using HotRod
Server on production.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:51 a.m.
[
https://issues.jboss.org/browse/ISPN-1654?page=com.atlassian.jira.plugin....
]
Galder Zamarreño commented on ISPN-1654:
----------------------------------------
Side note, this and http://community.jboss.org/message/644169#644169 could be related.
Topology view management in Hotrod Server is not reliable
---------------------------------------------------------
Key: ISPN-1654
URL: https://issues.jboss.org/browse/ISPN-1654
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 5.1.0.CR1
Reporter: Jacek Gerbszt
Assignee: Galder Zamarreño
Priority: Blocker
Fix For: 5.1.0.CR3
Attachments: client.log.gz, front-21.log.gz, front-24.log.gz
There is a problem with management of cluster topology view in address cache. You can see
that by using remote client - there is the only way I know to see what is inside the
address cache.
When I restart the whole cluster and make a call from remote client, I receive full
cluster topology (25 nodes):
{code}
INFO 02 sty 11:24:38 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.150:11311, /10.0.36.134:11311, /10.0.36.102:11311,
/10.0.36.110:11311, /10.0.36.142:11311, /10.0.36.140:11311, /10.0.36.132:11311,
/10.0.36.120:11311, /10.0.36.116:11311, /10.0.36.104:11311, /10.0.36.118:11311,
/10.0.36.136:11311, /10.0.36.128:11311, /10.0.36.108:11311, /10.0.36.144:11311,
/10.0.36.126:11311, /10.0.36.138:11311, /10.0.36.114:11311, /10.0.36.148:11311,
/10.0.36.130:11311, /10.0.36.106:11311, /10.0.36.122:11311, /10.0.36.124:11311,
/10.0.36.146:11311, /10.0.36.112:11311]
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.150:11311), adding to the pool.
...
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.112:11311), adding to the pool.
{code}
Next I stop one node (10.0.36.106 in this case) and receive another topology, but not
that what I expected:
{code}
INFO 02 sty 11:26:39 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311]
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.148:11311), removing from the pool.
...
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.140:11311), removing from the pool.
{code}
The client is seeing only two nodes: the coordinator - 10.0.36.102 and the regular node -
10.0.36.104.
Now I start a not running node back. And that's the reported topology:
{code}
INFO 02 sty 11:29:29 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311, /10.0.36.106:11311]
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.106:11311), adding to the pool.
{code}
The topology is still not valid. Whatever I do, I never receive the full cluster view,
until the restart of all nodes.
But the worse happens after stopping a coordinator. The client receives an empty
topology:
{code}
INFO 02 sty 12:01:15 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: []
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.104:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.102:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.106:11311), removing from the pool.
{code}
Subsequent calls end with exceptions:
{code}
java.lang.IllegalStateException: We should not reach here!
at
org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:78)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:216)
at org.infinispan.CacheSupport.put(CacheSupport.java:52)
...
{code}
Unfortunately this not reliable behaviour of remote client stops me from using HotRod
Server on production.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:50 a.m.
[
https://issues.jboss.org/browse/ISPN-1654?page=com.atlassian.jira.plugin....
]
SBS JIRA Integration updated ISPN-1654:
---------------------------------------
Forum Reference: http://community.jboss.org/message/643246#643246
Topology view management in Hotrod Server is not reliable
---------------------------------------------------------
Key: ISPN-1654
URL: https://issues.jboss.org/browse/ISPN-1654
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 5.1.0.CR1
Reporter: Jacek Gerbszt
Assignee: Galder Zamarreño
Priority: Blocker
Fix For: 5.1.0.CR3
Attachments: client.log.gz, front-21.log.gz, front-24.log.gz
There is a problem with management of cluster topology view in address cache. You can see
that by using remote client - there is the only way I know to see what is inside the
address cache.
When I restart the whole cluster and make a call from remote client, I receive full
cluster topology (25 nodes):
{code}
INFO 02 sty 11:24:38 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.150:11311, /10.0.36.134:11311, /10.0.36.102:11311,
/10.0.36.110:11311, /10.0.36.142:11311, /10.0.36.140:11311, /10.0.36.132:11311,
/10.0.36.120:11311, /10.0.36.116:11311, /10.0.36.104:11311, /10.0.36.118:11311,
/10.0.36.136:11311, /10.0.36.128:11311, /10.0.36.108:11311, /10.0.36.144:11311,
/10.0.36.126:11311, /10.0.36.138:11311, /10.0.36.114:11311, /10.0.36.148:11311,
/10.0.36.130:11311, /10.0.36.106:11311, /10.0.36.122:11311, /10.0.36.124:11311,
/10.0.36.146:11311, /10.0.36.112:11311]
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.150:11311), adding to the pool.
...
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.112:11311), adding to the pool.
{code}
Next I stop one node (10.0.36.106 in this case) and receive another topology, but not
that what I expected:
{code}
INFO 02 sty 11:26:39 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311]
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.148:11311), removing from the pool.
...
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.140:11311), removing from the pool.
{code}
The client is seeing only two nodes: the coordinator - 10.0.36.102 and the regular node -
10.0.36.104.
Now I start a not running node back. And that's the reported topology:
{code}
INFO 02 sty 11:29:29 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311, /10.0.36.106:11311]
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.106:11311), adding to the pool.
{code}
The topology is still not valid. Whatever I do, I never receive the full cluster view,
until the restart of all nodes.
But the worse happens after stopping a coordinator. The client receives an empty
topology:
{code}
INFO 02 sty 12:01:15 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: []
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.104:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.102:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.106:11311), removing from the pool.
{code}
Subsequent calls end with exceptions:
{code}
java.lang.IllegalStateException: We should not reach here!
at
org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:78)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:216)
at org.infinispan.CacheSupport.put(CacheSupport.java:52)
...
{code}
Unfortunately this not reliable behaviour of remote client stops me from using HotRod
Server on production.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:52 a.m.
[
https://issues.jboss.org/browse/ISPN-1654?page=com.atlassian.jira.plugin....
]
Galder Zamarreño commented on ISPN-1654:
----------------------------------------
The logs attached to the forum entry appear to show an issue with the filtering of
members:
{code}2012-01-03 10:13:26,026 TRACE (notification-thread-2)
[org.infinispan.server.hotrod.HotRodServer] View change received on coordinator:
EventImpl{type=VIEW_CHANGED, newMembers=[n1-16004, n2-22348], oldMembers=[n1-16004,
n3-34263, n2-22348], localAddress=n1-16004, viewId=3,
subgroupsMerged=null, mergeView=false}
2012-01-03 10:13:26,027 TRACE (notification-thread-2)
[org.infinispan.server.hotrod.HotRodServer] Somone left the cluster, oldMembers=non-empty
iterator newMembers=empty iterator
2012-01-03 10:13:26,028 TRACE (notification-thread-2)
[org.infinispan.server.hotrod.HotRodServer] Remove n3-34263 from address cache
2012-01-03 10:13:26,064 TRACE (notification-thread-2)
[org.infinispan.server.hotrod.HotRodServer] Remove n2-22348 from address cache{code}
n2 should have not been removed, investigating
Topology view management in Hotrod Server is not reliable
---------------------------------------------------------
Key: ISPN-1654
URL: https://issues.jboss.org/browse/ISPN-1654
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 5.1.0.CR1
Reporter: Jacek Gerbszt
Assignee: Galder Zamarreño
Priority: Blocker
Fix For: 5.1.0.CR3
Attachments: client.log.gz, front-21.log.gz, front-24.log.gz
There is a problem with management of cluster topology view in address cache. You can see
that by using remote client - there is the only way I know to see what is inside the
address cache.
When I restart the whole cluster and make a call from remote client, I receive full
cluster topology (25 nodes):
{code}
INFO 02 sty 11:24:38 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.150:11311, /10.0.36.134:11311, /10.0.36.102:11311,
/10.0.36.110:11311, /10.0.36.142:11311, /10.0.36.140:11311, /10.0.36.132:11311,
/10.0.36.120:11311, /10.0.36.116:11311, /10.0.36.104:11311, /10.0.36.118:11311,
/10.0.36.136:11311, /10.0.36.128:11311, /10.0.36.108:11311, /10.0.36.144:11311,
/10.0.36.126:11311, /10.0.36.138:11311, /10.0.36.114:11311, /10.0.36.148:11311,
/10.0.36.130:11311, /10.0.36.106:11311, /10.0.36.122:11311, /10.0.36.124:11311,
/10.0.36.146:11311, /10.0.36.112:11311]
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.150:11311), adding to the pool.
...
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.112:11311), adding to the pool.
{code}
Next I stop one node (10.0.36.106 in this case) and receive another topology, but not
that what I expected:
{code}
INFO 02 sty 11:26:39 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311]
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.148:11311), removing from the pool.
...
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.140:11311), removing from the pool.
{code}
The client is seeing only two nodes: the coordinator - 10.0.36.102 and the regular node -
10.0.36.104.
Now I start a not running node back. And that's the reported topology:
{code}
INFO 02 sty 11:29:29 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311, /10.0.36.106:11311]
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.106:11311), adding to the pool.
{code}
The topology is still not valid. Whatever I do, I never receive the full cluster view,
until the restart of all nodes.
But the worse happens after stopping a coordinator. The client receives an empty
topology:
{code}
INFO 02 sty 12:01:15 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: []
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.104:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.102:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.106:11311), removing from the pool.
{code}
Subsequent calls end with exceptions:
{code}
java.lang.IllegalStateException: We should not reach here!
at
org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:78)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:216)
at org.infinispan.CacheSupport.put(CacheSupport.java:52)
...
{code}
Unfortunately this not reliable behaviour of remote client stops me from using HotRod
Server on production.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:10 a.m.
[
https://issues.jboss.org/browse/ISPN-1654?page=com.atlassian.jira.plugin....
]
Work on ISPN-1654 started by Galder Zamarreño.
Topology view management in Hotrod Server is not reliable
---------------------------------------------------------
Key: ISPN-1654
URL: https://issues.jboss.org/browse/ISPN-1654
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 5.1.0.CR1
Reporter: Jacek Gerbszt
Assignee: Galder Zamarreño
Priority: Blocker
Fix For: 5.1.0.CR3
Attachments: client.log.gz, front-21.log.gz, front-24.log.gz
There is a problem with management of cluster topology view in address cache. You can see
that by using remote client - there is the only way I know to see what is inside the
address cache.
When I restart the whole cluster and make a call from remote client, I receive full
cluster topology (25 nodes):
{code}
INFO 02 sty 11:24:38 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.150:11311, /10.0.36.134:11311, /10.0.36.102:11311,
/10.0.36.110:11311, /10.0.36.142:11311, /10.0.36.140:11311, /10.0.36.132:11311,
/10.0.36.120:11311, /10.0.36.116:11311, /10.0.36.104:11311, /10.0.36.118:11311,
/10.0.36.136:11311, /10.0.36.128:11311, /10.0.36.108:11311, /10.0.36.144:11311,
/10.0.36.126:11311, /10.0.36.138:11311, /10.0.36.114:11311, /10.0.36.148:11311,
/10.0.36.130:11311, /10.0.36.106:11311, /10.0.36.122:11311, /10.0.36.124:11311,
/10.0.36.146:11311, /10.0.36.112:11311]
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.150:11311), adding to the pool.
...
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.112:11311), adding to the pool.
{code}
Next I stop one node (10.0.36.106 in this case) and receive another topology, but not
that what I expected:
{code}
INFO 02 sty 11:26:39 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311]
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.148:11311), removing from the pool.
...
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.140:11311), removing from the pool.
{code}
The client is seeing only two nodes: the coordinator - 10.0.36.102 and the regular node -
10.0.36.104.
Now I start a not running node back. And that's the reported topology:
{code}
INFO 02 sty 11:29:29 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311, /10.0.36.106:11311]
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.106:11311), adding to the pool.
{code}
The topology is still not valid. Whatever I do, I never receive the full cluster view,
until the restart of all nodes.
But the worse happens after stopping a coordinator. The client receives an empty
topology:
{code}
INFO 02 sty 12:01:15 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: []
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.104:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.102:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.106:11311), removing from the pool.
{code}
Subsequent calls end with exceptions:
{code}
java.lang.IllegalStateException: We should not reach here!
at
org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:78)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:216)
at org.infinispan.CacheSupport.put(CacheSupport.java:52)
...
{code}
Unfortunately this not reliable behaviour of remote client stops me from using HotRod
Server on production.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:22 a.m.
[
https://issues.jboss.org/browse/ISPN-1654?page=com.atlassian.jira.plugin....
]
Galder Zamarreño commented on ISPN-1654:
----------------------------------------
Nice catch guys. It was a problem with the different order. I was using Scala iterators
which returns true to contains only if the element is present and it's in the same
position. The fact that in the old view n3 and n2 were swapped around was causing issues.
Switching to iterables works fine. This is just a problem for 5.1 beta/cr releases which
is when I changed this code.
Topology view management in Hotrod Server is not reliable
---------------------------------------------------------
Key: ISPN-1654
URL: https://issues.jboss.org/browse/ISPN-1654
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 5.1.0.CR1
Reporter: Jacek Gerbszt
Assignee: Galder Zamarreño
Priority: Blocker
Fix For: 5.1.0.CR3
Attachments: client.log.gz, front-21.log.gz, front-24.log.gz
There is a problem with management of cluster topology view in address cache. You can see
that by using remote client - there is the only way I know to see what is inside the
address cache.
When I restart the whole cluster and make a call from remote client, I receive full
cluster topology (25 nodes):
{code}
INFO 02 sty 11:24:38 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.150:11311, /10.0.36.134:11311, /10.0.36.102:11311,
/10.0.36.110:11311, /10.0.36.142:11311, /10.0.36.140:11311, /10.0.36.132:11311,
/10.0.36.120:11311, /10.0.36.116:11311, /10.0.36.104:11311, /10.0.36.118:11311,
/10.0.36.136:11311, /10.0.36.128:11311, /10.0.36.108:11311, /10.0.36.144:11311,
/10.0.36.126:11311, /10.0.36.138:11311, /10.0.36.114:11311, /10.0.36.148:11311,
/10.0.36.130:11311, /10.0.36.106:11311, /10.0.36.122:11311, /10.0.36.124:11311,
/10.0.36.146:11311, /10.0.36.112:11311]
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.150:11311), adding to the pool.
...
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.112:11311), adding to the pool.
{code}
Next I stop one node (10.0.36.106 in this case) and receive another topology, but not
that what I expected:
{code}
INFO 02 sty 11:26:39 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311]
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.148:11311), removing from the pool.
...
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.140:11311), removing from the pool.
{code}
The client is seeing only two nodes: the coordinator - 10.0.36.102 and the regular node -
10.0.36.104.
Now I start a not running node back. And that's the reported topology:
{code}
INFO 02 sty 11:29:29 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311, /10.0.36.106:11311]
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.106:11311), adding to the pool.
{code}
The topology is still not valid. Whatever I do, I never receive the full cluster view,
until the restart of all nodes.
But the worse happens after stopping a coordinator. The client receives an empty
topology:
{code}
INFO 02 sty 12:01:15 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: []
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.104:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.102:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.106:11311), removing from the pool.
{code}
Subsequent calls end with exceptions:
{code}
java.lang.IllegalStateException: We should not reach here!
at
org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:78)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:216)
at org.infinispan.CacheSupport.put(CacheSupport.java:52)
...
{code}
Unfortunately this not reliable behaviour of remote client stops me from using HotRod
Server on production.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:38 a.m.
[
https://issues.jboss.org/browse/ISPN-1654?page=com.atlassian.jira.plugin....
]
Galder Zamarreño updated ISPN-1654:
-----------------------------------
Status: Pull Request Sent (was: Coding In Progress)
Git Pull Request: https://github.com/infinispan/infinispan/pull/780
Topology view management in Hotrod Server is not reliable
---------------------------------------------------------
Key: ISPN-1654
URL: https://issues.jboss.org/browse/ISPN-1654
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 5.1.0.CR1
Reporter: Jacek Gerbszt
Assignee: Galder Zamarreño
Priority: Blocker
Fix For: 5.1.0.CR3
Attachments: client.log.gz, front-21.log.gz, front-24.log.gz
There is a problem with management of cluster topology view in address cache. You can see
that by using remote client - there is the only way I know to see what is inside the
address cache.
When I restart the whole cluster and make a call from remote client, I receive full
cluster topology (25 nodes):
{code}
INFO 02 sty 11:24:38 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.150:11311, /10.0.36.134:11311, /10.0.36.102:11311,
/10.0.36.110:11311, /10.0.36.142:11311, /10.0.36.140:11311, /10.0.36.132:11311,
/10.0.36.120:11311, /10.0.36.116:11311, /10.0.36.104:11311, /10.0.36.118:11311,
/10.0.36.136:11311, /10.0.36.128:11311, /10.0.36.108:11311, /10.0.36.144:11311,
/10.0.36.126:11311, /10.0.36.138:11311, /10.0.36.114:11311, /10.0.36.148:11311,
/10.0.36.130:11311, /10.0.36.106:11311, /10.0.36.122:11311, /10.0.36.124:11311,
/10.0.36.146:11311, /10.0.36.112:11311]
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.150:11311), adding to the pool.
...
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.112:11311), adding to the pool.
{code}
Next I stop one node (10.0.36.106 in this case) and receive another topology, but not
that what I expected:
{code}
INFO 02 sty 11:26:39 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311]
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.148:11311), removing from the pool.
...
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.140:11311), removing from the pool.
{code}
The client is seeing only two nodes: the coordinator - 10.0.36.102 and the regular node -
10.0.36.104.
Now I start a not running node back. And that's the reported topology:
{code}
INFO 02 sty 11:29:29 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311, /10.0.36.106:11311]
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.106:11311), adding to the pool.
{code}
The topology is still not valid. Whatever I do, I never receive the full cluster view,
until the restart of all nodes.
But the worse happens after stopping a coordinator. The client receives an empty
topology:
{code}
INFO 02 sty 12:01:15 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: []
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.104:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.102:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.106:11311), removing from the pool.
{code}
Subsequent calls end with exceptions:
{code}
java.lang.IllegalStateException: We should not reach here!
at
org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:78)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:216)
at org.infinispan.CacheSupport.put(CacheSupport.java:52)
...
{code}
Unfortunately this not reliable behaviour of remote client stops me from using HotRod
Server on production.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:09 a.m.
[
https://issues.jboss.org/browse/ISPN-1654?page=com.atlassian.jira.plugin....
]
Manik Surtani updated ISPN-1654:
--------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
Topology view management in Hotrod Server is not reliable
---------------------------------------------------------
Key: ISPN-1654
URL: https://issues.jboss.org/browse/ISPN-1654
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 5.1.0.CR1
Reporter: Jacek Gerbszt
Assignee: Galder Zamarreño
Priority: Blocker
Fix For: 5.1.0.CR3
Attachments: client.log.gz, front-21.log.gz, front-24.log.gz
There is a problem with management of cluster topology view in address cache. You can see
that by using remote client - there is the only way I know to see what is inside the
address cache.
When I restart the whole cluster and make a call from remote client, I receive full
cluster topology (25 nodes):
{code}
INFO 02 sty 11:24:38 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.150:11311, /10.0.36.134:11311, /10.0.36.102:11311,
/10.0.36.110:11311, /10.0.36.142:11311, /10.0.36.140:11311, /10.0.36.132:11311,
/10.0.36.120:11311, /10.0.36.116:11311, /10.0.36.104:11311, /10.0.36.118:11311,
/10.0.36.136:11311, /10.0.36.128:11311, /10.0.36.108:11311, /10.0.36.144:11311,
/10.0.36.126:11311, /10.0.36.138:11311, /10.0.36.114:11311, /10.0.36.148:11311,
/10.0.36.130:11311, /10.0.36.106:11311, /10.0.36.122:11311, /10.0.36.124:11311,
/10.0.36.146:11311, /10.0.36.112:11311]
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.150:11311), adding to the pool.
...
INFO 02 sty 11:24:38 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.112:11311), adding to the pool.
{code}
Next I stop one node (10.0.36.106 in this case) and receive another topology, but not
that what I expected:
{code}
INFO 02 sty 11:26:39 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311]
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.148:11311), removing from the pool.
...
INFO 02 sty 11:26:39 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.140:11311), removing from the pool.
{code}
The client is seeing only two nodes: the coordinator - 10.0.36.102 and the regular node -
10.0.36.104.
Now I start a not running node back. And that's the reported topology:
{code}
INFO 02 sty 11:29:29 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: [/10.0.36.102:11311, /10.0.36.104:11311, /10.0.36.106:11311]
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.102:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.104:11311), adding to the pool.
INFO 02 sty 11:29:29 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004014: New
server added(/10.0.36.106:11311), adding to the pool.
{code}
The topology is still not valid. Whatever I do, I never receive the full cluster view,
until the restart of all nodes.
But the worse happens after stopping a coordinator. The client receives an empty
topology:
{code}
INFO 02 sty 12:01:15 [main] org.infinispan.client.hotrod.impl.protocol.Codec11 -
ISPN004006: New topology: []
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.104:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.102:11311), removing from the pool.
INFO 02 sty 12:01:15 [main]
org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory - ISPN004016: Server
not in cluster anymore(/10.0.36.106:11311), removing from the pool.
{code}
Subsequent calls end with exceptions:
{code}
java.lang.IllegalStateException: We should not reach here!
at
org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:78)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:216)
at org.infinispan.CacheSupport.put(CacheSupport.java:52)
...
{code}
Unfortunately this not reliable behaviour of remote client stops me from using HotRod
Server on production.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
4740
days inactive
4743
days old
infinispan-issues@lists.jboss.org
13 comments
9 participants
participants (9)
-
Galder Zamarreño (Assigned) (JIRA) -
Galder Zamarreño (Commented) (JIRA)
-
Galder Zamarreño (Updated) (JIRA)
-
Galder Zamarreño (Work started) (JIRA)
-
Jacek Gerbszt (Commented) (JIRA)
-
Jacek Gerbszt (Created) (JIRA)
-
Jacek Gerbszt (Updated) (JIRA)
-
Manik Surtani (Updated) (JIRA)
-
SBS JIRA Integration (Updated) (JIRA)