[infinispan-issues] [JBoss JIRA] (ISPN-9220) Client listeners are sometimes re-registered twice after infinispan-server restart
Marek Posolda (JIRA)
issues at jboss.org
Wed May 30 04:04:00 EDT 2018
[ https://issues.jboss.org/browse/ISPN-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marek Posolda updated ISPN-9220:
--------------------------------
Steps to Reproduce:
I was not able to reliably reproduce why the client listener is re-registered twice without debugger. I've needed to use the debugger and this helps to reliable reproduce it in my environment.
1) Unzip infinispan-server 9.2.1.Final and run with:
{code}
cd infinispan-server-9.2.1.Final/bin/
./standalone.sh -c clustered.xml
{code}
2) Run this Java application from IDE in debug mode https://github.com/mposolda/misc/blob/master/ispn-client-listener/src/main/java/org/mposolda/ClientListenerFailoverTest.java . Make sure that infinispan version (Version of infinispan on client side) is 9.2.1.Final (property "ispn.version" in this file: https://github.com/mposolda/misc/blob/master/ispn-client-listener/pom.xml
After application is executed, you can see that items are added and client listener is triggered as expected:
{code}
May 30, 2018 9:53:12 AM org.mposolda.ClientListenerFailoverTest main
INFO: RemoteCache created
May 30, 2018 9:53:13 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-0
May 30, 2018 9:53:13 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-0
May 30, 2018 9:53:14 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-1
May 30, 2018 9:53:14 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-1
May 30, 2018 9:53:15 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-2
May 30, 2018 9:53:15 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-2
May 30, 2018 9:53:16 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-3
May 30, 2018 9:53:16 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-3
May 30, 2018 9:53:17 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-4
May 30, 2018 9:53:17 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-4
May 30, 2018 9:53:18 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-5
May 30, 2018 9:53:18 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-5
....
{code}
3) Add breakpoint to class org.infinispan.client.hotrod.impl.operations.AddClientListenerOperation to the beginning of method "execute(Channel channel)"
4) Restart infinispan-server
5) After server is restarted, the method AddClientListenerOperation.execute is invoked. Wait with breakpoint at least 10 seconds to reproduce the issue.
6) Continue with breakpoint and see that method is executed another time. Continue with breakpoint again.
7) Now I can see that clientListener is executed twice for each added item, which is NOT ok.
{code}
....
INFO: Adding key: key-189
May 30, 2018 9:56:23 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-189
May 30, 2018 9:56:23 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-189
May 30, 2018 9:56:24 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-190
May 30, 2018 9:56:24 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-190
May 30, 2018 9:56:24 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-190
May 30, 2018 9:56:25 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-191
May 30, 2018 9:56:25 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-191
May 30, 2018 9:56:25 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-191
May 30, 2018 9:56:26 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-192
May 30, 2018 9:56:26 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-192
May 30, 2018 9:56:26 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-192
...
{code}
Sorry for the brittle steps to reproduce with the need of debugger, but that was the only way to simulate it reliably. Without debugger, the listener is usually re-registered once (correct behaviour), but sometimes twice (incorrect behaviour), so I think some issue is here.
was:
I was not able to reliably reproduce why the client listener is re-registered twice without debugger. I've needed to use the debugger and this helps to reliable reproduce it in my environment.
1) Unzip infinispan-server 9.2.1.Final and run with:
{code}
cd infinispan-server-9.2.1.Final/bin/
./standalone.sh -c clustered.xml
{code}
2) Run this Java application from IDE in debug mode https://github.com/mposolda/misc/blob/master/ispn-client-listener/src/main/java/org/mposolda/ClientListenerFailoverTest.java . Make sure that infinispan version (Version of infinispan on client side) is 9.2.1.Final (property "ispn.version" in this file: https://github.com/mposolda/misc/blob/master/ispn-client-listener/pom.xml
After application is executed, you can see that items are added and client listener is triggered as expected:
{code}
May 30, 2018 9:53:12 AM org.mposolda.ClientListenerFailoverTest main
INFO: RemoteCache created
May 30, 2018 9:53:13 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-0
May 30, 2018 9:53:13 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-0
May 30, 2018 9:53:14 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-1
May 30, 2018 9:53:14 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-1
May 30, 2018 9:53:15 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-2
May 30, 2018 9:53:15 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-2
May 30, 2018 9:53:16 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-3
May 30, 2018 9:53:16 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-3
May 30, 2018 9:53:17 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-4
May 30, 2018 9:53:17 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-4
May 30, 2018 9:53:18 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-5
May 30, 2018 9:53:18 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-5
....
{code}
3) Add breakpoint to class org.infinispan.client.hotrod.impl.operations.AddClientListenerOperation to the beginning of method "execute(Channel channel)
4) Restart infinispan-server
5) After server is restarted, the method AddClientListenerOperation.execute is invoked. Wait with breakpoint at least 10 seconds to reproduce the issue.
6) Continue with breakpoint and see that method is executed another time. Continue with breakpoint again.
7) Now I can see that clientListener is executed twice for each added item, which is NOT ok.
{code}
....
INFO: Adding key: key-189
May 30, 2018 9:56:23 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-189
May 30, 2018 9:56:23 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-189
May 30, 2018 9:56:24 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-190
May 30, 2018 9:56:24 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-190
May 30, 2018 9:56:24 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-190
May 30, 2018 9:56:25 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-191
May 30, 2018 9:56:25 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-191
May 30, 2018 9:56:25 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-191
May 30, 2018 9:56:26 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-192
May 30, 2018 9:56:26 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-192
May 30, 2018 9:56:26 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-192
...
{code}
Sorry for the brittle steps to reproduce with the need of debugger, but that was the only way to simulate it reliably. Without debugger, the listener is usually re-registered once (correct behaviour), but sometimes twice (incorrect behaviour), so I think some issue is here.
> Client listeners are sometimes re-registered twice after infinispan-server restart
> ----------------------------------------------------------------------------------
>
> Key: ISPN-9220
> URL: https://issues.jboss.org/browse/ISPN-9220
> Project: Infinispan
> Issue Type: Bug
> Components: Hot Rod
> Affects Versions: 9.2.1.Final
> Reporter: Marek Posolda
>
> During test of summit, we saw some issue with client listeners. Setup was this:
> - Every site had one hotrod client (RHSSO), which used infinispan of version 8.2.8.Final and one infinispan-server, which I think was version 9.2.1.Final
> - Client (RH-SSO) had some client listener registered against remoteCache
> - After restart of infinispan-server, the RH-SSO was able to re-connect to it and CRUD the items to the remoteCache. But previously registered clientListeners were lost.
> Note that setup demo used just 1 infinispan-server on every site, so there were not any topology changes (no possibility of re-balance client listener to other node).
> I've tried to do simple reproducer for the scenario above and verified that with client 8.2.8.Final and server 9.2.1.Final, the client listener is not automatically re-registered after restart.
> BUT after upgrade client to 9.2.1.Final, re-registration worked fine and client listener was re-registered automatically. This is important as from long term perspective, we are upgrading to infinispan 9 on RH-SSO side. But I had the opposite issue, that sometimes the client listener is registered twice. See steps to reproduce for the details.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
More information about the infinispan-issues
mailing list