[JBoss JIRA] (ISPN-9220) Client listeners are sometimes re-registered twice after infinispan-server restart
by Marek Posolda (JIRA)
[ https://issues.jboss.org/browse/ISPN-9220?page=com.atlassian.jira.plugin.... ]
Marek Posolda updated ISPN-9220:
--------------------------------
Steps to Reproduce:
I was not able to reliably reproduce why the client listener is re-registered twice without debugger. I've needed to use the debugger and this helps to reliable reproduce it in my environment.
1) Unzip infinispan-server 9.2.1.Final and run with:
{code}
cd infinispan-server-9.2.1.Final/bin/
./standalone.sh -c clustered.xml
{code}
2) Run this Java application from IDE in debug mode https://github.com/mposolda/misc/blob/master/ispn-client-listener/src/mai... . Make sure that infinispan version (Version of infinispan on client side) is 9.2.1.Final (property "ispn.version" in this file: https://github.com/mposolda/misc/blob/master/ispn-client-listener/pom.xml
After application is executed, you can see that items are added and client listener is triggered as expected:
{code}
May 30, 2018 9:53:12 AM org.mposolda.ClientListenerFailoverTest main
INFO: RemoteCache created
May 30, 2018 9:53:13 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-0
May 30, 2018 9:53:13 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-0
May 30, 2018 9:53:14 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-1
May 30, 2018 9:53:14 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-1
May 30, 2018 9:53:15 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-2
May 30, 2018 9:53:15 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-2
May 30, 2018 9:53:16 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-3
May 30, 2018 9:53:16 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-3
May 30, 2018 9:53:17 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-4
May 30, 2018 9:53:17 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-4
May 30, 2018 9:53:18 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-5
May 30, 2018 9:53:18 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-5
....
{code}
3) Add breakpoint to class org.infinispan.client.hotrod.impl.operations.AddClientListenerOperation to the beginning of method "execute(Channel channel)"
4) Restart infinispan-server
5) After server is restarted, the method AddClientListenerOperation.execute is invoked. Wait with breakpoint at least 10 seconds to reproduce the issue.
6) Continue with breakpoint and see that method is executed another time. Continue with breakpoint again.
7) Now I can see that clientListener is executed twice for each added item, which is NOT ok.
{code}
....
INFO: Adding key: key-189
May 30, 2018 9:56:23 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-189
May 30, 2018 9:56:23 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-189
May 30, 2018 9:56:24 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-190
May 30, 2018 9:56:24 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-190
May 30, 2018 9:56:24 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-190
May 30, 2018 9:56:25 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-191
May 30, 2018 9:56:25 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-191
May 30, 2018 9:56:25 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-191
May 30, 2018 9:56:26 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-192
May 30, 2018 9:56:26 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-192
May 30, 2018 9:56:26 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-192
...
{code}
Sorry for the brittle steps to reproduce with the need of debugger, but that was the only way to simulate it reliably. Without debugger, the listener is usually re-registered once (correct behaviour), but sometimes twice (incorrect behaviour), so I think some issue is here.
was:
I was not able to reliably reproduce why the client listener is re-registered twice without debugger. I've needed to use the debugger and this helps to reliable reproduce it in my environment.
1) Unzip infinispan-server 9.2.1.Final and run with:
{code}
cd infinispan-server-9.2.1.Final/bin/
./standalone.sh -c clustered.xml
{code}
2) Run this Java application from IDE in debug mode https://github.com/mposolda/misc/blob/master/ispn-client-listener/src/mai... . Make sure that infinispan version (Version of infinispan on client side) is 9.2.1.Final (property "ispn.version" in this file: https://github.com/mposolda/misc/blob/master/ispn-client-listener/pom.xml
After application is executed, you can see that items are added and client listener is triggered as expected:
{code}
May 30, 2018 9:53:12 AM org.mposolda.ClientListenerFailoverTest main
INFO: RemoteCache created
May 30, 2018 9:53:13 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-0
May 30, 2018 9:53:13 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-0
May 30, 2018 9:53:14 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-1
May 30, 2018 9:53:14 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-1
May 30, 2018 9:53:15 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-2
May 30, 2018 9:53:15 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-2
May 30, 2018 9:53:16 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-3
May 30, 2018 9:53:16 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-3
May 30, 2018 9:53:17 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-4
May 30, 2018 9:53:17 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-4
May 30, 2018 9:53:18 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-5
May 30, 2018 9:53:18 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-5
....
{code}
3) Add breakpoint to class org.infinispan.client.hotrod.impl.operations.AddClientListenerOperation to the beginning of method "execute(Channel channel)
4) Restart infinispan-server
5) After server is restarted, the method AddClientListenerOperation.execute is invoked. Wait with breakpoint at least 10 seconds to reproduce the issue.
6) Continue with breakpoint and see that method is executed another time. Continue with breakpoint again.
7) Now I can see that clientListener is executed twice for each added item, which is NOT ok.
{code}
....
INFO: Adding key: key-189
May 30, 2018 9:56:23 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-189
May 30, 2018 9:56:23 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-189
May 30, 2018 9:56:24 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-190
May 30, 2018 9:56:24 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-190
May 30, 2018 9:56:24 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-190
May 30, 2018 9:56:25 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-191
May 30, 2018 9:56:25 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-191
May 30, 2018 9:56:25 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-191
May 30, 2018 9:56:26 AM org.mposolda.ClientListenerFailoverTest main
INFO: Adding key: key-192
May 30, 2018 9:56:26 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-192
May 30, 2018 9:56:26 AM org.mposolda.ClientListenerFailoverTest$HotRodListener created
INFO: Listener: created: key-192
...
{code}
Sorry for the brittle steps to reproduce with the need of debugger, but that was the only way to simulate it reliably. Without debugger, the listener is usually re-registered once (correct behaviour), but sometimes twice (incorrect behaviour), so I think some issue is here.
> Client listeners are sometimes re-registered twice after infinispan-server restart
> ----------------------------------------------------------------------------------
>
> Key: ISPN-9220
> URL: https://issues.jboss.org/browse/ISPN-9220
> Project: Infinispan
> Issue Type: Bug
> Components: Hot Rod
> Affects Versions: 9.2.1.Final
> Reporter: Marek Posolda
>
> During test of summit, we saw some issue with client listeners. Setup was this:
> - Every site had one hotrod client (RHSSO), which used infinispan of version 8.2.8.Final and one infinispan-server, which I think was version 9.2.1.Final
> - Client (RH-SSO) had some client listener registered against remoteCache
> - After restart of infinispan-server, the RH-SSO was able to re-connect to it and CRUD the items to the remoteCache. But previously registered clientListeners were lost.
> Note that setup demo used just 1 infinispan-server on every site, so there were not any topology changes (no possibility of re-balance client listener to other node).
> I've tried to do simple reproducer for the scenario above and verified that with client 8.2.8.Final and server 9.2.1.Final, the client listener is not automatically re-registered after restart.
> BUT after upgrade client to 9.2.1.Final, re-registration worked fine and client listener was re-registered automatically. This is important as from long term perspective, we are upgrading to infinispan 9 on RH-SSO side. But I had the opposite issue, that sometimes the client listener is registered twice. See steps to reproduce for the details.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 10 months
[JBoss JIRA] (ISPN-9220) Client listeners are sometimes re-registered twice after infinispan-server restart
by Marek Posolda (JIRA)
Marek Posolda created ISPN-9220:
-----------------------------------
Summary: Client listeners are sometimes re-registered twice after infinispan-server restart
Key: ISPN-9220
URL: https://issues.jboss.org/browse/ISPN-9220
Project: Infinispan
Issue Type: Bug
Components: Hot Rod
Affects Versions: 9.2.1.Final
Reporter: Marek Posolda
During test of summit, we saw some issue with client listeners. Setup was this:
- Every site had one hotrod client (RHSSO), which used infinispan of version 8.2.8.Final and one infinispan-server, which I think was version 9.2.1.Final
- Client (RH-SSO) had some client listener registered against remoteCache
- After restart of infinispan-server, the RH-SSO was able to re-connect to it and CRUD the items to the remoteCache. But previously registered clientListeners were lost.
Note that setup demo used just 1 infinispan-server on every site, so there were not any topology changes (no possibility of re-balance client listener to other node).
I've tried to do simple reproducer for the scenario above and verified that with client 8.2.8.Final and server 9.2.1.Final, the client listener is not automatically re-registered after restart.
BUT after upgrade client to 9.2.1.Final, re-registration worked fine and client listener was re-registered automatically. This is important as from long term perspective, we are upgrading to infinispan 9 on RH-SSO side. But I had the opposite issue, that sometimes the client listener is registered twice. See steps to reproduce for the details.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 10 months
[JBoss JIRA] (ISPN-9219) WeakCounterAPITest.testAdd random failures
by Dan Berindei (JIRA)
Dan Berindei created ISPN-9219:
----------------------------------
Summary: WeakCounterAPITest.testAdd random failures
Key: ISPN-9219
URL: https://issues.jboss.org/browse/ISPN-9219
Project: Infinispan
Issue Type: Bug
Components: Clustered Counter
Affects Versions: 9.3.0.Beta1
Reporter: Dan Berindei
Assignee: Pedro Ruivo
Priority: Critical
Fix For: 9.3.0.Final
The exact assertion message and stack trace varies:
{noformat}
java.lang.AssertionError: expected:<0> but was:<-40>
java.lang.AssertionError: expected:<0> but was:<-20>
at org.infinispan.server.hotrod.counter.impl.WeakCounterImplTestStrategy.assertCounterValue(WeakCounterImplTestStrategy.java:62)
at org.infinispan.server.hotrod.counter.impl.WeakCounterImplTestStrategy.assertCounterValue(WeakCounterImplTestStrategy.java:27)
at org.infinispan.server.hotrod.counter.impl.BaseCounterImplTest.testAdd(BaseCounterImplTest.java:68)
at org.infinispan.client.hotrod.counter.WeakCounterAPITest.testAdd(WeakCounterAPITest.java:31)
java.lang.AssertionError: expected:<20> but was:<40>
java.lang.AssertionError: expected:<20> but was:<30>
at org.infinispan.server.hotrod.counter.impl.WeakCounterImplTestStrategy.assertCounterValue(WeakCounterImplTestStrategy.java:62)
at org.infinispan.server.hotrod.counter.impl.WeakCounterImplTestStrategy.assertCounterValue(WeakCounterImplTestStrategy.java:27)
at org.infinispan.server.hotrod.counter.impl.BaseCounterImplTest.testAdd(BaseCounterImplTest.java:65)
at org.infinispan.client.hotrod.counter.WeakCounterAPITest.testAdd(WeakCounterAPITest.java:31)
{noformat}
https://ci.infinispan.org/job/Infinispan/job/master/630/
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 10 months
[JBoss JIRA] (ISPN-9219) WeakCounterAPITest.testAdd random failures
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-9219?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-9219:
-------------------------------
Status: Open (was: New)
> WeakCounterAPITest.testAdd random failures
> ------------------------------------------
>
> Key: ISPN-9219
> URL: https://issues.jboss.org/browse/ISPN-9219
> Project: Infinispan
> Issue Type: Bug
> Components: Clustered Counter
> Affects Versions: 9.3.0.Beta1
> Reporter: Dan Berindei
> Assignee: Pedro Ruivo
> Priority: Critical
> Labels: testsuite_stability
> Fix For: 9.3.0.Final
>
>
> The exact assertion message and stack trace varies:
> {noformat}
> java.lang.AssertionError: expected:<0> but was:<-40>
> java.lang.AssertionError: expected:<0> but was:<-20>
> at org.infinispan.server.hotrod.counter.impl.WeakCounterImplTestStrategy.assertCounterValue(WeakCounterImplTestStrategy.java:62)
> at org.infinispan.server.hotrod.counter.impl.WeakCounterImplTestStrategy.assertCounterValue(WeakCounterImplTestStrategy.java:27)
> at org.infinispan.server.hotrod.counter.impl.BaseCounterImplTest.testAdd(BaseCounterImplTest.java:68)
> at org.infinispan.client.hotrod.counter.WeakCounterAPITest.testAdd(WeakCounterAPITest.java:31)
> java.lang.AssertionError: expected:<20> but was:<40>
> java.lang.AssertionError: expected:<20> but was:<30>
> at org.infinispan.server.hotrod.counter.impl.WeakCounterImplTestStrategy.assertCounterValue(WeakCounterImplTestStrategy.java:62)
> at org.infinispan.server.hotrod.counter.impl.WeakCounterImplTestStrategy.assertCounterValue(WeakCounterImplTestStrategy.java:27)
> at org.infinispan.server.hotrod.counter.impl.BaseCounterImplTest.testAdd(BaseCounterImplTest.java:65)
> at org.infinispan.client.hotrod.counter.WeakCounterAPITest.testAdd(WeakCounterAPITest.java:31)
> {noformat}
> https://ci.infinispan.org/job/Infinispan/job/master/630/
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 10 months
[JBoss JIRA] (ISPN-9173) Availability mode should be updated atomically with the actual members
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-9173?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-9173:
-------------------------------
Git Pull Request: https://github.com/infinispan/infinispan/pull/6021
> Availability mode should be updated atomically with the actual members
> ----------------------------------------------------------------------
>
> Key: ISPN-9173
> URL: https://issues.jboss.org/browse/ISPN-9173
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.3.0.Beta1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Labels: testsuite_stability
> Fix For: 9.3.0.CR1
>
>
> This is a follow-up on ISPN-7682, which asks for the topology itself to be updated atomically.
> {{LocalTopologyManagerImpl}} has additional logic to update the availability mode first when the cache becomes degraded and to update it last when the cache becomes available, which means any delay between the updates cannot cause data inconsistencies.
> But that logic doesn't really belong in {{LocalTopologyManagerImpl}}, and it's easy to forget it's there (and in fact we had a bug there related to the new rebalance phases).
> In addition, tests that want to check the cache behaviour in degraded mode and wait only for the availability mode change will fail if there's a big delay between the availability mode change. I actually hit this while testing my ISPN-8731/ISPN-7682 changes, and I had added a random delay in {{StateConsumerImpl}} before {{distributionManager.setCacheTopology()}}.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 10 months