[JBoss JIRA] (ISPN-5021) Nodes that finish the rebalance later can see outdated values
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-5021?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-5021:
------------------------------------
This is causing random failures in {{JCacheTwoCachesAnnotationsTest}}. The test has one per-class node that is reused by all test methods, and 2 per-method nodes that are created just during the test method (without waiting for the rebalance to finish.)
The scenario is like this:
# Rebalance starts, readCH = \{node1, node2\}, writeCH = \{node1, node2, node3\}
# Rebalance ends, but CH_UPDATE is delayed on node2
# {{cache1.put(k, v)}} updates the value on node1 and node3
# {{cache2.entrySet().iterator()}} sees that all segments are local and returns 0 entries.
# {{cache2.get(k)}} would also return {{null}} at this point, but the test fails before calling it.
> Nodes that finish the rebalance later can see outdated values
> -------------------------------------------------------------
>
> Key: ISPN-5021
> URL: https://issues.jboss.org/browse/ISPN-5021
> Project: Infinispan
> Issue Type: Bug
> Components: Core, State Transfer
> Affects Versions: 7.0.2.Final
> Reporter: Dan Berindei
> Assignee: Pedro Ruivo
> Priority: Critical
> Labels: testsuite_stability
>
> Copied from [ISPN-4444|https://issues.jboss.org/browse/ISPN-4444?focusedCommentId=1302...]
> If the CH_UPDATE command is delayed on the old owner, the new owners might update the key without the old owner knowing, and a locality check on the old owner won't help.
> I remember one thing that struck me when reading the Raft algorithm was that they install configuration changes symmetrically, in 3 phases. We might need to do the same for our rebalance:
> 1. T0: read_ch=old, write_ch=old
> 2. start a rebalance
> 3. T1: read_ch=old, write_ch=old+new
> 4. new owners have all the data
> 5. T2: read_ch=new, write_ch=old+new
> 6. remove old cache entries and ignore further writes
> 7. T3: read_ch=new, write_ch=new
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 4 months
[JBoss JIRA] (ISPN-5021) Nodes that finish the rebalance later can see outdated values
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-5021?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-5021:
-------------------------------
Labels: testsuite_stability (was: )
> Nodes that finish the rebalance later can see outdated values
> -------------------------------------------------------------
>
> Key: ISPN-5021
> URL: https://issues.jboss.org/browse/ISPN-5021
> Project: Infinispan
> Issue Type: Bug
> Components: Core, State Transfer
> Affects Versions: 7.0.2.Final
> Reporter: Dan Berindei
> Assignee: Pedro Ruivo
> Priority: Critical
> Labels: testsuite_stability
>
> Copied from [ISPN-4444|https://issues.jboss.org/browse/ISPN-4444?focusedCommentId=1302...]
> If the CH_UPDATE command is delayed on the old owner, the new owners might update the key without the old owner knowing, and a locality check on the old owner won't help.
> I remember one thing that struck me when reading the Raft algorithm was that they install configuration changes symmetrically, in 3 phases. We might need to do the same for our rebalance:
> 1. T0: read_ch=old, write_ch=old
> 2. start a rebalance
> 3. T1: read_ch=old, write_ch=old+new
> 4. new owners have all the data
> 5. T2: read_ch=new, write_ch=old+new
> 6. remove old cache entries and ignore further writes
> 7. T3: read_ch=new, write_ch=new
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 4 months
[JBoss JIRA] (ISPN-7160) RemoteSpringSessionTest random failures
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-7160?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-7160:
------------------------------------
Same problem in {{InfinispanEmbeddedSessionRepositoryTest}} and {{InfinispanRemoteSessionRepositoryTest}}:
{noformat}
00:41:03,352 ERROR (testng-InfinispanRemoteSessionRepositoryTest:[]) [TestSuiteProgress] Test setup failed: org.infinispan.spring.session.InfinispanRemoteSessionRepositoryTest.beforeClass
org.infinispan.jmx.JmxDomainConflictException: ISPN000034: There's already a JMX MBean instance type=CacheManager,name="DefaultCacheManager" already registered under 'org.infinispan' JMX domain. If you want to allow multiple instances configured with same JMX domain enable 'allowDuplicateDomains' attribute in 'globalJmxStatistics' config element
at org.infinispan.jmx.JmxUtil.buildJmxDomain(JmxUtil.java:53) ~[infinispan-core.jar:9.0.0-SNAPSHOT]
at org.infinispan.jmx.CacheManagerJmxRegistration.updateDomain(CacheManagerJmxRegistration.java:79) ~[infinispan-core.jar:9.0.0-SNAPSHOT]
at org.infinispan.jmx.CacheManagerJmxRegistration.buildRegistrar(CacheManagerJmxRegistration.java:73) ~[infinispan-core.jar:9.0.0-SNAPSHOT]
at org.infinispan.jmx.AbstractJmxRegistration.registerMBeans(AbstractJmxRegistration.java:37) ~[infinispan-core.jar:9.0.0-SNAPSHOT]
at org.infinispan.jmx.CacheManagerJmxRegistration.start(CacheManagerJmxRegistration.java:41) ~[infinispan-core.jar:9.0.0-SNAPSHOT]
at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:661) ~[infinispan-core.jar:9.0.0-SNAPSHOT]
at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:235) ~[infinispan-core.jar:9.0.0-SNAPSHOT]
at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:167) ~[infinispan-core.jar:9.0.0-SNAPSHOT]
at org.infinispan.spring.session.InfinispanRemoteSessionRepositoryTest.beforeClass(InfinispanRemoteSessionRepositoryTest.java:28) ~[test-classes/:?]
{noformat}
> RemoteSpringSessionTest random failures
> ---------------------------------------
>
> Key: ISPN-7160
> URL: https://issues.jboss.org/browse/ISPN-7160
> Project: Infinispan
> Issue Type: Bug
> Components: Spring Integration
> Reporter: Dan Berindei
> Assignee: Sebastian Łaskawiec
> Priority: Critical
> Labels: testsuite_stability
> Fix For: 9.0.0.Beta1
>
>
> {{RemoteSpringSessionTest}} and {{EmbeddedSpringSessionTest}} use the default cache jmx configuration, so they try to register the same JMX object names. If they run in parallel, one of them will fail to start the cache manager:
> {noformat}
> org.infinispan.jmx.JmxDomainConflictException: ISPN000034: There's already a JMX MBean instance type=CacheManager,name="DefaultCacheManager" already registered under 'org.infinispan' JMX domain. If you want to allow multiple instances configured with same JMX domain enable 'allowDuplicateDomains' attribute in 'globalJmxStatistics' config element
> at org.infinispan.jmx.JmxUtil.buildJmxDomain(JmxUtil.java:53)
> at org.infinispan.jmx.CacheManagerJmxRegistration.updateDomain(CacheManagerJmxRegistration.java:79)
> at org.infinispan.jmx.CacheManagerJmxRegistration.buildRegistrar(CacheManagerJmxRegistration.java:73)
> at org.infinispan.jmx.AbstractJmxRegistration.registerMBeans(AbstractJmxRegistration.java:37)
> at org.infinispan.jmx.CacheManagerJmxRegistration.start(CacheManagerJmxRegistration.java:41)
> at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:661)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:235)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:147)
> at org.infinispan.integrationtests.spring.boot.session.remote.RemoteSpringSessionTest.beforeclass(RemoteSpringSessionTest.java:29)
> {noformat}
> If the cache manager was not created, teardown also fails with a NPE:
> {noformat}
> java.lang.NullPointerException
> at org.infinispan.integrationtests.spring.boot.session.remote.RemoteSpringSessionTest.afterClass(RemoteSpringSessionTest.java:41)
> {noformat}
> The test should use the {{TestCacheManagerFactory}} methods to create the cache manager, which will set a unique JMX domain name for each instance.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 4 months
[JBoss JIRA] (ISPN-6621) Server Management Console not showing up data in Cache containers page when tried with hotrod connector alone
by Pedro Zapata (JIRA)
[ https://issues.jboss.org/browse/ISPN-6621?page=com.atlassian.jira.plugin.... ]
Pedro Zapata updated ISPN-6621:
-------------------------------
Fix Version/s: (was: 8.2.1.Final)
> Server Management Console not showing up data in Cache containers page when tried with hotrod connector alone
> -------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-6621
> URL: https://issues.jboss.org/browse/ISPN-6621
> Project: Infinispan
> Issue Type: Bug
> Components: Console
> Affects Versions: 8.2.1.Final
> Reporter: Krishna Kalesh Balakrishnan
> Assignee: Vladimir Blagojevic
> Attachments: clustter_page_hotrod_alone.PNG, enabled_all_connectors.PNG, enabled_hotrod_alone.PNG
>
>
> While running infinispan-server-8.2.1.Final in domain mode, tried to use only hotrod-connector after removing memcached-connector, rest-connector & websocket-connector from <subsystem xmlns="urn:infinispan:server:endpoint:8.0">
> After removing memcached-connector, rest-connector & websocket-connector, Server Management Console not showing up anything in Cache container view.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 4 months
[JBoss JIRA] (ISPN-4131) Lock acquired forever with delayed PrepareCommand
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-4131?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-4131:
-----------------------------------------------
Petr Jurak <pjurak(a)redhat.com> changed the Status of [bug 1378877|https://bugzilla.redhat.com/show_bug.cgi?id=1378877] from NEW to ASSIGNED
> Lock acquired forever with delayed PrepareCommand
> -------------------------------------------------
>
> Key: ISPN-4131
> URL: https://issues.jboss.org/browse/ISPN-4131
> Project: Infinispan
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 6.0.2.Final, 7.0.0.Alpha1
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
> Labels: 630betablocker
> Fix For: 7.0.0.Beta1, 7.0.0.Final
>
>
> Distributed transactional cache:
> 1. A sends Prepare to B
> 2. B receives Prepare, but due to ongoing ST it is blocked
> 3. B replication timeout elapses
> 4. B sends Rollback, this does not find the TX as Prepare was not executed yet. The transaction is put into completedTransactions.
> 5. Completed transactions timeout elapses. This is by default 15 seconds, way shorter than ST timeout (due to which the Prepare was blocked)
> 6. Prepare is executed on B, acquiring lock on K
> Nobody will rollback the TX as originator thinks it was already rolled back.
> Result: key K will be locked forever, all attempts to update/remove it will fail.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 4 months
[JBoss JIRA] (ISPN-7166) WriteSkewConsistencyTest.testValidationOnlyInPrimaryOwner[DIST_SYNC] random failures
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-7166?page=com.atlassian.jira.plugin.... ]
William Burns updated ISPN-7166:
--------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> WriteSkewConsistencyTest.testValidationOnlyInPrimaryOwner[DIST_SYNC] random failures
> ------------------------------------------------------------------------------------
>
> Key: ISPN-7166
> URL: https://issues.jboss.org/browse/ISPN-7166
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 9.0.0.Alpha4
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Labels: testsuite_stability
> Fix For: 9.0.0.Beta1
>
>
> The test requires a key to have a specific primary owner and backup owner, but because is uses the default {{SynchronizedConsistentHashFactory}}, sometimes there's no segment meeting that requirement.
> {noformat}
> 16:52:02,516 ERROR (testng-WriteSkewConsistencyTest[DIST_SYNC]:[]) [TestSuiteProgress] Test failed: org.infinispan.container.versioning.WriteSkewConsistencyTest.testValidationOnlyInPrimaryOwner[DIST_SYNC]
> java.lang.IllegalStateException: Could not find any segment owned by Cache '___defaultcache'@WriteSkewConsistencyTest[DIST_SYNC]-NodeB-47455, [Cache '___defaultcache'@WriteSkewConsistencyTest[DIST_SYNC]-NodeA-21455], primary segments: [33, 2, 3, 5, 37, 6, 41, 10, 45, 15, 48, 18, 50, 25, 59, 29], backup segments: {Cache '___defaultcache'@WriteSkewConsistencyTest[DIST_SYNC]-NodeA-21455=[35, 8, 40, 12, 14, 52, 21, 53, 22, 24, 57, 58, 27, 31]}
> at org.infinispan.distribution.MagicKey.<init>(MagicKey.java:85) ~[test-classes/:?]
> at org.infinispan.distribution.MagicKey.<init>(MagicKey.java:136) ~[test-classes/:?]
> at org.infinispan.container.versioning.WriteSkewConsistencyTest.testValidationOnlyInPrimaryOwner(WriteSkewConsistencyTest.java:58) ~[test-classes/:?]
> {noformat}
> The test should either use a fixed key and accept random owners, or use a {{ControlledConsistentHashFactory}} to pin the owners.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 4 months