[JBoss JIRA] (ISPN-3366) Data loss when entry forwarding to primary owner and primary owner shutdown
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-3366?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-3366:
-------------------------------
Status: Pull Request Sent (was: Reopened)
Git Pull Request: https://github.com/infinispan/infinispan/pull/1995, https://github.com/infinispan/infinispan/pull/2003 (was: https://github.com/infinispan/infinispan/pull/1995)
> Data loss when entry forwarding to primary owner and primary owner shutdown
> ---------------------------------------------------------------------------
>
> Key: ISPN-3366
> URL: https://issues.jboss.org/browse/ISPN-3366
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.4.Final, 6.0.0.Alpha1
> Reporter: Takayoshi Kimura
> Assignee: Dan Berindei
> Priority: Critical
> Fix For: 5.2.8.Final, 6.0.0.Alpha3, 6.0.0.Final
>
> Attachments: ISPN-3366-full-logs-3rd.zip, ISPN-3366-full-logs-4th.zip, ISPN-3366-logs.zip
>
>
> Looks like a problem in entry forwarding.
> Here is test scenario:
> * DIST numOwners=2, start with 4 nodes cluster then normal shutdown 1 node during load
> * HotRod putIfAbsent accesses from 40 threads (1 process, 1 remote cache instance), 40000 entries total
> After the test run, the numberOfEntries on each node are:
> * node1: 26608
> * node2: 26622
> * node3: 26746
> * node4: 0
> Total is 79976 and HotRod client received 11 errors, so 79976 + (11 * 2) = 79998. It means 1 entry is completely missing.
> Let's take a look at the missing entry, hash(thread16key59) = 574ff563.
> Current CH: owners(574ff563) are [node4, node1]
> The events sequence is:
> * hotrod -> node1
> * node1 forwarding it to primary owner node4
> * node4 doesn't process the forwarded entry, shutdown
> Result owners(7c29bccb) is [] empty. This entry is completely lost without any errors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 8 months
[JBoss JIRA] (ISPN-3404) ClusterCacheLoader - clusteredGetCommand response value is null
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3404?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration updated ISPN-3404:
------------------------------------------
Bugzilla Update: Perform
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=996045
> ClusterCacheLoader - clusteredGetCommand response value is null
> ---------------------------------------------------------------
>
> Key: ISPN-3404
> URL: https://issues.jboss.org/browse/ISPN-3404
> Project: Infinispan
> Issue Type: Bug
> Reporter: Tomas Sykora
> Assignee: Mircea Markus
> Attachments: clusterCacheLoaderNullPointerResponseValueError.txt, server1DR2.log, server2DR2.log
>
>
> We found this issue in DR2 (Alpha2). In Alpha1 this was ok.
> Our internal test:
> {code:title=Bar.java|borderStyle=solid}
> @Test
> public void testLazyLoadingWhenStateTransferDisabled() throws Exception {
> controller.start(CONTAINER1);
> mc = new MemcachedHelper(server1.getMemcachedEndpoint().getInetAddress().getHostName(), server1.getMemcachedEndpoint()
> .getPort());
> mc.set("k1", "v1");
> mc.set("k2", "v2");
> assertEquals("v1", mc.get("k1"));
> assertEquals("v2", mc.get("k2"));
> assertEquals(2, server1.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> controller.start(CONTAINER2);
> mc2 = new MemcachedHelper(server2.getMemcachedEndpoint().getInetAddress().getHostName(), server2.getMemcachedEndpoint()
> .getPort());
> assertEquals(2, server2.getDefaultCacheManager().getClusterSize());
> //state-transfer = false -> no entries in the newly joined node
> assertEquals(0, server2.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> assertEquals("v1", mc2.get("k1")); //lazily load the entries
> assertEquals("v2", mc2.get("k2"));
> assertEquals(2, server2.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> mc2.set("k3", "v3"); // THIS IS FAILING HERE
> assertEquals(3, server2.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> assertEquals(3, server1.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> controller.kill(CONTAINER2);
> controller.kill(CONTAINER1);
> }
> {code}
> *mc2.set("k3", "v3"); = put into memcached cache, is throwing an exception: see in attachment.*
> You can see MemcachedHelper class here: https://code.engineering.redhat.com/gerrit/gitweb?p=jdg-functional-tests....
> I also include trace logs for both memcached jdg servers if they can be useful.
> ---------------
> I was able to replicate the same error in Infinispan test suite for Rest endpoint. It was enough to start 2 REST nodes and issue 1 put resulting into the same error.
> {code:title=Bar.java|borderStyle=solid}
> ConfigurationBuilder cfgBuilder = AbstractCacheTest.getDefaultClusteredCacheConfig(CacheMode.REPL_SYNC, false);
> cfgBuilder.clustering().hash().numOwners(1);
> cfgBuilder.clustering().stateTransfer().fetchInMemoryState(false);
> cfgBuilder.clustering().stateTransfer().timeout(20000);
> cfgBuilder.loaders().addClusterCacheLoader().remoteCallTimeout(60000);
> RestServerConfigurationBuilder restCfgBuilder = new RestServerConfigurationBuilder();
> addServer("1", 8890, TestCacheManagerFactory.createClusteredCacheManager(cfgBuilder), restCfgBuilder.build());
> addServer("2", 8891, TestCacheManagerFactory.createClusteredCacheManager(cfgBuilder), restCfgBuilder.build());
> startServers();
> ......
> PutMethod put = new PutMethod(PATH1 + "key1");
> put.setRequestEntity(new StringRequestEntity("value1", "application/text", null));
> call(put);
> assertEquals(put.getStatusCode(), HttpServletResponse.SC_OK);
> put.releaseConnection();
> {code}
> This scenario is different from aforementioned JDG internal test suite one but I expect this should be ok and should be performed without any problems.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 8 months
[JBoss JIRA] (ISPN-3404) ClusterCacheLoader - clusteredGetCommand response value is null
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3404?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-3404:
-----------------------------------------------
Tomas Sykora <tsykora(a)redhat.com> made a comment on [bug 996045|https://bugzilla.redhat.com/show_bug.cgi?id=996045]
See https://issues.jboss.org/browse/ISPN-3404 for details.
> ClusterCacheLoader - clusteredGetCommand response value is null
> ---------------------------------------------------------------
>
> Key: ISPN-3404
> URL: https://issues.jboss.org/browse/ISPN-3404
> Project: Infinispan
> Issue Type: Bug
> Reporter: Tomas Sykora
> Assignee: Mircea Markus
> Attachments: clusterCacheLoaderNullPointerResponseValueError.txt, server1DR2.log, server2DR2.log
>
>
> We found this issue in DR2 (Alpha2). In Alpha1 this was ok.
> Our internal test:
> {code:title=Bar.java|borderStyle=solid}
> @Test
> public void testLazyLoadingWhenStateTransferDisabled() throws Exception {
> controller.start(CONTAINER1);
> mc = new MemcachedHelper(server1.getMemcachedEndpoint().getInetAddress().getHostName(), server1.getMemcachedEndpoint()
> .getPort());
> mc.set("k1", "v1");
> mc.set("k2", "v2");
> assertEquals("v1", mc.get("k1"));
> assertEquals("v2", mc.get("k2"));
> assertEquals(2, server1.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> controller.start(CONTAINER2);
> mc2 = new MemcachedHelper(server2.getMemcachedEndpoint().getInetAddress().getHostName(), server2.getMemcachedEndpoint()
> .getPort());
> assertEquals(2, server2.getDefaultCacheManager().getClusterSize());
> //state-transfer = false -> no entries in the newly joined node
> assertEquals(0, server2.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> assertEquals("v1", mc2.get("k1")); //lazily load the entries
> assertEquals("v2", mc2.get("k2"));
> assertEquals(2, server2.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> mc2.set("k3", "v3"); // THIS IS FAILING HERE
> assertEquals(3, server2.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> assertEquals(3, server1.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> controller.kill(CONTAINER2);
> controller.kill(CONTAINER1);
> }
> {code}
> *mc2.set("k3", "v3"); = put into memcached cache, is throwing an exception: see in attachment.*
> You can see MemcachedHelper class here: https://code.engineering.redhat.com/gerrit/gitweb?p=jdg-functional-tests....
> I also include trace logs for both memcached jdg servers if they can be useful.
> ---------------
> I was able to replicate the same error in Infinispan test suite for Rest endpoint. It was enough to start 2 REST nodes and issue 1 put resulting into the same error.
> {code:title=Bar.java|borderStyle=solid}
> ConfigurationBuilder cfgBuilder = AbstractCacheTest.getDefaultClusteredCacheConfig(CacheMode.REPL_SYNC, false);
> cfgBuilder.clustering().hash().numOwners(1);
> cfgBuilder.clustering().stateTransfer().fetchInMemoryState(false);
> cfgBuilder.clustering().stateTransfer().timeout(20000);
> cfgBuilder.loaders().addClusterCacheLoader().remoteCallTimeout(60000);
> RestServerConfigurationBuilder restCfgBuilder = new RestServerConfigurationBuilder();
> addServer("1", 8890, TestCacheManagerFactory.createClusteredCacheManager(cfgBuilder), restCfgBuilder.build());
> addServer("2", 8891, TestCacheManagerFactory.createClusteredCacheManager(cfgBuilder), restCfgBuilder.build());
> startServers();
> ......
> PutMethod put = new PutMethod(PATH1 + "key1");
> put.setRequestEntity(new StringRequestEntity("value1", "application/text", null));
> call(put);
> assertEquals(put.getStatusCode(), HttpServletResponse.SC_OK);
> put.releaseConnection();
> {code}
> This scenario is different from aforementioned JDG internal test suite one but I expect this should be ok and should be performed without any problems.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 8 months
[JBoss JIRA] (ISPN-3404) ClusterCacheLoader - clusteredGetCommand response value is null
by Tomas Sykora (JIRA)
[ https://issues.jboss.org/browse/ISPN-3404?page=com.atlassian.jira.plugin.... ]
Tomas Sykora updated ISPN-3404:
-------------------------------
Attachment: clusterCacheLoaderNullPointerResponseValueError.txt
server1DR2.log
server2DR2.log
> ClusterCacheLoader - clusteredGetCommand response value is null
> ---------------------------------------------------------------
>
> Key: ISPN-3404
> URL: https://issues.jboss.org/browse/ISPN-3404
> Project: Infinispan
> Issue Type: Bug
> Reporter: Tomas Sykora
> Assignee: Mircea Markus
> Attachments: clusterCacheLoaderNullPointerResponseValueError.txt, server1DR2.log, server2DR2.log
>
>
> We found this issue in DR2 (Alpha2). In Alpha1 this was ok.
> Our internal test:
> {code:title=Bar.java|borderStyle=solid}
> @Test
> public void testLazyLoadingWhenStateTransferDisabled() throws Exception {
> controller.start(CONTAINER1);
> mc = new MemcachedHelper(server1.getMemcachedEndpoint().getInetAddress().getHostName(), server1.getMemcachedEndpoint()
> .getPort());
> mc.set("k1", "v1");
> mc.set("k2", "v2");
> assertEquals("v1", mc.get("k1"));
> assertEquals("v2", mc.get("k2"));
> assertEquals(2, server1.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> controller.start(CONTAINER2);
> mc2 = new MemcachedHelper(server2.getMemcachedEndpoint().getInetAddress().getHostName(), server2.getMemcachedEndpoint()
> .getPort());
> assertEquals(2, server2.getDefaultCacheManager().getClusterSize());
> //state-transfer = false -> no entries in the newly joined node
> assertEquals(0, server2.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> assertEquals("v1", mc2.get("k1")); //lazily load the entries
> assertEquals("v2", mc2.get("k2"));
> assertEquals(2, server2.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> mc2.set("k3", "v3"); // THIS IS FAILING HERE
> assertEquals(3, server2.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> assertEquals(3, server1.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> controller.kill(CONTAINER2);
> controller.kill(CONTAINER1);
> }
> {code}
> *mc2.set("k3", "v3"); = put into memcached cache, is throwing an exception: see in attachment.*
> You can see MemcachedHelper class here: https://code.engineering.redhat.com/gerrit/gitweb?p=jdg-functional-tests....
> I also include trace logs for both memcached jdg servers if they can be useful.
> ---------------
> I was able to replicate the same error in Infinispan test suite for Rest endpoint. It was enough to start 2 REST nodes and issue 1 put resulting into the same error.
> {code:title=Bar.java|borderStyle=solid}
> ConfigurationBuilder cfgBuilder = AbstractCacheTest.getDefaultClusteredCacheConfig(CacheMode.REPL_SYNC, false);
> cfgBuilder.clustering().hash().numOwners(1);
> cfgBuilder.clustering().stateTransfer().fetchInMemoryState(false);
> cfgBuilder.clustering().stateTransfer().timeout(20000);
> cfgBuilder.loaders().addClusterCacheLoader().remoteCallTimeout(60000);
> RestServerConfigurationBuilder restCfgBuilder = new RestServerConfigurationBuilder();
> addServer("1", 8890, TestCacheManagerFactory.createClusteredCacheManager(cfgBuilder), restCfgBuilder.build());
> addServer("2", 8891, TestCacheManagerFactory.createClusteredCacheManager(cfgBuilder), restCfgBuilder.build());
> startServers();
> ......
> PutMethod put = new PutMethod(PATH1 + "key1");
> put.setRequestEntity(new StringRequestEntity("value1", "application/text", null));
> call(put);
> assertEquals(put.getStatusCode(), HttpServletResponse.SC_OK);
> put.releaseConnection();
> {code}
> This scenario is different from aforementioned JDG internal test suite one but I expect this should be ok and should be performed without any problems.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 8 months
[JBoss JIRA] (ISPN-3404) ClusterCacheLoader - clusteredGetCommand response value is null
by Tomas Sykora (JIRA)
Tomas Sykora created ISPN-3404:
----------------------------------
Summary: ClusterCacheLoader - clusteredGetCommand response value is null
Key: ISPN-3404
URL: https://issues.jboss.org/browse/ISPN-3404
Project: Infinispan
Issue Type: Bug
Reporter: Tomas Sykora
Assignee: Mircea Markus
We found this issue in DR2 (Alpha2). In Alpha1 this was ok.
Our internal test:
{code:title=Bar.java|borderStyle=solid}
@Test
public void testLazyLoadingWhenStateTransferDisabled() throws Exception {
controller.start(CONTAINER1);
mc = new MemcachedHelper(server1.getMemcachedEndpoint().getInetAddress().getHostName(), server1.getMemcachedEndpoint()
.getPort());
mc.set("k1", "v1");
mc.set("k2", "v2");
assertEquals("v1", mc.get("k1"));
assertEquals("v2", mc.get("k2"));
assertEquals(2, server1.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
controller.start(CONTAINER2);
mc2 = new MemcachedHelper(server2.getMemcachedEndpoint().getInetAddress().getHostName(), server2.getMemcachedEndpoint()
.getPort());
assertEquals(2, server2.getDefaultCacheManager().getClusterSize());
//state-transfer = false -> no entries in the newly joined node
assertEquals(0, server2.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
assertEquals("v1", mc2.get("k1")); //lazily load the entries
assertEquals("v2", mc2.get("k2"));
assertEquals(2, server2.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
mc2.set("k3", "v3"); // THIS IS FAILING HERE
assertEquals(3, server2.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
assertEquals(3, server1.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
controller.kill(CONTAINER2);
controller.kill(CONTAINER1);
}
{code}
*mc2.set("k3", "v3"); = put into memcached cache, is throwing an exception: see in attachment.*
You can see MemcachedHelper class here: https://code.engineering.redhat.com/gerrit/gitweb?p=jdg-functional-tests....
I also include trace logs for both memcached jdg servers if they can be useful.
---------------
I was able to replicate the same error in Infinispan test suite for Rest endpoint. It was enough to start 2 REST nodes and issue 1 put resulting into the same error.
{code:title=Bar.java|borderStyle=solid}
ConfigurationBuilder cfgBuilder = AbstractCacheTest.getDefaultClusteredCacheConfig(CacheMode.REPL_SYNC, false);
cfgBuilder.clustering().hash().numOwners(1);
cfgBuilder.clustering().stateTransfer().fetchInMemoryState(false);
cfgBuilder.clustering().stateTransfer().timeout(20000);
cfgBuilder.loaders().addClusterCacheLoader().remoteCallTimeout(60000);
RestServerConfigurationBuilder restCfgBuilder = new RestServerConfigurationBuilder();
addServer("1", 8890, TestCacheManagerFactory.createClusteredCacheManager(cfgBuilder), restCfgBuilder.build());
addServer("2", 8891, TestCacheManagerFactory.createClusteredCacheManager(cfgBuilder), restCfgBuilder.build());
startServers();
......
PutMethod put = new PutMethod(PATH1 + "key1");
put.setRequestEntity(new StringRequestEntity("value1", "application/text", null));
call(put);
assertEquals(put.getStatusCode(), HttpServletResponse.SC_OK);
put.releaseConnection();
{code}
This scenario is different from aforementioned JDG internal test suite one but I expect this should be ok and should be performed without any problems.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 8 months
[JBoss JIRA] (ISPN-3357) Insufficient owners with putIfAbsent during rebalance
by Takayoshi Kimura (JIRA)
[ https://issues.jboss.org/browse/ISPN-3357?page=com.atlassian.jira.plugin.... ]
Takayoshi Kimura edited comment on ISPN-3357 at 8/12/13 3:09 AM:
-----------------------------------------------------------------
It's rare but this issue also happens with node leave rebalance so I've updated the title accordingly.
I've attached logs from my test - 8 nodes dist cluster, a node leave with kill -9, 6 backups are missing. The event sequence is basically same as node join rebalance.
was (Author: tkimura):
It's rare but this issue also happens with node leave rebalance so I've updated the title accordingly.
I've attached logs from my test - 8 nodes dist cluster, a node leave with kill -9, 6 backups are missing.
> Insufficient owners with putIfAbsent during rebalance
> -----------------------------------------------------
>
> Key: ISPN-3357
> URL: https://issues.jboss.org/browse/ISPN-3357
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.4.Final, 6.0.0.Alpha1
> Reporter: Takayoshi Kimura
> Assignee: Dan Berindei
> Priority: Critical
> Attachments: 7c29bccb.log, ISPN-3357-full-logs-leave.zip
>
>
> Here is test scenario:
> * DIST numOwners=2, start with 3 nodes cluster then join 1 node during load
> * HotRod putIfAbsent accesses from 40 threads (1 process, 1 remote cache instance), 40000 entries total
> After the test run, the numberOfEntries on each node are:
> * node1: 20074
> * node2: 19888
> * node3: 20114
> * node4: 18885
> Total is 78961, 1039 entries are missing. No error on HotRod client side so 80000 entries should be there.
> Let's take a look at example missing entry, hash(thread01key151) = 7c29bccb.
> Current CH: owners(7c29bccb) are [node1, node2]
> Pending CH: owners(7c29bccb) are [node1, node2, node4]
> Balanced CH: owners(7c29bccb) are [node1, node4]
> The events sequence is:
> * hotrod -> node1
> * node1 -> node2, node4
> * node2 committed entry
> * node4 performed clustered get before write, got a value from node2 and will not commit the entry because this node thinks it's not changed/created
> * node1 committed entry
> * node2 invalidates the entry because it's no longer an owner
> Result owners(7c29bccb) are only node1 and node4 is missing. This entry may be completely lost by further rebalances when node4 is donor for this segment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 8 months
[JBoss JIRA] (ISPN-3357) Insufficient owners with putIfAbsent during rebalance
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3357?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration updated ISPN-3357:
------------------------------------------
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=989808, https://bugzilla.redhat.com/show_bug.cgi?id=989809 (was: https://bugzilla.redhat.com/show_bug.cgi?id=989808)
> Insufficient owners with putIfAbsent during rebalance
> -----------------------------------------------------
>
> Key: ISPN-3357
> URL: https://issues.jboss.org/browse/ISPN-3357
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.4.Final, 6.0.0.Alpha1
> Reporter: Takayoshi Kimura
> Assignee: Dan Berindei
> Priority: Critical
> Attachments: 7c29bccb.log, ISPN-3357-full-logs-leave.zip
>
>
> Here is test scenario:
> * DIST numOwners=2, start with 3 nodes cluster then join 1 node during load
> * HotRod putIfAbsent accesses from 40 threads (1 process, 1 remote cache instance), 40000 entries total
> After the test run, the numberOfEntries on each node are:
> * node1: 20074
> * node2: 19888
> * node3: 20114
> * node4: 18885
> Total is 78961, 1039 entries are missing. No error on HotRod client side so 80000 entries should be there.
> Let's take a look at example missing entry, hash(thread01key151) = 7c29bccb.
> Current CH: owners(7c29bccb) are [node1, node2]
> Pending CH: owners(7c29bccb) are [node1, node2, node4]
> Balanced CH: owners(7c29bccb) are [node1, node4]
> The events sequence is:
> * hotrod -> node1
> * node1 -> node2, node4
> * node2 committed entry
> * node4 performed clustered get before write, got a value from node2 and will not commit the entry because this node thinks it's not changed/created
> * node1 committed entry
> * node2 invalidates the entry because it's no longer an owner
> Result owners(7c29bccb) are only node1 and node4 is missing. This entry may be completely lost by further rebalances when node4 is donor for this segment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 8 months
[JBoss JIRA] (ISPN-3357) Insufficient owners with putIfAbsent during rebalance
by Takayoshi Kimura (JIRA)
[ https://issues.jboss.org/browse/ISPN-3357?page=com.atlassian.jira.plugin.... ]
Takayoshi Kimura updated ISPN-3357:
-----------------------------------
Attachment: ISPN-3357-full-logs-leave.zip
It's rare but this issue also happens with node leave rebalance so I've updated the title accordingly.
I've attached logs from my test - 8 nodes dist cluster, a node leave with kill -9, 6 backups are missing.
> Insufficient owners with putIfAbsent during rebalance
> -----------------------------------------------------
>
> Key: ISPN-3357
> URL: https://issues.jboss.org/browse/ISPN-3357
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.4.Final, 6.0.0.Alpha1
> Reporter: Takayoshi Kimura
> Assignee: Dan Berindei
> Priority: Critical
> Attachments: 7c29bccb.log, ISPN-3357-full-logs-leave.zip
>
>
> Here is test scenario:
> * DIST numOwners=2, start with 3 nodes cluster then join 1 node during load
> * HotRod putIfAbsent accesses from 40 threads (1 process, 1 remote cache instance), 40000 entries total
> After the test run, the numberOfEntries on each node are:
> * node1: 20074
> * node2: 19888
> * node3: 20114
> * node4: 18885
> Total is 78961, 1039 entries are missing. No error on HotRod client side so 80000 entries should be there.
> Let's take a look at example missing entry, hash(thread01key151) = 7c29bccb.
> Current CH: owners(7c29bccb) are [node1, node2]
> Pending CH: owners(7c29bccb) are [node1, node2, node4]
> Balanced CH: owners(7c29bccb) are [node1, node4]
> The events sequence is:
> * hotrod -> node1
> * node1 -> node2, node4
> * node2 committed entry
> * node4 performed clustered get before write, got a value from node2 and will not commit the entry because this node thinks it's not changed/created
> * node1 committed entry
> * node2 invalidates the entry because it's no longer an owner
> Result owners(7c29bccb) are only node1 and node4 is missing. This entry may be completely lost by further rebalances when node4 is donor for this segment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 8 months
[JBoss JIRA] (ISPN-3357) Insufficient owners with putIfAbsent during rebalance
by Takayoshi Kimura (JIRA)
[ https://issues.jboss.org/browse/ISPN-3357?page=com.atlassian.jira.plugin.... ]
Takayoshi Kimura updated ISPN-3357:
-----------------------------------
Summary: Insufficient owners with putIfAbsent during rebalance (was: Insufficient owners with putIfAbsent during node join rebalance)
> Insufficient owners with putIfAbsent during rebalance
> -----------------------------------------------------
>
> Key: ISPN-3357
> URL: https://issues.jboss.org/browse/ISPN-3357
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.4.Final, 6.0.0.Alpha1
> Reporter: Takayoshi Kimura
> Assignee: Dan Berindei
> Priority: Critical
> Attachments: 7c29bccb.log
>
>
> Here is test scenario:
> * DIST numOwners=2, start with 3 nodes cluster then join 1 node during load
> * HotRod putIfAbsent accesses from 40 threads (1 process, 1 remote cache instance), 40000 entries total
> After the test run, the numberOfEntries on each node are:
> * node1: 20074
> * node2: 19888
> * node3: 20114
> * node4: 18885
> Total is 78961, 1039 entries are missing. No error on HotRod client side so 80000 entries should be there.
> Let's take a look at example missing entry, hash(thread01key151) = 7c29bccb.
> Current CH: owners(7c29bccb) are [node1, node2]
> Pending CH: owners(7c29bccb) are [node1, node2, node4]
> Balanced CH: owners(7c29bccb) are [node1, node4]
> The events sequence is:
> * hotrod -> node1
> * node1 -> node2, node4
> * node2 committed entry
> * node4 performed clustered get before write, got a value from node2 and will not commit the entry because this node thinks it's not changed/created
> * node1 committed entry
> * node2 invalidates the entry because it's no longer an owner
> Result owners(7c29bccb) are only node1 and node4 is missing. This entry may be completely lost by further rebalances when node4 is donor for this segment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 8 months
[JBoss JIRA] (ISPN-3403) NullPointerException when invoking JMX operation with debug logging enabled
by Fabian Stäber (JIRA)
[ https://issues.jboss.org/browse/ISPN-3403?page=com.atlassian.jira.plugin.... ]
Fabian Stäber updated ISPN-3403:
--------------------------------
Git Pull Request: https://github.com/infinispan/infinispan/pull/2004
Description:
When the CLI client runs the "cache ___defaultcache" command to select a cache, the first things that happens is that the "createSessionId" call is invoked via JMX on the server.
When the command is run for the first time, there is no previously selected cache. Therefore, the "active cache" parameter for the "createSessionId" command is null.
On the server side, in ResourceDMBean.invoke(), there is a debug log message that would log the class of the arguments received using args[i].getClass(). As args[i] is null for the "active cache" parameter, the call fails with a NullPointerException.
As a result, it is not possible to select a cache with the CLI client when debug logging is enabled in ResourceDMBean.
---
I have created a very small pull request for this on GitHub.
was:
When the CLI client runs the "cache ___defaultcache" command to select a cache, the first things that happens is that the "createSessionId" call is invoked via JMX on the server.
When the command is run for the first time, there is no previously selected cache. Therefore, the "active cache" parameter for the "createSessionId" command is null.
On the server side, in ResourceDMBean.invoke(), there is a debug log message that would log the class of the arguments received using args[i].getClass(). As args[i] is null for the "active cache" parameter, the call fails with a NullPointerException.
As a result, it is not possible to select a cache with the CLI client when debug logging is enabled in ResourceDMBean.
---
I will create a very small pull request for this on GitHub.
> NullPointerException when invoking JMX operation with debug logging enabled
> ---------------------------------------------------------------------------
>
> Key: ISPN-3403
> URL: https://issues.jboss.org/browse/ISPN-3403
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 6.0.0.Alpha2
> Reporter: Fabian Stäber
> Assignee: Mircea Markus
>
> When the CLI client runs the "cache ___defaultcache" command to select a cache, the first things that happens is that the "createSessionId" call is invoked via JMX on the server.
> When the command is run for the first time, there is no previously selected cache. Therefore, the "active cache" parameter for the "createSessionId" command is null.
> On the server side, in ResourceDMBean.invoke(), there is a debug log message that would log the class of the arguments received using args[i].getClass(). As args[i] is null for the "active cache" parameter, the call fails with a NullPointerException.
> As a result, it is not possible to select a cache with the CLI client when debug logging is enabled in ResourceDMBean.
> ---
> I have created a very small pull request for this on GitHub.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 8 months