5:40 p.m.
Lost cache data on insert and state transfer with async DIST cache
------------------------------------------------------------------
Key: ISPN-654
URL: https://jira.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Manik Surtani
We have a performance test that is putting 10K objects into a 4-node cluster, with each of
the nodes on separate physical machines. We are seeing loss of data after an insert and
loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:42 p.m.
New subject: [JBoss JIRA] Updated: (ISPN-654) Lost cache data on insert and state transfer with async DIST cache
[
https://jira.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.sys...
]
Erik Salter updated ISPN-654:
-----------------------------
Attachment: ISPN.txt
Lost cache data on insert and state transfer with async DIST cache
------------------------------------------------------------------
Key: ISPN-654
URL: https://jira.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Manik Surtani
Attachments: ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:42 p.m.
New subject: [JBoss JIRA] Updated: (ISPN-654) Lost cache data on insert and state transfer with async DIST cache
[
https://jira.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.sys...
]
Erik Salter updated ISPN-654:
-----------------------------
Attachment: PerformanceTest.zip
Lost cache data on insert and state transfer with async DIST cache
------------------------------------------------------------------
Key: ISPN-654
URL: https://jira.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Manik Surtani
Attachments: ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:49 a.m.
New subject: [JBoss JIRA] Updated: (ISPN-654) Lost cache data on insert and state transfer with async DIST cache
[
https://jira.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.sys...
]
Manik Surtani updated ISPN-654:
-------------------------------
Fix Version/s: 4.2.0.ALPHA2
4.2.0.Final
Lost cache data on insert and state transfer with async DIST cache
------------------------------------------------------------------
Key: ISPN-654
URL: https://jira.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Manik Surtani
Fix For: 4.2.0.ALPHA2, 4.2.0.Final
Attachments: ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:11 a.m.
New subject: [JBoss JIRA] Updated: (ISPN-654) Lost cache data on insert and state transfer with async DIST cache
[
https://jira.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.sys...
]
Manik Surtani updated ISPN-654:
-------------------------------
Assignee: Mircea Markus (was: Manik Surtani)
Priority: Critical (was: Major)
Complexity: High
Lost cache data on insert and state transfer with async DIST cache
------------------------------------------------------------------
Key: ISPN-654
URL: https://jira.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Critical
Fix For: 4.2.0.ALPHA2, 4.2.0.Final
Attachments: ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:19 a.m.
New subject: [JBoss JIRA] Updated: (ISPN-654) Lost cache data on insert and state transfer with async DIST cache
[
https://jira.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.sys...
]
Mircea Markus updated ISPN-654:
-------------------------------
Fix Version/s: 4.2.0.BETA1
(was: 4.2.0.ALPHA2)
Lost cache data on insert and state transfer with async DIST cache
------------------------------------------------------------------
Key: ISPN-654
URL: https://jira.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Critical
Fix For: 4.2.0.BETA1, 4.2.0.Final
Attachments: ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:53 a.m.
New subject: [JBoss JIRA] Commented: (ISPN-654) Lost cache data on insert and state transfer with async DIST cache
[
https://jira.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.sys...
]
Mircea Markus commented on ISPN-654:
------------------------------------
I've tried to reproduce this locally using the same code and cluster config with no
success.
Note: the call is configured to run async, so getCacheSize might return a different value
on a different cache, if replication didn't finish in time.
Can you please provide me with more details on how to set this up and reproduce it? E.g.
what as version should I use, what to build, copy, how to run etc.
Lost cache data on insert and state transfer with async DIST cache
------------------------------------------------------------------
Key: ISPN-654
URL: https://jira.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Critical
Fix For: 4.2.0.BETA1, 4.2.0.Final
Attachments: ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:22 a.m.
New subject: [JBoss JIRA] Commented: (ISPN-654) Lost cache data on insert and state transfer with async DIST cache
[
https://jira.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.sys...
]
Erik Salter commented on ISPN-654:
----------------------------------
This was run with ISPN 4.1.0.FINAL. I ran this on JBossESB 4.7 with the directions
detailed in the text file. In a nutshell, all artifacts/config were run on all 4 nodes of
the cluster.
Were there other specifics you needed?
Lost cache data on insert and state transfer with async DIST cache
------------------------------------------------------------------
Key: ISPN-654
URL: https://jira.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Critical
Fix For: 4.2.0.BETA1, 4.2.0.Final
Attachments: ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:11 p.m.
New subject: [JBoss JIRA] Commented: (ISPN-654) Lost cache data on insert and state transfer with async DIST cache
[
https://jira.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.sys...
]
Mircea Markus commented on ISPN-654:
------------------------------------
related: http://community.jboss.org/message/566530#566530
Lost cache data on insert and state transfer with async DIST cache
------------------------------------------------------------------
Key: ISPN-654
URL: https://jira.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Critical
Fix For: 4.2.0.BETA1, 4.2.0.Final
Attachments: ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:34 a.m.
New subject: [JBoss JIRA] Assigned: (ISPN-654) Lost cache data on insert and state transfer with async DIST cache
[
https://jira.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.sys...
]
Manik Surtani reassigned ISPN-654:
----------------------------------
Assignee: Manik Surtani (was: Mircea Markus)
Lost cache data on insert and state transfer with async DIST cache
------------------------------------------------------------------
Key: ISPN-654
URL: https://jira.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Manik Surtani
Priority: Critical
Fix For: 4.2.0.BETA1, 4.2.0.Final
Attachments: ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:05 a.m.
New subject: [JBoss JIRA] Commented: (ISPN-654) Lost cache data on insert and state transfer with async DIST cache
[
https://jira.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.sys...
]
Manik Surtani commented on ISPN-654:
------------------------------------
@Erik could you also pls attach your infinispan config file? I don't think it was a
part of the ZIP file.
Lost cache data on insert and state transfer with async DIST cache
------------------------------------------------------------------
Key: ISPN-654
URL: https://jira.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Manik Surtani
Priority: Critical
Fix For: 4.2.0.BETA1, 4.2.0.Final
Attachments: ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:42 a.m.
New subject: [JBoss JIRA] Updated: (ISPN-654) Lost cache data on insert and state transfer with async DIST cache
[
https://jira.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.sys...
]
Erik Salter updated ISPN-654:
-----------------------------
Attachment: ispn-654.xml
Hi Manik:
This was on the forum post. I recreated it for convenience. Thanks!
Lost cache data on insert and state transfer with async DIST cache
------------------------------------------------------------------
Key: ISPN-654
URL: https://jira.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Manik Surtani
Priority: Critical
Fix For: 4.2.0.BETA1, 4.2.0.Final
Attachments: ispn-654.xml, ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:51 p.m.
New subject: [JBoss JIRA] Updated: (ISPN-654) Lost cache data on insert and state transfer with async DIST cache
[
https://jira.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.sys...
]
Mircea Markus updated ISPN-654:
-------------------------------
Fix Version/s: 4.2.0.BETA1
(was: 4.2.0.ALPHA4)
Lost cache data on insert and state transfer with async DIST cache
------------------------------------------------------------------
Key: ISPN-654
URL: https://jira.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Manik Surtani
Priority: Critical
Fix For: 4.2.0.BETA1, 4.2.0.Final
Attachments: ispn-654.xml, ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:12 a.m.
New subject: [JBoss JIRA] Updated: (ISPN-654) Lost cache data on insert and state transfer with async DIST cache
[
https://jira.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.sys...
]
Manik Surtani updated ISPN-654:
-------------------------------
Assignee: Mircea Markus (was: Manik Surtani)
Lost cache data on insert and state transfer with async DIST cache
------------------------------------------------------------------
Key: ISPN-654
URL: https://jira.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Critical
Fix For: 4.2.0.BETA1, 4.2.0.Final
Attachments: ispn-654.xml, ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:23 a.m.
New subject: [JBoss JIRA] Updated: (ISPN-654) Lost cache data on insert and state transfer with async DIST cache
[
https://jira.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.sys...
]
Manik Surtani updated ISPN-654:
-------------------------------
Fix Version/s: 4.2.0.CR1
(was: 4.2.0.BETA1)
Lost cache data on insert and state transfer with async DIST cache
------------------------------------------------------------------
Key: ISPN-654
URL: https://jira.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Critical
Fix For: 4.2.0.CR1, 4.2.0.Final
Attachments: ispn-654.xml, ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:58 a.m.
New subject: [JBoss JIRA] Updated: (ISPN-654) Inconsistent state during nodes leaving with async DIST cache
[
https://jira.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.sys...
]
Mircea Markus updated ISPN-654:
-------------------------------
Summary: Inconsistent state during nodes leaving with async DIST cache (was: Lost
cache data on insert and state transfer with async DIST cache)
Inconsistent state during nodes leaving with async DIST cache
-------------------------------------------------------------
Key: ISPN-654
URL: https://jira.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Critical
Fix For: 4.2.0.CR1, 4.2.0.Final
Attachments: ispn-654.xml, ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:58 a.m.
New subject: [JBoss JIRA] Resolved: (ISPN-654) Inconsistent state during nodes leaving with async DIST cache
[
https://jira.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.sys...
]
Mircea Markus resolved ISPN-654.
--------------------------------
Resolution: Done
Root cause: cache.get(k) first checks if the key maps locally and if it is not then it
goes remotely. During ST the k might be mapped locally but it might not have been fetched
from a remote node. Fix was to check weather ST happens right now and if cannot find key
locally I also go remotely
Inconsistent state during nodes leaving with async DIST cache
-------------------------------------------------------------
Key: ISPN-654
URL: https://jira.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Critical
Fix For: 4.2.0.CR1, 4.2.0.Final
Attachments: ispn-654.xml, ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:20 a.m.
New subject: [JBoss JIRA] Commented: (ISPN-654) Inconsistent state during nodes leaving with async DIST cache
[
https://issues.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.s...
]
Erik Salter commented on ISPN-654:
----------------------------------
We're still seeing errors with a 3-node cluster:
Test 1:
1. Create 60K of "BigObject" data in the cache. (The WS call is
"batchIngestObjects")
2. Shutdown a node.
3. Try to retrieve all of the objects from the remaining nodes
("getObjectsFromCache").
We see:
07:39:38,288 ERROR [InvocationContextInterceptor] Execution error:
org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses!
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:417)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:101)
at
org.infinispan.distribution.DistributionManagerImpl.retrieveFromRemoteSource(DistributionManagerImpl.java:341)
at
org.infinispan.interceptors.DistributionInterceptor.realRemoteGet(DistributionInterceptor.java:143)
at
org.infinispan.interceptors.DistributionInterceptor.remoteGetAndStoreInL1(DistributionInterceptor.java:131)
at
org.infinispan.interceptors.DistributionInterceptor.visitGetKeyValueCommand(DistributionInterceptor.java:97)
at
org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:59)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.LockingInterceptor.visitGetKeyValueCommand(LockingInterceptor.java:150)
at
org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:59)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at
org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:88)
at
org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:59)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at
org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:88)
at
org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:59)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:169)
at
org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:160)
at
org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:59)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:87)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:58)
at
org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:88)
at
org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:59)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273)
at org.infinispan.CacheDelegate.get(CacheDelegate.java:207)
at
net.beaumaris.InfinispanQueryPerformanceWS.service.impl.PerformanceServiceImpl.getObjectsFromCache(PerformanceServiceImpl.java:301)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy99.getObjectsFromCache(Unknown Source)
at
net.beaumaris.InfinispanQueryPerformanceWS.PerformanceWSImpl.getObjectsFromCache(PerformanceWSImpl.java:230)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.jboss.wsf.container.jboss42.InvocationHandlerJSE.invoke(InvocationHandlerJSE.java:102)
at
org.jboss.ws.core.server.ServiceEndpointInvoker.invoke(ServiceEndpointInvoker.java:221)
at
org.jboss.wsf.stack.jbws.RequestHandlerImpl.processRequest(RequestHandlerImpl.java:466)
at
org.jboss.wsf.stack.jbws.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:284)
at org.jboss.wsf.stack.jbws.RequestHandlerImpl.doPost(RequestHandlerImpl.java:201)
at
org.jboss.wsf.stack.jbws.RequestHandlerImpl.handleHttpRequest(RequestHandlerImpl.java:134)
at org.jboss.wsf.stack.jbws.EndpointServlet.service(EndpointServlet.java:84)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
at java.lang.Thread.run(Thread.java:619)
Inconsistent state during nodes leaving with async DIST cache
-------------------------------------------------------------
Key: ISPN-654
URL: https://issues.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Critical
Fix For: 4.2.0.CR1, 4.2.0.Final
Attachments: ispn-654.xml, ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:29 a.m.
New subject: [JBoss JIRA] Commented: (ISPN-654) Inconsistent state during nodes leaving with async DIST cache
[
https://issues.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.s...
]
Erik Salter commented on ISPN-654:
----------------------------------
Test 2:
The rehashing takes a long time, and can exhaust memory.
We use those steps to record the rehash used time and memory usage:
2. Ingest that "BigObject"
3. Shutdown one node. Record leave rehash used time and memory usage.
4. Restart the node. Record join rehash used time and memory usage.
For 60K objects, a LEAVE rehash uses about 200s, JOIN rehash uses about 100s.
For 90K objects, a LEAVE rehash uses about 270s, JOIN rehash uses about 320s.
With the 90K objects case, we're seeing OOMs on a JOIN rehash:
07:48:55,993 INFO [InfinispanCacheDeployer] Initialize cache cache_nojta_sync
07:48:56,005 INFO [TransactionLoggerImpl] Starting transaction logging
07:53:59,521 ERROR [JoinTask] Caught exception!
org.infinispan.CacheException: Remote (iptv-srm-26284) failed unexpectedly
at
org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:74)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:414)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:101)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:125)
at org.infinispan.distribution.JoinTask.performRehash(JoinTask.java:113)
at org.infinispan.distribution.RehashTask.call(RehashTask.java:53)
at org.infinispan.distribution.RehashTask.call(RehashTask.java:33)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.infinispan.io.ExposedByteArrayOutputStream.write(ExposedByteArrayOutputStream.java:90)
at org.jboss.marshalling.Marshalling$6.write(Marshalling.java:378)
at org.jboss.marshalling.UTFUtils.writeUTFBytes(UTFUtils.java:129)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:328)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1140)
at
org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1096)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:966)
at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:423)
at
org.infinispan.container.entries.ImmortalCacheValue$Externalizer.writeObject(ImmortalCacheValue.java:99)
at
org.infinispan.marshall.jboss.ConstantObjectTable$ExternalizerAdapter.writeObject(ConstantObjectTable.java:322)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:147)
at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:423)
at org.infinispan.marshall.MarshallUtil.marshallMap(MarshallUtil.java:59)
at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:61)
at
org.infinispan.marshall.jboss.ConstantObjectTable$ExternalizerAdapter.writeObject(ConstantObjectTable.java:322)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:147)
at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:423)
at
org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.writeObject(SuccessfulResponse.java:59)
at
org.infinispan.marshall.jboss.ConstantObjectTable$ExternalizerAdapter.writeObject(ConstantObjectTable.java:322)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:147)
at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:423)
at
org.infinispan.marshall.jboss.GenericJBossMarshaller.objectToObjectStream(GenericJBossMarshaller.java:98)
at
org.infinispan.marshall.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:93)
at org.infinispan.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:31)
at
org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:22)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:595)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:489)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:365)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771)
at org.jgroups.JChannel.up(JChannel.java:1465)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:954)
at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:430)
07:53:59,525 INFO [TransactionLoggerImpl] Stopping transaction logging
07:53:59,525 INFO [JoinTask] iptv-srm-3-39091 completed join rehash!
NOTE: In our environment, every time a node is restarted, it gets an available IP address,
which may not be the one it previously had. I don't know how this affects the hash
code of the underlying JGroupAddress's UUID -- if at all.
Inconsistent state during nodes leaving with async DIST cache
-------------------------------------------------------------
Key: ISPN-654
URL: https://issues.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Critical
Fix For: 4.2.0.CR1, 4.2.0.Final
Attachments: ispn-654.xml, ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:33 a.m.
New subject: [JBoss JIRA] Updated: (ISPN-654) Inconsistent state during nodes leaving with async DIST cache
[
https://issues.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.s...
]
Manik Surtani updated ISPN-654:
-------------------------------
Fix Version/s: 4.2.1.Final
Inconsistent state during nodes leaving with async DIST cache
-------------------------------------------------------------
Key: ISPN-654
URL: https://issues.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Critical
Fix For: 4.2.0.CR1, 4.2.0.Final, 4.2.1.Final
Attachments: ispn-654.xml, ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:19 a.m.
New subject: [JBoss JIRA] Commented: (ISPN-654) Inconsistent state during nodes leaving with async DIST cache
[
https://issues.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.s...
]
Richard Qin commented on ISPN-654:
----------------------------------
I have the same rehashing problem commented by Erik Salter. Will this issue be reopened
and fixed in 4.2.1 version?
Inconsistent state during nodes leaving with async DIST cache
-------------------------------------------------------------
Key: ISPN-654
URL: https://issues.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Critical
Fix For: 4.2.0.CR1, 4.2.0.Final, 4.2.1.Final
Attachments: ispn-654.xml, ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:27 a.m.
New subject: [JBoss JIRA] Updated: (ISPN-654) Inconsistent state during nodes leaving with async DIST cache
[
https://issues.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.s...
]
Manik Surtani updated ISPN-654:
-------------------------------
Fix Version/s: 4.2.1.CR1
(was: 4.2.0.Final)
(was: 4.2.0.CR1)
Inconsistent state during nodes leaving with async DIST cache
-------------------------------------------------------------
Key: ISPN-654
URL: https://issues.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Critical
Fix For: 4.2.1.CR1, 4.2.1.Final
Attachments: ispn-654.xml, ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:31 a.m.
New subject: [JBoss JIRA] Commented: (ISPN-654) Inconsistent state during nodes leaving with async DIST cache
[
https://issues.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.s...
]
Manik Surtani commented on ISPN-654:
------------------------------------
@Richard try it on 4.2.1-SNAPSHOT and 4.2.1.CR1 (being released as I type)
Inconsistent state during nodes leaving with async DIST cache
-------------------------------------------------------------
Key: ISPN-654
URL: https://issues.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Critical
Fix For: 4.2.1.CR1, 4.2.1.Final
Attachments: ispn-654.xml, ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:39 a.m.
New subject: [JBoss JIRA] Commented: (ISPN-654) Inconsistent state during nodes leaving with async DIST cache
[
https://issues.jboss.org/browse/ISPN-654?page=com.atlassian.jira.plugin.s...
]
Richard Qin commented on ISPN-654:
----------------------------------
thank Manik Surtani for quick response, I'll test it on 4.2.1-SNAPSHOT and wish have a
good result.
Inconsistent state during nodes leaving with async DIST cache
-------------------------------------------------------------
Key: ISPN-654
URL: https://issues.jboss.org/browse/ISPN-654
Project: Infinispan
Issue Type: Bug
Affects Versions: 4.1.0.Final
Reporter: Erik Salter
Assignee: Mircea Markus
Priority: Critical
Fix For: 4.2.1.CR1, 4.2.1.Final
Attachments: ispn-654.xml, ISPN.txt, PerformanceTest.zip
We have a performance test that is putting 10K objects into a 4-node cluster, with each
of the nodes on separate physical machines. We are seeing loss of data after an insert
and loss of a data node.
The test directions and code are attached to this JIRA.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5086
days inactive
5211
days old
infinispan-issues@lists.jboss.org
23 comments
4 participants
participants (4)
-
Erik Salter (JIRA) -
Manik Surtani (JIRA)
-
Mircea Markus (JIRA)
-
Richard Qin (JIRA)