[JBoss JIRA] (ISPN-1586) inconsistent cache data in replication cluster with local (not shared) cache store
11:56 a.m.
inconsistent cache data in replication cluster with local (not shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Manik Surtani
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName" value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale data
in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed the
problem in my configuration:
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where we
can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data and
new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:10 a.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
Manik Surtani reassigned ISPN-1586:
-----------------------------------
Assignee: Dan Berindei (was: Manik Surtani)
@Dex have you tried with 5.1.0.BETA5?
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:41 a.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
Galder Zamarreño commented on ISPN-1586:
----------------------------------------
Forum post indicates this happens with latest snapshots... marking CR2 as fix.
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Fix For: 5.1.0.CR2, 5.1.0.FINAL
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:41 a.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
Galder Zamarreño updated ISPN-1586:
-----------------------------------
Fix Version/s: 5.1.0.CR2
5.1.0.FINAL
Affects Version/s: 5.1.0.CR1
Forum Reference: http://community.jboss.org/message/639159#639159,
http://community.jboss.org/message/638770#638770 (was:
http://community.jboss.org/message/639159#639159,
http://community.jboss.org/message/638770#638770)
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Fix For: 5.1.0.CR2, 5.1.0.FINAL
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:56 a.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
Dan Berindei commented on ISPN-1586:
------------------------------------
This is caused by my fix for ISPN-1171. We didn't have this problem prior to that
because state transfer cleared the local cache store before applying any changes.
As a temporary fix we could restore that behaviour and force {{purgeOnStartup}} to be
enabled when {{fetchPersistentState}} is enabled. Since we already log a warning about
stale entries when the cache store is not shared and {{purgeOnStartup}} and
{{fetchPersistentState}} are both disabled, we can just remove the
{{fetchPersistentState}} check and warn about stale entries whenever {{purgeOnStartup}} is
disabled.
A proper fix should come with ISPN-1195, when we get around to implementing it.
I wouldn't use your {{cacheStore.syncState(state)}} approach, because it would force
us to hold all the state we received in memory at once so we can do a single
{{cacheStore.syncState(state)}} call. That's going against the path we started on with
5.1, receiving state from many different nodes and even splitting the state from one node
into multiple chunks with ISPN-284.
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Fix For: 5.1.0.CR2, 5.1.0.FINAL
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:43 a.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
dex chen commented on ISPN-1586:
--------------------------------
To enforce purgeOnStartup on when fetchPersistentState is enabled will NOT work for use
cases where cache stores are used as a data persistent mechanism (as in my case).
But, you might want to do purgeOnStartup ONLY on nodes (not the first node in the cluster)
joining with other nodes. In this case, the data from the first node's cache store is
loaded to cache and used as the "golden" copy. Indeed, this was the approach I
first tried to work around the problem.
I agree ISPN-1195 is related to this issue, and this issue could be addressed as a part of
ISPN-1195.
Regrading your comment on holding/passing state to cache loaders, we already holde the
whole state in StateTransferManager right now.
My suggestion is really about that we need to some how delegate the applyState() to
CacheLoader, since, I think, CacheLoader is the right place for merging data.
How much data (of state) needed to pass to CacheLoader is more or less implementation
detail and performance optimization. This way (delegation) also allows customized
CacheLoader to implement its own data merge algorithm, if needs to.
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Fix For: 5.1.0.CR2, 5.1.0.FINAL
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:55 p.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
Dan Berindei commented on ISPN-1586:
------------------------------------
{quote}To enforce purgeOnStartup on when fetchPersistentState is enabled will NOT work for
use cases where cache stores are used as a data persistent mechanism (as in my case).
But, you might want to do purgeOnStartup ONLY on nodes (not the first node in the cluster)
joining with other nodes. In this case, the data from the first node's cache store is
loaded to cache and used as the "golden" copy. Indeed, this was the approach I
first tried to work around the problem.{quote}
You're right, it would be reasonable to load from the cache store only on the first
node to start. Actually that was the behaviour before ISPN-1171 (see ISPN-370). It could
probably work the same in replicated mode and in distributed mode.
Note that sometimes the two nodes can start as two separate clusters and then merge to
form a single cluster. JGroups 3.0.1 significantly reduces the possibility of a merge
during startup, but it doesn't eliminate it completely. So until we handle merges
better (ISPN-263), there will always be a chance of stale/missing entries.
If you have the option, I would recommend using a shared cache store for persistence
instead.
{quote}Regrading your comment on holding/passing state to cache loaders, we already holde
the whole state in StateTransferManager right now.{quote}
No, we don't: even in replicated mode, a joiner will receive state from all the
existing members, and {{StateTransferManager.applyState(state)}} will be called multiple
times with different state information. With chunked state transfer even the state from a
single existing member will arive as multiple {{applyState}} calls.
The state received in those calls is directly applied to the data container (and
indirectly saved to the cache store). But we don't keep a copy of that state in
{{StateTransferManager}} to use during future {{applystate}} calls.
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Fix For: 5.1.0.CR2, 5.1.0.FINAL
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:15 a.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
dex chen commented on ISPN-1586:
--------------------------------
Dan: could you comment on my workaround (below) of this issue for replication mode?
It seems working reasonably well in my tests.
>>
if (configuration.getCacheMode().isReplicated())
{
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new state:
" + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
}
Here, I basically try to remove local (staled) entries which do not exist in state. Sure,
this is suboptimal since it may remove some entries and then add them again (due to the
checked "state").
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Fix For: 5.1.0.CR2, 5.1.0.FINAL
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:23 a.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
Dan Berindei edited comment on ISPN-1586 at 12/9/11 12:22 PM:
--------------------------------------------------------------
Dan: could you comment on my workaround (below) of this issue for replication mode?
It seems working reasonably well in my tests.
{noformat}
if (configuration.getCacheMode().isReplicated()) {
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new state:
" + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
}
{noformat}
Here, I basically try to remove local (staled) entries which do not exist in state. Sure,
this is suboptimal since it may remove some entries and then add them again (due to the
checked "state").
was (Author: dex80526):
Dan: could you comment on my workaround (below) of this issue for replication mode?
It seems working reasonably well in my tests.
>>
if (configuration.getCacheMode().isReplicated())
{
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new state:
" + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
}
Here, I basically try to remove local (staled) entries which do not exist in state. Sure,
this is suboptimal since it may remove some entries and then add them again (due to the
checked "state").
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Fix For: 5.1.0.CR2, 5.1.0.FINAL
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:11 p.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
Dan Berindei commented on ISPN-1586:
------------------------------------
Dex, your workaround will work (assuming you're preloading the entire cache store),
but I think clearing the data container and the cache store would be more efficient. I
don't think updating an entry would be faster than inserting an entry, in any cache
store, and deleting keys by hand will certainly be slower than clearing the entire cache
store.
It would be even faster if you could only preload on the first node that joins the
cluster, but unfortunately we preload before joining the cluster so at that point we
don't yet know if we're the first.
It should be possible to change the order: join the cluster, clear the cache store if
we're joining an existing cluster (regardless of how {{purgeOnStartup}} is
configured), do the state transfer, then preload (if we haven't clear the cache store
already).
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Fix For: 5.1.0.CR2, 5.1.0.FINAL
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:15 p.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
Dan Berindei updated ISPN-1586:
-------------------------------
Description:
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
{code:xml}
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName" value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
{code}
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale data
in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
{code:java}
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
{code}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed the
problem in my configuration:
{code:java}
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
{code}
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where we
can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data and
new state, and apply the changes there.
was:
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName" value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale data
in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed the
problem in my configuration:
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where we
can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data and
new state, and apply the changes there.
Forum Reference: http://community.jboss.org/message/639159#639159,
http://community.jboss.org/message/638770#638770 (was:
http://community.jboss.org/message/639159#639159,
http://community.jboss.org/message/638770#638770)
> inconsistent cache data in replication cluster with local (not shared) cache store
> ----------------------------------------------------------------------------------
>
> Key: ISPN-1586
> URL: https://issues.jboss.org/browse/ISPN-1586
> Project: Infinispan
> Issue Type: Bug
> Components: Core API
> Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
> Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
> Java 1.7
> Linux Cent OS
> Reporter: dex chen
> Assignee: Dan Berindei
> Fix For: 5.1.0.CR2, 5.1.0.FINAL
>
>
> I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot
code.
> It is configured in "replication", using local cache store, and
preload=true, purgeOnStartup=false .. (see the whole config below).
> I will get the inconsistent data among the nodes in the following scenario:
> 1) start 2 node cluster
> 2) after the cluster is formed, add some data to the cache
> k1-->v1
> k2-->v2
> I will see the data replication working perfectly at this point.
> 3) bring node 2 down
> 4) delete entry k1-->v1 through node1
> Note: At this point, on the local (persistent) cache store on the node2 have 2
entries.
> 5) start node2, and wait to join the cluster
> 6) after state merging, you will see now that node1 has 1 entry and nod2 has 2
entries.
> I am expecting that the data should be consistent across the cluster.
> Here is the infinispan config:
> {code:xml}
> <infinispan
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
> xmlns="urn:infinispan:config:5.0">
> <global>
> <transport clusterName="demoCluster"
> machineId="node1"
> rackId="r1" nodeName="dexlaptop"
> > <properties>
> <property name="configurationFile"
value="./jgroups-tcp.xml" />
> </properties>
> </transport>
> <globalJmxStatistics enabled="true"/>
> </global>
> <default>
> <locking
> isolationLevel="READ_COMMITTED"
> lockAcquisitionTimeout="20000"
> writeSkewCheck="false"
> concurrencyLevel="5000"
> useLockStriping="false"
> />
> <jmxStatistics enabled="true"/>
> <clustering mode="replication">
> <stateRetrieval
> timeout="240000"
> fetchInMemoryState="true"
> alwaysProvideInMemoryState="false"
> />
> <!--
> Network calls are synchronous.
> -->
> <sync replTimeout="20000"/>
> </clustering>
> <loaders
> passivation="false"
> shared="false"
> preload="true">
> <loader
>
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
> fetchPersistentState="true"
> purgeOnStartup="false">
> <!-- set to true for not first node in the cluster in testing/demo
-->
> <properties>
> <property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
> <property name="idColumnName"
value="ID_COLUMN"/>
> <property name="dataColumnName"
value="DATA_COLUMN"/>
> <property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
> <property name="timestampColumnType"
value="BIGINT"/>
> <property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
> <property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
> <property name="userName" value="sa"/>
> <property name="driverClass"
value="org.h2.Driver"/>
> <property name="idColumnType"
value="VARCHAR(255)"/>
> <property name="dataColumnType"
value="BINARY"/>
> <property name="dropTableOnExit"
value="false"/>
> <property name="createTableOnStart"
value="true"/>
> </properties>
> <!--
> <async enabled="false" />
> -->
> </loader>
> </loaders>
> </default>
> </infinispan>
> {code}
> Basically, current ISPN implementation in state transfer will result in data
insistence among nodes in replication mode and each node has local cache store.
> I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
> Here is the code snipt of applyState():
> {code:java}
> public void applyState(Collection<InternalCacheEntry> state,
> Address sender, int viewId) throws InterruptedException {
> .....
>
> for (InternalCacheEntry e : state) {
> InvocationContext ctx = icc.createInvocationContext(false, 1);
> // locking not necessary as during rehashing we block all transactions
> ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
> SKIP_OWNERSHIP_CHECK);
> try {
> PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(),
e.getValue(), e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
> interceptorChain.invoke(ctx, put);
> } catch (Exception ee) {
> log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
> }
> }
>
> ...
> }
> {code}
> As we can see that the code bascically try to add all data entryies got from the
cluster (other node). Hence, it does not know any previous entries were deleted from the
cluster which exist in its local cache store. This is exactly my test case (my
confiuration is that each node has its own cache store and in replication mode).
> To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
> I modified the above method by adding the following code before put loop, and it
fixed the problem in my configuration:
> {code:java}
> //Remove entries which no loger exist in the new state from local cache/cache store
> for (InternalCacheEntry ie: dataContainer.entrySet()) {
>
> if (!state.contains(ie)) {
> log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
> InvocationContext ctx = icc.createInvocationContext(false, 1);
> // locking not necessary as during rehashing we block all transactions
> ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
> SKIP_OWNERSHIP_CHECK);
> try {
> RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(),
ie.getValue(), ctx.getFlags());
> interceptorChain.invoke(ctx, remove);
> dataContainer.remove(ie.getKey());
> } catch (Exception ee) {
> log.error("failed to delete local store entry", ee);
> }
> }
> }
> ...
> {code}
> Obvious, the above "fix" is based on assumption/configure that
dataContainer will have all local entries, i.e., preload=true, no enviction replication.
> The real fix, I think, we need delegate the syncState(state) to cache store impl,
where we can check the configurations and do the right thing.
> For example, in the cache store impl, we can calculate the changes based on local
data and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:35 p.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
dex chen commented on ISPN-1586:
--------------------------------
Dan, thanks for the response. I did look into clearing the local cache when joining. As
you pointed the current order (pre-load first, then join) does not allow this; there is no
hook to do this at CacheLoader level. How to change the order? I was looking at some
thing like Interceptor stuff earlier, but ISPN does not expose the interceptor either.
Anyway, we are planning to use ISPN in our project, and this issue is kind of "show
blocker". Is possible to get some fix (even sub-optimal) of this issue to the 5.1
final release?
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Fix For: 5.1.0.CR2, 5.1.0.FINAL
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
{code:xml}
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
{code}
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
{code:java}
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
{code}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
{code:java}
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
{code}
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:05 a.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
Galder Zamarreño commented on ISPN-1586:
----------------------------------------
Dan, what's up with this? Are you deferring? Or fixing it for CR2?
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Fix For: 5.1.0.CR2, 5.1.0.FINAL
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
{code:xml}
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
{code}
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
{code:java}
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
{code}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
{code:java}
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
{code}
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:23 p.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
Galder Zamarreño updated ISPN-1586:
-----------------------------------
Fix Version/s: 5.1.0.CR3
(was: 5.1.0.CR2)
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Fix For: 5.1.0.CR3, 5.1.0.FINAL
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
{code:xml}
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
{code}
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
{code:java}
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
{code}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
{code:java}
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
{code}
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:21 p.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
Galder Zamarreño updated ISPN-1586:
-----------------------------------
Fix Version/s: (was: 5.1.0.CR3)
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Fix For: 5.1.0.FINAL
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
{code:xml}
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
{code}
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
{code:java}
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
{code}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
{code:java}
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
{code}
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:19 a.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
Dan Berindei updated ISPN-1586:
-------------------------------
Fix Version/s: 5.2.0.FINAL
(was: 5.1.0.FINAL)
Forum Reference: http://community.jboss.org/message/639159#639159,
http://community.jboss.org/message/638770#638770 (was:
http://community.jboss.org/message/639159#639159,
http://community.jboss.org/message/638770#638770)
Postponing to 5.2.
Note that DefaultCacheManager.getCache() and
InboundInvocationHandlerImpl.waitForJoinToStart() will also have to change to wait for the
preload to end instead of waiting for the join to end/start.
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Fix For: 5.2.0.FINAL
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
{code:xml}
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
{code}
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
{code:java}
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
{code}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
{code:java}
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
{code}
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:56 a.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
dex chen commented on ISPN-1586:
--------------------------------
Dan: We really need this feature/fix as soon as possible. Is there any way I can help to
move this fix to 5.1? Thanks.
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Fix For: 5.2.0.FINAL
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
{code:xml}
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
{code}
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
{code:java}
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
{code}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
{code:java}
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
{code}
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:56 p.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
Manik Surtani commented on ISPN-1586:
-------------------------------------
@Dex: this is too big a change to introduce into 5.1 at this stage (we're already on
CR4). If you want, I suggest building 5.1.x from scratch and backport this patch from
5.2.0 (once it is in).
If you'd like to help implement this for 5.2.0 as well, that would be appreciated.
I'm sure Dan will be happy to provide guidance. :)
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Fix For: 5.2.0.FINAL
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
{code:xml}
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
{code}
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
{code:java}
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
{code}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
{code:java}
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
{code}
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:44 a.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-1586:
--------------------------------
Fix Version/s: 5.2.0.CR1
(was: 5.2.0.Final)
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.2.0.CR1
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
{code:xml}
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
{code}
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
{code:java}
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
{code}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
{code:java}
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
{code}
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:44 a.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-1586:
--------------------------------
Priority: Critical (was: Major)
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.2.0.CR1
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
{code:xml}
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
{code}
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
{code:java}
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
{code}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
{code:java}
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
{code}
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 a.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
Dan Berindei commented on ISPN-1586:
------------------------------------
While working on NBST I have introduced a fix for ISPN-1586 as described in my comment
(join first, then preload only on the first node) to fix some tests.
Unfortunately it had some issues, among others the fact that preload doesn't work even
if in-memory state transfer (fetchInMemoryState) is disabled. This broke
org.infinispan.query.cacheloaders.SharedCacheLoaderQueryIndexTest. We are discussing
alternatives on the mailing list: http://markmail.org/thread/w6ednao2m76rnpzm
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.2.0.CR1
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
{code:xml}
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
{code}
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
{code:java}
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
{code}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
{code:java}
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
{code}
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:37 p.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-1586:
--------------------------------
Fix Version/s: 5.2.0.CR1
(was: 5.2.0.Beta2)
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.2.0.CR1
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
{code:xml}
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
{code}
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
{code:java}
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
{code}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
{code:java}
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
{code}
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:26 p.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-1586:
--------------------------------
Fix Version/s: 5.2.0.CR1
(was: 5.2.0.Beta3)
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.2.0.CR1
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
{code:xml}
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
{code}
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
{code:java}
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
{code}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
{code:java}
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
{code}
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:39 a.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-1586:
--------------------------------
Fix Version/s: 5.2.0.CR1
(was: 5.2.0.Beta4)
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Priority: Critical
Fix For: 5.2.0.CR1
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
{code:xml}
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
{code}
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
{code:java}
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
{code}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
{code:java}
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
{code}
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:09 a.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
Mircea Markus updated ISPN-1586:
--------------------------------
Fix Version/s: 6.0.0.Final
(was: 5.2.0.CR1)
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Priority: Critical
Fix For: 6.0.0.Final
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
{code:xml}
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
{code}
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
{code:java}
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
{code}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
{code:java}
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
{code}
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:24 a.m.
[
https://issues.jboss.org/browse/ISPN-1586?page=com.atlassian.jira.plugin....
]
Dan Berindei commented on ISPN-1586:
------------------------------------
I have reverted the ISPN-1586 work with the fix for ISPN-2495. It wasn't good enough,
because the coordinator couldn't block other nodes from starting and requesting state
while it was still preloading data. Hopefully we'll have a better fix in 6.0.
inconsistent cache data in replication cluster with local (not
shared) cache store
----------------------------------------------------------------------------------
Key: ISPN-1586
URL: https://issues.jboss.org/browse/ISPN-1586
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.0.0.FINAL, 5.1.0.CR1
Environment: ISPN 5.0.0. Final and ISPN 5.1 sanpshot
Java 1.7
Linux Cent OS
Reporter: dex chen
Assignee: Dan Berindei
Priority: Critical
Fix For: 6.0.0.Final
I rerun my test (an embedded ISPN cluser) with ISPN 5.0.0. final and 5.1 Sanpshot code.
It is configured in "replication", using local cache store, and preload=true,
purgeOnStartup=false .. (see the whole config below).
I will get the inconsistent data among the nodes in the following scenario:
1) start 2 node cluster
2) after the cluster is formed, add some data to the cache
k1-->v1
k2-->v2
I will see the data replication working perfectly at this point.
3) bring node 2 down
4) delete entry k1-->v1 through node1
Note: At this point, on the local (persistent) cache store on the node2 have 2 entries.
5) start node2, and wait to join the cluster
6) after state merging, you will see now that node1 has 1 entry and nod2 has 2 entries.
I am expecting that the data should be consistent across the cluster.
Here is the infinispan config:
{code:xml}
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0
http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<transport clusterName="demoCluster"
machineId="node1"
rackId="r1" nodeName="dexlaptop"
>
<properties>
<property name="configurationFile"
value="./jgroups-tcp.xml" />
</properties>
</transport>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<locking
isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false"
/>
<jmxStatistics enabled="true"/>
<clustering mode="replication">
<stateRetrieval
timeout="240000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="false"
/>
<!--
Network calls are synchronous.
-->
<sync replTimeout="20000"/>
</clustering>
<loaders
passivation="false"
shared="false"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
purgeOnStartup="false">
<!-- set to true for not first node in the cluster in testing/demo -->
<properties>
<property name="stringsTableNamePrefix"
value="ISPN_STRING_TABLE"/>
<property name="idColumnName"
value="ID_COLUMN"/>
<property name="dataColumnName"
value="DATA_COLUMN"/>
<property name="timestampColumnName"
value="TIMESTAMP_COLUMN"/>
<property name="timestampColumnType"
value="BIGINT"/>
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>
<property name="connectionUrl"
value="jdbc:h2:file:/var/tmp/h2cachestore;DB_CLOSE_DELAY=-1"/>
<property name="userName" value="sa"/>
<property name="driverClass"
value="org.h2.Driver"/>
<property name="idColumnType"
value="VARCHAR(255)"/>
<property name="dataColumnType" value="BINARY"/>
<property name="dropTableOnExit" value="false"/>
<property name="createTableOnStart"
value="true"/>
</properties>
<!--
<async enabled="false" />
-->
</loader>
</loaders>
</default>
</infinispan>
{code}
Basically, current ISPN implementation in state transfer will result in data insistence
among nodes in replication mode and each node has local cache store.
I found code BaseStateTransferManagerImpl's applyState code does not remove stale
data in the local cache store and result in inconsistent data when joins a cluster:
Here is the code snipt of applyState():
{code:java}
public void applyState(Collection<InternalCacheEntry> state,
Address sender, int viewId) throws InterruptedException {
.....
for (InternalCacheEntry e : state) {
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
PutKeyValueCommand put = cf.buildPutKeyValueCommand(e.getKey(), e.getValue(),
e.getLifespan(), e.getMaxIdle(), ctx.getFlags());
interceptorChain.invoke(ctx, put);
} catch (Exception ee) {
log.problemApplyingStateForKey(ee.getMessage(), e.getKey());
}
}
...
}
{code}
As we can see that the code bascically try to add all data entryies got from the cluster
(other node). Hence, it does not know any previous entries were deleted from the cluster
which exist in its local cache store. This is exactly my test case (my confiuration is
that each node has its own cache store and in replication mode).
To fix this, we need to delete any entries from the local cache/cache store which no
longer exist in the new state.
I modified the above method by adding the following code before put loop, and it fixed
the problem in my configuration:
{code:java}
//Remove entries which no loger exist in the new state from local cache/cache store
for (InternalCacheEntry ie: dataContainer.entrySet()) {
if (!state.contains(ie)) {
log.debug("Try to delete local store entry no loger exists in the new
state: " + ie.getKey());
InvocationContext ctx = icc.createInvocationContext(false, 1);
// locking not necessary as during rehashing we block all transactions
ctx.setFlags(CACHE_MODE_LOCAL, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP,
SKIP_SHARED_CACHE_STORE, SKIP_LOCKING,
SKIP_OWNERSHIP_CHECK);
try {
RemoveCommand remove = cf.buildRemoveCommand(ie.getKey(), ie.getValue(),
ctx.getFlags());
interceptorChain.invoke(ctx, remove);
dataContainer.remove(ie.getKey());
} catch (Exception ee) {
log.error("failed to delete local store entry", ee);
}
}
}
...
{code}
Obvious, the above "fix" is based on assumption/configure that dataContainer
will have all local entries, i.e., preload=true, no enviction replication.
The real fix, I think, we need delegate the syncState(state) to cache store impl, where
we can check the configurations and do the right thing.
For example, in the cache store impl, we can calculate the changes based on local data
and new state, and apply the changes there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4363
days inactive
4725
days old
infinispan-issues@lists.jboss.org
25 comments
12 participants
participants (12)
-
Dan Berindei (Commented) (JIRA) -
Dan Berindei (Issue Comment Edited) (JIRA)
-
Dan Berindei (JIRA)
-
Dan Berindei (Updated) (JIRA)
-
dex chen (Commented) (JIRA)
-
dex chen (Created) (JIRA)
-
dex chen (JIRA)
-
Galder Zamarreño (Commented) (JIRA)
-
Galder Zamarreño (Updated) (JIRA)
-
Manik Surtani (Assigned) (JIRA)
-
Manik Surtani (JIRA)
-
Mircea Markus (JIRA)