[infinispan-issues] [JBoss JIRA] (ISPN-3431) Lost version info during state transfer causes overwtite of newer data that joining node has read from store
RH Bugzilla Integration (JIRA)
jira-events at lists.jboss.org
Wed Sep 11 13:01:04 EDT 2013
[ https://issues.jboss.org/browse/ISPN-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803609#comment-12803609 ]
RH Bugzilla Integration commented on ISPN-3431:
-----------------------------------------------
Randy <randythomas34 at gmail.com> made a comment on [bug 1006998|https://bugzilla.redhat.com/show_bug.cgi?id=1006998]
The problem starts with three remote nodes running in replicated mode. Each node is using a local cache store from which data is loaded upon startup.
1 Start node1, node2, node3
2 Put A⇒A1 from node1
3 Put B⇒B1 from node2
4 Put C⇒C1 from node3
5 Gracefully shutdown node1 saving the data
6 Gracefully shutdown node2 saving the data
7 Put C⇒C2 from node3
8 Gracefully shutdown node3 saving the data
9 Start node1 loading the data
10 Start node2 loading the data
11 Start node3 loading the data
12 Loaded from store on node3 - version information shows its newer than nodes 1 and 2 have: org.infinispan.interceptors.CallInterceptor Executing command: PutKeyValueCommandC, value=C2, flags=[CACHE_MODE_LOCAL, SKIP_LOCKING, SKIP_CACHE_STORE, SKIP_INDEXING, SKIP_OWNERSHIP_CHECK, IGNORE_RETURN_VALUES], putIfAbsent=false, metadata=EmbeddedMetadataSimpleClusteredVersion{topologyId=8, version=2}, successful=true, ignorePreviousValue=false.
13 During state transfer older value C1 is received with version information being null: org.infinispan.statetransfer.StateTransferInterceptor handleNonTxWriteCommand for command PutKeyValueCommandC, value=C1, flags=[CACHE_MODE_LOCAL, SKIP_REMOTE_LOOKUP, PUT_FOR_STATE_TRANSFER, SKIP_SHARED_CACHE_STORE, SKIP_OWNERSHIP_CHECK, IGNORE_RETURN_VALUES, SKIP_XSITE_BACKUP], putIfAbsent=false, metadata=EmbeddedMetadatanull, successful=true, ignorePreviousValue=false
14 As a result newer value is replaced with older: org.infinispan.interceptors.EntryWrappingInterceptor About to commit entry ClusteredRepeatableReadEntry(323f265b)C, value=C1, oldValue=C2, isCreated=false, isChanged=true, isRemoved=false, isValid=true, skipRemoteGet=true, metadata=EmbeddedMetadataSimpleClusteredVersion{topologyId=8, version=2}
Actual results
The old value for entry C replaces the newer value.
Expected results
The new value of C should be retained on node3 and overwrite the old value of C on nodes 1 and 2.
How to Reproduced with the Attached archive
Please find attached archive with the Maven project that contains code implementing the scenario and bundling all configuration files for Infinispan and JGroups.
To run the scenario:
1 Start node1: java -Djava.net.preferIPv4Stack=true -Dignore.bind.address=true -Djgroups.tcpping.initial_hosts=localhost[7800] -Dinfinispan.store_name=A -jar infinispan-proto-5-1.0-SNAPSHOT.jar A
2 Start node2 in separate console: java -Djava.net.preferIPv4Stack=true -Dignore.bind.address=true -Djgroups.tcpping.initial_hosts=localhost[7800] -Djgroups.tcp.port=7801 -Dinfinispan.store_name=B -jar infinispan-proto-5-1.0-SNAPSHOT.jar B
3 Start node3 in separate console: java -Djava.net.preferIPv4Stack=true -Dignore.bind.address=true -Djgroups.tcpping.initial_hosts=localhost[7800] -Djgroups.tcp.port=7802 -Dinfinispan.store_name=C -jar infinispan-proto-5-1.0-SNAPSHOT.jar C
4 At this time all nodes will put their values to map and exit persisting data in standard Infinispan’s file store. Before node3 exists, it waits for node1 and node2 to leave and then changes its value C one again before persisting data and exiting.
5 Start node1: java -Djava.net.preferIPv4Stack=true -Dignore.bind.address=true -Djgroups.tcpping.initial_hosts=localhost[7800] -Dinfinispan.store_name=A -jar infinispan-proto-5-1.0-SNAPSHOT.jar A
6 Start node2 in separate console: java -Djava.net.preferIPv4Stack=true -Dignore.bind.address=true -Djgroups.tcpping.initial_hosts=localhost[7800] -Djgroups.tcp.port=7801 -Dinfinispan.store_name=B -jar infinispan-proto-5-1.0-SNAPSHOT.jar B
7 Start node3 in separate console: java -Djava.net.preferIPv4Stack=true -Dignore.bind.address=true -Djgroups.tcpping.initial_hosts=localhost[7800] -Djgroups.tcp.port=7802 -Dinfinispan.store_name=C -jar infinispan-proto-5-1.0-SNAPSHOT.jar C
8 At this time newer version of configuration written by node3 gets reset to older value and last configuration update is lost. This can be seen by “C ⇒ C1 ←--------- Should be C2” in console for node3 (well, as in other consoles too) (Delete file store content to re-run scenario again.)
> Lost version info during state transfer causes overwtite of newer data that joining node has read from store
> ------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-3431
> URL: https://issues.jboss.org/browse/ISPN-3431
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 6.0.0.Alpha3
> Environment: uname -s ; cat /etc/redhat-release ; java -version
> Linux
> Red Hat Enterprise Linux Server release 5.9 (Tikanga)
> java version "1.6.0_30"
> Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)
> Reporter: Nikolay Martynov
> Assignee: Pedro Ruivo
> Priority: Critical
> Fix For: 6.0.0.CR1
>
>
> When state transfer sends data to newly joining node no version information is provided despite the fact that this information is available to the cluster (for example, loaded from store). When newly joining node has loaded from its store newer version of data than cluster has then this newer version of data is overwritten because no version information is provided during state transfer.
> Scenario:
> {noformat}
> 1. Start node1, node2, node3
> 2. Put {A=>A1} from node1
> 3. Put {B=>B1} from node2
> 4. Put {C=>C1} from node3
> 5. Gracefully shutdown node1 saving the data
> 6. Gracefully shutdown node2 saving the data
> 7. Put {C=>C2} from node3
> 8. Gracefully shutdown node3 saving the data
> 9. Start node1 loading the data
> 10. Start node2 loading the data
> 11. Start node3 loading the data
> {noformat}
> Loaded from store on node3 - version information shows its newer than nodes 1 and 2 have:
> {noformat}
> org.infinispan.interceptors.CallInterceptor
> Executing command: PutKeyValueCommand{key=C, value=C2, flags=[CACHE_MODE_LOCAL, SKIP_LOCKING, SKIP_CACHE_STORE, SKIP_INDEXING, SKIP_OWNERSHIP_CHECK, IGNORE_RETURN_VALUES], putIfAbsent=false, metadata=EmbeddedMetadata{version=SimpleClusteredVersion{topologyId=8, version=2}}, successful=true, ignorePreviousValue=false}.
> {noformat}
> During state transfer older value C1 is received with version information being null:
> {noformat}
> org.infinispan.statetransfer.StateTransferInterceptor
> handleNonTxWriteCommand for command PutKeyValueCommand{key=C, value=C1, flags=[CACHE_MODE_LOCAL, SKIP_REMOTE_LOOKUP, PUT_FOR_STATE_TRANSFER, SKIP_SHARED_CACHE_STORE, SKIP_OWNERSHIP_CHECK, IGNORE_RETURN_VALUES, SKIP_XSITE_BACKUP], putIfAbsent=false, metadata=EmbeddedMetadata{version=null}, successful=true, ignorePreviousValue=false}
> {noformat}
> As a result newer value is replaced with older:
> {noformat}
> org.infinispan.interceptors.EntryWrappingInterceptor
> About to commit entry ClusteredRepeatableReadEntry(323f265b){key=C, value=C1, oldValue=C2, isCreated=false, isChanged=true, isRemoved=false, isValid=true, skipRemoteGet=true, metadata=EmbeddedMetadata{version=SimpleClusteredVersion{topologyId=8, version=2}}}
> {noformat}
> As far as i can see, any key on any node that it didnt written but has received during resync has version=null after loading from store.
> Config:
> {code:xml}
> <?xml version="1.0" encoding="UTF-8"?>
> <!--
> This is just a very simplistic example configuration file. For more information, please see
> http://docs.jboss.org/infinispan/5.3/configdocs/infinispan-config-5.3.html
> -->
> <infinispan
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="urn:infinispan:config:6.0 http://docs.jboss.org/infinispan/schemas/infinispan-config-6.0.xsd"
> xmlns="urn:infinispan:config:6.0">
> <global>
> <globalJmxStatistics enabled="true" jmxDomain="Infinispan" />
> <transport>
> <properties>
> <property name="configurationFile" value="jgroups.xml" />
> </properties>
> </transport>
> </global>
> <namedCache name="routing_table">
> <clustering mode="REPL">
> <stateTransfer awaitInitialTransfer="true" fetchInMemoryState="true"/>
> <sync replTimeout="1000"/>
> <!--async useReplQueue="true" replQueueMaxElements="100" /-->
> </clustering>
> <loaders passivation="false" shared="false" preload="true">
> <singleFileStore
> location="target/routing_table${infinispan.store_name:not_specified}/">
> </singleFileStore>
> </loaders>
> <versioning enabled="true" versioningScheme="SIMPLE"/>
> <transaction
> transactionMode="TRANSACTIONAL"
> autoCommit="true"
> transactionManagerLookupClass="org.infinispan.transaction.lookup.GenericTransactionManagerLookup"
> lockingMode="OPTIMISTIC"
> />
> <locking
> writeSkewCheck="true"
> isolationLevel="REPEATABLE_READ"/>
> </namedCache>
> </infinispan>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the infinispan-issues
mailing list