[JBoss JIRA] (ISPN-3712) HotRod Rolling Upgrades from Delirium 5.2.4 to latest server needs dropped org.infinispan.util.ByteArrayKey
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3712?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-3712:
-----------------------------------------------
Tomas Sykora <tsykora(a)redhat.com> changed the Status of [bug 1030485|https://bugzilla.redhat.com/show_bug.cgi?id=1030485] from ON_QA to ASSIGNED
> HotRod Rolling Upgrades from Delirium 5.2.4 to latest server needs dropped org.infinispan.util.ByteArrayKey
> -----------------------------------------------------------------------------------------------------------
>
> Key: ISPN-3712
> URL: https://issues.jboss.org/browse/ISPN-3712
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 6.0.0.CR1
> Environment: Processing HotRod rolling upgrades from source Infinispan 'Delirium' 5.2.4.Final-redhat-1 server to the latest Infinispan 'Infinium' 6.0.0-SNAPSHOT. Using old standalone.xml for source node and examples/standalone-hotrod-rolling-upgrade.xml for new target server.
> Reporter: Tomas Sykora
> Assignee: Tristan Tarrant
> Labels: 620
> Fix For: 6.0.0.Final
>
>
> When it is running from 6.0.0-SNAPSHOT to 6.0.0-SNAPSHOT it is ok. It looks like some back-porting compatibility issue.
> See stacktrace:
> javax.management.MBeanException
> at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:273)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:791)
> at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.invoke(PluggableMBeanServerImpl.java:527)
> at org.jboss.as.jmx.PluggableMBeanServerImpl.invoke(PluggableMBeanServerImpl.java:263)
> at org.jboss.remotingjmx.protocol.v2.ServerProxy$InvokeHandler.handle(ServerProxy.java:915)
> at org.jboss.remotingjmx.protocol.v2.ServerCommon$MessageReciever$1.run(ServerCommon.java:152)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:271)
> ... 9 more
> Caused by: org.infinispan.commons.CacheException: java.lang.ClassNotFoundException: org.infinispan.util.ByteArrayKey from [Module "org.infinispan:main" from local module loader @4099a992 (finder: local module finder @284bd160 (roots: /home/tsykora/programs/eclipseWorkspace/Infinispan-server-tsykora/infinispan-server/testsuite/target/server/node1/modules,/home/tsykora/programs/eclipseWorkspace/Infinispan-server-tsykora/infinispan-server/testsuite/target/server/node1/modules/system/layers/base))]
> at org.infinispan.persistence.remote.upgrade.HotRodTargetMigrator.synchronizeData(HotRodTargetMigrator.java:63)
> at org.infinispan.upgrade.RollingUpgradeManager.synchronizeData(RollingUpgradeManager.java:59)
> ... 14 more
> Caused by: java.lang.ClassNotFoundException: org.infinispan.util.ByteArrayKey from [Module "org.infinispan:main" from local module loader @4099a992 (finder: local module finder @284bd160 (roots: /home/tsykora/programs/eclipseWorkspace/Infinispan-server-tsykora/infinispan-server/testsuite/target/server/node1/modules,/home/tsykora/programs/eclipseWorkspace/Infinispan-server-tsykora/infinispan-server/testsuite/target/server/node1/modules/system/layers/base))]
> at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:190)
> at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:468)
> at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:456)
> at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
> at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:120)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at org.jboss.marshalling.AbstractClassResolver.loadClass(AbstractClassResolver.java:135)
> at org.jboss.marshalling.AbstractClassResolver.resolveClass(AbstractClassResolver.java:116)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:947)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1259)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadCollectionObject(RiverUnmarshaller.java:184)
> at org.jboss.marshalling.river.RiverUnmarshaller.readCollectionData(RiverUnmarshaller.java:777)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:656)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
> at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
> at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:140)
> at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromByteBuffer(AbstractJBossMarshaller.java:118)
> at org.infinispan.commons.marshall.AbstractMarshaller.objectFromByteBuffer(AbstractMarshaller.java:82)
> at org.infinispan.persistence.remote.upgrade.HotRodTargetMigrator.synchronizeData(HotRodTargetMigrator.java:61)
> ... 15 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 3 months
[JBoss JIRA] (ISPN-3838) L1 entry added by ST when already invalidated
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-3838?page=com.atlassian.jira.plugin.... ]
Work on ISPN-3838 started by William Burns.
> L1 entry added by ST when already invalidated
> ---------------------------------------------
>
> Key: ISPN-3838
> URL: https://issues.jboss.org/browse/ISPN-3838
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 6.0.0.Final
> Reporter: Radim Vansa
> Assignee: William Burns
> Priority: Critical
>
> Non-transactional cache with L1 enabled. Node A is losing ownership of an entry, the entry is not removed during ST but is going to L1.
> 1. ST builds the invalidation command, EntryWrapping interceptor starts committing all the entries
> 2. Write on primary owner (B) occurs
> 3. A gets the InvalidateL1Command, removes the ImmortalCacheEntry from data container (as it does not own the entry anymore)
> 4. The ST invalidation command commits the MortalCacheEntry with old value, storing it into the data container.
> Result: Outdated value is in L1 cache.
> As the entry is not locked during the ST, it can be committed as MortalCacheEntry only if it was not changed (removed and possibly then cached again with different value).
> (I understand that this wouldn't be easy to implement as the check is not to be executed in perform, but during the actual commit - and atomically in the container.)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 4 months
[JBoss JIRA] (ISPN-3704) StateTransfer's PutKeyValueCommand may trigger remote get
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/ISPN-3704?page=com.atlassian.jira.plugin.... ]
Radim Vansa commented on ISPN-3704:
-----------------------------------
[~dan.berindei]: Sorry, I don't have the logs where the remote get occurred anymore. But I don't think that there were two parallel rebalances.
> StateTransfer's PutKeyValueCommand may trigger remote get
> ---------------------------------------------------------
>
> Key: ISPN-3704
> URL: https://issues.jboss.org/browse/ISPN-3704
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 6.0.0.CR1
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
> Labels: 620
> Fix For: 6.1.0.Final
>
>
> In TX mode with write skew check on, ST executing puts may trigger a remote get.
> The condition in TxDistributionInterceptor.handleTxWriteCommand should probably be switched from
> {code}
> if (ctx.isOriginLocal() && !skipRemoteGet || command.isConditional() || shouldFetchRemoteValuesForWriteSkewCheck(ctx, command))
> remoteGetBeforeWrite(ctx, command, recipientGenerator);
> {code}
> to
> {code}
> if (!skipRemoteGet && (ctx.isOriginLocal() || command.isConditional() || shouldFetchRemoteValuesForWriteSkewCheck(ctx, command)))
> remoteGetBeforeWrite(ctx, command, recipientGenerator);
> {code}
> EDIT:
> I have also registered a situation where the Prepare/Commit command was executed remotely from within the ST because the topology has changed during the remote get. This should be avoided as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 4 months
[JBoss JIRA] (ISPN-3838) L1 entry added by ST when already invalidated
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/ISPN-3838?page=com.atlassian.jira.plugin.... ]
Radim Vansa commented on ISPN-3838:
-----------------------------------
Similar situation happens in transactional mode.
> L1 entry added by ST when already invalidated
> ---------------------------------------------
>
> Key: ISPN-3838
> URL: https://issues.jboss.org/browse/ISPN-3838
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 6.0.0.Final
> Reporter: Radim Vansa
> Assignee: William Burns
> Priority: Critical
>
> Non-transactional cache with L1 enabled. Node A is losing ownership of an entry, the entry is not removed during ST but is going to L1.
> 1. ST builds the invalidation command, EntryWrapping interceptor starts committing all the entries
> 2. Write on primary owner (B) occurs
> 3. A gets the InvalidateL1Command, removes the ImmortalCacheEntry from data container (as it does not own the entry anymore)
> 4. The ST invalidation command commits the MortalCacheEntry with old value, storing it into the data container.
> Result: Outdated value is in L1 cache.
> As the entry is not locked during the ST, it can be committed as MortalCacheEntry only if it was not changed (removed and possibly then cached again with different value).
> (I understand that this wouldn't be easy to implement as the check is not to be executed in perform, but during the actual commit - and atomically in the container.)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 4 months
[JBoss JIRA] (ISPN-2255) FineGrainedAtomicMap missing key, value pairs in some cluster nodes in distributed mode, embedded infinispan cache
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-2255?page=com.atlassian.jira.plugin.... ]
Work on ISPN-2255 started by Pedro Ruivo.
> FineGrainedAtomicMap missing key,value pairs in some cluster nodes in distributed mode, embedded infinispan cache
> -----------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-2255
> URL: https://issues.jboss.org/browse/ISPN-2255
> Project: Infinispan
> Issue Type: Bug
> Components: Fine-grained API
> Affects Versions: 5.1.5.FINAL
> Environment: Details about cluster:
> 1. Infinispan 5.1.5
> 2. java 7u5 64bit
> 3. linux 64bit (two nodes debian and one node ubuntu)
> 4. tomcat 7.0.28
> 5. jbossTM 4.16.0 JTA only
> Tomcat integration code is here:
> https://github.com/zvrablik/tomcatInfinispanSessionManager/tree/master/to...
> 6. all machines are on real hw ( no virtual machines), date and time is synchronized and shouldn't vary more than 1 second
> Reporter: Zdenek Henek
> Assignee: Pedro Ruivo
> Priority: Critical
> Fix For: 6.1.0.Final
>
> Attachments: fgamissueTest2.zip, jgroupsConfig.xml, sessionInfinispanConfigtestLB.xml
>
>
> I am using FineGrainedAtomicMap to store data into clustering distributed cache. When clustering set to replicated doesn't cause any issues or at least I haven't run
> into any. When I switch to distributed mode I don't see some key,value pairs in some nodes. This happens randomly. The numOwners is set to two and I have cluster of three nodes.
> more details:
> https://community.jboss.org/thread/203420?tstart=60
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 4 months
[JBoss JIRA] (ISPN-3704) StateTransfer's PutKeyValueCommand may trigger remote get
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-3704?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-3704:
------------------------------------
Do you have a log for the Prepare/Commit command for state transfer executing remotely, [~rvansa]? It shouldn't be possible for the coordinator to start a new rebalance before all the nodes have confirmed that they applied all the new state. OTOH, it may be possible that the entry was locked after the remote get, and that's what caused the Prepare/Commit command to be replicated remotely.
> StateTransfer's PutKeyValueCommand may trigger remote get
> ---------------------------------------------------------
>
> Key: ISPN-3704
> URL: https://issues.jboss.org/browse/ISPN-3704
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 6.0.0.CR1
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
> Labels: 620
> Fix For: 6.1.0.Final
>
>
> In TX mode with write skew check on, ST executing puts may trigger a remote get.
> The condition in TxDistributionInterceptor.handleTxWriteCommand should probably be switched from
> {code}
> if (ctx.isOriginLocal() && !skipRemoteGet || command.isConditional() || shouldFetchRemoteValuesForWriteSkewCheck(ctx, command))
> remoteGetBeforeWrite(ctx, command, recipientGenerator);
> {code}
> to
> {code}
> if (!skipRemoteGet && (ctx.isOriginLocal() || command.isConditional() || shouldFetchRemoteValuesForWriteSkewCheck(ctx, command)))
> remoteGetBeforeWrite(ctx, command, recipientGenerator);
> {code}
> EDIT:
> I have also registered a situation where the Prepare/Commit command was executed remotely from within the ST because the topology has changed during the remote get. This should be avoided as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 4 months
[JBoss JIRA] (ISPN-3838) L1 entry added by ST when already invalidated
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/ISPN-3838?page=com.atlassian.jira.plugin.... ]
Radim Vansa commented on ISPN-3838:
-----------------------------------
Giving it a second thought - it could be fixed by adding the key to updated keys by the InvalidateL1Command iff it is not owner (not ignoring the invalidation as we do now after ISPN-3633 / ISPN-3780, and then running the InvalidateL1Command with isPutForStateTransfer=true.
> L1 entry added by ST when already invalidated
> ---------------------------------------------
>
> Key: ISPN-3838
> URL: https://issues.jboss.org/browse/ISPN-3838
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 6.0.0.Final
> Reporter: Radim Vansa
> Assignee: William Burns
> Priority: Critical
>
> Non-transactional cache with L1 enabled. Node A is losing ownership of an entry, the entry is not removed during ST but is going to L1.
> 1. ST builds the invalidation command, EntryWrapping interceptor starts committing all the entries
> 2. Write on primary owner (B) occurs
> 3. A gets the InvalidateL1Command, removes the ImmortalCacheEntry from data container (as it does not own the entry anymore)
> 4. The ST invalidation command commits the MortalCacheEntry with old value, storing it into the data container.
> Result: Outdated value is in L1 cache.
> As the entry is not locked during the ST, it can be committed as MortalCacheEntry only if it was not changed (removed and possibly then cached again with different value).
> (I understand that this wouldn't be easy to implement as the check is not to be executed in perform, but during the actual commit - and atomically in the container.)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 4 months
[JBoss JIRA] (ISPN-3838) L1 entry added by ST when already invalidated
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/ISPN-3838?page=com.atlassian.jira.plugin.... ]
Radim Vansa commented on ISPN-3838:
-----------------------------------
Logs: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jdg-radargun-resilie...
You're interested in entry with key "key_0000000000003E39" removed at 03:55:18,173 and re-added at 03:55:18,268
> L1 entry added by ST when already invalidated
> ---------------------------------------------
>
> Key: ISPN-3838
> URL: https://issues.jboss.org/browse/ISPN-3838
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 6.0.0.Final
> Reporter: Radim Vansa
> Assignee: William Burns
> Priority: Critical
>
> Non-transactional cache with L1 enabled. Node A is losing ownership of an entry, the entry is not removed during ST but is going to L1.
> 1. ST builds the invalidation command, EntryWrapping interceptor starts committing all the entries
> 2. Write on primary owner (B) occurs
> 3. A gets the InvalidateL1Command, removes the ImmortalCacheEntry from data container (as it does not own the entry anymore)
> 4. The ST invalidation command commits the MortalCacheEntry with old value, storing it into the data container.
> Result: Outdated value is in L1 cache.
> As the entry is not locked during the ST, it can be committed as MortalCacheEntry only if it was not changed (removed and possibly then cached again with different value).
> (I understand that this wouldn't be easy to implement as the check is not to be executed in perform, but during the actual commit - and atomically in the container.)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 4 months
[JBoss JIRA] (ISPN-3633) InvalidateL1Command during ST should not cancel writing the entry by ST
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3633?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-3633:
-----------------------------------------------
Radim Vansa <rvansa(a)redhat.com> changed the Status of [bug 1019348|https://bugzilla.redhat.com/show_bug.cgi?id=1019348] from ON_QA to VERIFIED
> InvalidateL1Command during ST should not cancel writing the entry by ST
> -----------------------------------------------------------------------
>
> Key: ISPN-3633
> URL: https://issues.jboss.org/browse/ISPN-3633
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.3.0.Final
> Reporter: Radim Vansa
> Assignee: William Burns
> Priority: Critical
> Labels: 620
> Fix For: 6.0.0.Final
>
>
> When a node which is about to receive the entries for some segment receives InvalidateL1Command, this puts the key into StateConsumer.updatedKeys. After the entries for ST are received, the entry which was invalidated is not updated -> this result in losing the entry.
> Btw., in EntryWrappingInterceptor.visitInvalidateL1Command the trace logs all looked up entries for each entry - this causes some performance problems when tracing is on and there are many invalidated entries. Please, do this only once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 4 months