[JBoss JIRA] (ISPN-4995) ClusteredGet served for non-member of CH
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-4995?page=com.atlassian.jira.plugin.... ]
Dan Berindei resolved ISPN-4995.
--------------------------------
Fix Version/s: 7.0.3.Final
Resolution: Won't Fix
I documented the current behaviour in
https://github.com/infinispan/infinispan/wiki/Consistency-guarantees-in-I...
> ClusteredGet served for non-member of CH
> ----------------------------------------
>
> Key: ISPN-4995
> URL: https://issues.jboss.org/browse/ISPN-4995
> Project: Infinispan
> Issue Type: Bug
> Components: Core, State Transfer
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
> Fix For: 7.0.3.Final
>
>
> When nodes accept ClusteredGetCommand from node that is not member of CH, it can happen that when one thread does
> {code}
> put(K1, V1);
> put(K2, V2)
> {code}
> and another gets
> {code}
> get(K2) -> V2
> get(K1) -> V0 (some old value)
> {code}
> edg-perf01, 02 and 03 share this view and topology:
> {code}
> 04:40:08,714 TRACE [org.jgroups.protocols.FD_SOCK] (INT-8,edg-perf01-63779) edg-perf01-63779: i-have-sock: edg-perf02-45117 --> 172.18.1.3:37476 (cache is {edg-perf01-63779=172.18.1.1:40099, edg-perf02-45117=172.18.1.3:37476})
> 04:40:08,715 TRACE [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p2-t6) Received new cluster view: 8, isCoordinator = true, becameCoordinator = false
> 04:40:11,203 DEBUG [org.infinispan.topology.LocalTopologyManagerImpl] (transport-thread--p2-t1) Updating local consistent hash(es) for cache testCache: new topology = CacheTopology{id=16, rebalanceId=4, currentC
> H=DefaultConsistentHash{ns = 512, owners = (3)[edg-perf02-45117: 171+170, edg-perf03-6264: 171+171, edg-perf01-63779: 170+171]}, pendingCH=null, unionCH=null, actualMembers=[edg-perf02-45117, edg-perf03-6264, edg-perf01-63779]}
> {code}
> Later, edg-perf02 and edg-perf03 get new view and install a new topology, where edg-perf01 does not exist:
> {code}
> 04:41:13,681 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,edg-perf03-6264) ISPN000093: Received new, MERGED cluster view for channel default: MergeView::[edg-perf02-45117|9] (3) [edg-perf02-45117, edg-perf03-6264, edg-perf04-10989], 1 subgroups: [edg-perf04-10989|7] (1) [edg-perf04-10989]
> 04:41:13,681 TRACE [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p2-t22) Received new cluster view: 9, isCoordinator = false, becameCoordinator = false
> 04:41:13,760 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (remote-thread--p3-t32) Attempting to execute non-CacheRpcCommand command: CacheTopologyControlCommand{cache=testCache, type=CH_UPDATE, sender=edg-perf02-45117, joinInfo=null, topologyId=18, rebalanceId=4, currentCH=DefaultConsistentHash{ns = 512, owners = (2)[edg-perf02-45117: 256+85, edg-perf03-6264: 256+86]}, pendingCH=null, availabilityMode=AVAILABLE, actualMembers=[edg-perf02-45117, edg-perf03-6264], throwable=null, viewId=9}[sender=edg-perf02-45117]
> {code}
> After that, edg-perf04 writes to {{key_00000000000020DB}} which is currently owned only by edg-perf03 - this key servers as K1 in example above. It is not backed up to edg-perf01, but edg-perf01 still thinks it's an owner of this key as it did not get any new view (this is a log from edg-perf03) :
> {code}
> 04:41:30,884 TRACE [org.infinispan.remoting.rpc.RpcManagerImpl] (remote-thread--p3-t45) edg-perf03-6264 invoking PutKeyValueCommand{key=key_00000000000020DB, value=[33 #4: 0, 169, 284, 634, ], flags=[SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP], putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedMetadata{version=null}, successful=true} to recipient list [edg-perf03-6264] with options RpcOptions{timeout=60000, unit=MILLISECONDS, fifoOrder=true, totalOrder=false, responseFilter=null, responseMode=SYNCHRONOUS, skipReplicationQueue=false}
> {code}
> Later, edg-perf04 writes to another key {{stressor_33}} (K2 in the example) value with operationId=650 (previous value is 600) which is replicated to edg-perf02 and edg-perf03.
> Now a merge view with all 4 nodes is installed:
> {code}
> 04:41:31,258 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,edg-perf01-63779) ISPN000093: Received new, MERGED cluster view for channel default: MergeView::[edg-perf01-63779|10] (4) [edg-perf01-63779, edg-perf03-6264, edg-perf02-45117, edg-perf04-10989], 6 subgroups: [edg-perf02-45117|7] (2) [edg-perf02-45117, edg-perf03-6264], [edg-perf01-63779|4] (2) [edg-perf01-63779, edg-perf02-45117], [edg-perf02-45117|9] (3) [edg-perf02-45117, edg-perf03-6264, edg-perf04-10989], [edg-perf03-6264|4] (2) [edg-perf03-6264, edg-perf04-10989], [edg-perf01-63779|8] (3) [edg-perf01-63779, edg-perf02-45117, edg-perf03-6264], [edg-perf01-63779|6] (1) [edg-perf01-63779]
> 04:41:31,258 TRACE [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p2-t2) Received new cluster view: 10, isCoordinator = true, becameCoordinator = false
> {code}
> edg-perf01 now issues a remote get to edg-perf02 for key stressor_33 and receives the correct answer (operationId=650):
> {code}
> 04:41:32,494 TRACE [org.infinispan.remoting.rpc.RpcManagerImpl] (BackgroundOps-Checker-1) Response(s) to ClusteredGetCommand{key=stressor_33, flags=null} is {edg-perf02-45117=SuccessfulResponse{responseValue=ImmortalCacheValue {value=LastOperation{operationId=650, seed=0000A15A4C2DD25A}}} }
> {code}
> However, when edg-perf01 reads {{key_00000000000020DB}}, it loads the old value from local data container as no CH update/rebalance happened so far:
> {code}
> 04:41:32,496 TRACE [org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl] (BackgroundOps-Checker-1) Checking availability for key=key_00000000000020DB, status=AVAILABLE
> 04:41:32,497 ERROR [org.radargun.stages.cache.background.LogChecker] (BackgroundOps-Checker-1) Missing operation 634 for thread 33 on key 8411 (key_00000000000020DB)
> 04:41:32,499 DEBUG [org.radargun.service.InfinispanDebugable] (BackgroundOps-Checker-1) Debug info for key testCache key_00000000000020DB: owners=edg-perf01-63779, edg-perf03-6264, local=true, uncertain=false, container.key_00000000000020DB=ImmortalCacheEntry[key=key_00000000000020DB, value=[33 #3: 0, 169, 284, ], created=-1, isCreated=false, lastUsed=-1, isChanged=false, expires=-1, isExpired=false, canExpire=false, isEvicted=true, isRemoved=false, isValid=false, lifespan=-1, maxIdle=-1], segmentId=173
> {code}
> Note that this was found on branch https://github.com/infinispan/infinispan/pull/3062/files trying to fix ISPN-4949.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 11 months
[JBoss JIRA] (ISPN-5016) Specify and document cache consistency guarantees
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-5016?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-5016:
------------------------------------
{quote}
And in the end, I think that there should be a summarized table for the modes and ticks or crosses whether the cache will stay consistent after originator crash, primary owner crash, timeout exception/another exception.
{quote}
A table with just one "cache is consistent" tick (meaning all the nodes see the same value) should be doable, I'll give it a try.
> Specify and document cache consistency guarantees
> -------------------------------------------------
>
> Key: ISPN-5016
> URL: https://issues.jboss.org/browse/ISPN-5016
> Project: Infinispan
> Issue Type: Task
> Components: Documentation-Core
> Affects Versions: 7.0.2.Final
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
>
> We can't simply use the consistency model defined by Java Specification and broaden it for whole cache (maybe the expression "can't" is too strong, but we definitely don't want to do that in some cases).
> By consistency guarantees/model I mean mostly in which order are
> writes allowed to be observed: and we can't boil it down to simply
> causal, PRAM or any other consistency model as writes can be observed as non-atomic in Infinispan.
> Infinispan documentation is quite scarce about that, the only trace I've
> found is in Glossarry [2] "Infinispan has traditionally followed ACID
> principles as far as possible, however an eventually consistent mode
> embracing BASE is on the roadmap."
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 11 months
[JBoss JIRA] (ISPN-5016) Specify and document cache consistency guarantees
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-5016?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-5016:
------------------------------------
>> Because data is stored in the shared store after the invalidation command was executed on all the nodes, it’s also possible for a node to read the old value from the shared store after it has executed the invalidation.
> I don't understand the reason very well, could you elaborate a bit more?
Rephrased.
>> Invalidations triggered by write operations are synchronous and the operation will fail with a SuspectException if another node crashes. The entry will not be updated on the originator, but invalidation will still be performed on the non-crashed nodes.
> I thought SuspectExceptions are silently handled, aren't they?
Nope, they are silently handled only in some cases (e.g. L1 invalidation commands), but not in all cases (e.g. prepare commands, or regular invalidation commands). It does sound like a bug, I'll create one.
>> For multi-key operations like putAll(), the operation is sent to all the primary owners of the affected keys. When there is a single primary owner for all the affected keys, the operation succeeds and its effects are visible atomically. When there are multiple primary owners, the operation is not atomic: overlapping read operations will see random subsets of the updated values.
> As reads don't acquire any locks, I think that even update on single primary owner can be seen as non-atomic.
Yeah, I was trying to say that it preserve the semantics of a local cache, but multi-key operations in a local cache aren't atomic either.
>> If a node is suspected because of a Full GC, it might go from the initial JGroups view straight to the merge view. If that happens, its topology will be the largest one, and it will not be wiped, neither will it receive new data. Instead, it will keep the (possibly stale) entries it had before the Full GC.
> Sorry, this is not very clear to me. Why largest topology ID and why wouldn't that propagate its state to all other members (wiping their data)?
It's because the topology with the most nodes is elected as the merge topology. It seemed like a good idea when I implemented it, but perhaps it would have been better to use the one with the largest topology id instead.
>> If at least half of the nodes in the stable topology leave in quick succession
> What is quick succession? Does that depend on JGroups view installation, any timeout or the duration of rebalance?
It means before the stable topology is updated again. In replicated mode there is no state transfer when a node leaves, that means the coordinator just needs to receive the rebalance confirmation from all the nodes.
>> And if some of the nodes in the Available partition’s consistent hash are not really accessible after the merge, the cache might stay Degraded.
> 'the cache'? We should be talking rather about partitions, not the whole cache. I understand the first part of the special case, but not the latter.
Changed to 'the merged partition'.
>> If a node joins and becomes a backup owner after a write command was sent from the primary owner to the backups, but before the primary owner updates its own data container, it may not receive the value neither as a write command nor via state transfer.
> Sounds like a bug to me - is there a JIRA that could be linked? We could tolerate inconsistencies when the node crashes (if we can't fix it), but join or graceful leave should keep the cluster consistent.
I don't think it's any worse than the inconsistency in the previous paragraph.
>> When a write to the store fails, it will fail the write operation.
> Does this hold for write-behind, too?
That's the next paragraph...
>> With write-behind or asynchronous replication enabled, store write failures are hidden from the user (unless the originator is the primary owner, with async replication).
> When originator == primary and write-behind, how can the failure to write to the store be propagated to the user? I thought that user thread initiates both write to the store and async replication and returns.
Reworded, I meant with async replication but without write-behind.
>> When the partitions merge back, there is no effort to replicate the values from one partition to another.
> Why is that different from non-tx mode, where the partitions with non-highest topology id are wiped? Moreover, in optimistic tx you write
>> Same as with pessimistic and non-transactional caches.
> - what version, then?
It's actually the same, I added that to the pessimistic mode description.
>> If the primary owners of the keys written by the transaction are all in the local transaction,
> local partition?
Fixed.
>> If one partition stays available, its entries will replace all the other partitions' entries on merge, undoing partial commits in those partitions.
> Do I understand correctly that degraded partition may commit transaction, and this transaction will be later ignored (the data will later be overwritten to the previous values). Why is this behaviour desired?
Simplicity: whenever we have a merge, we pick one winning partition, and we wipe the nodes in other partitions.
Ideally one partition shouldn't be able to commit when it doesn't have a majority of nodes, but there's no way to prevent partial commits with single-phase commit.
>> Transactions already prepared, however, will commit successfully even in minority partitions
> Is that true even if the originator is not in this minority partition?
I intended to say that the tx will succeed (will not throw any exception) even if the originator ends up in a minority partition.
Obviously the same applies if the originator's partition stays Available.
>> When a transaction needs to acquire more than one key lock with the same primary node, they are always acquired in the same order, so this will not cause a deadlock.
> If the keys have the same hashCodes, they can be locked in different order, though. ISPN-2491
Reference added.
>> The commit is always synchronous on the originator, so a transaction T3 started on node A after T1 finished will see T1’s updates.
> Will it see all T1's updates, or just updates on those entries owned by A?
This is replicated mode, so it's all the updates.
>> The write to the attached cache store(s) is performed during the one-phase prepare command or the commit command, depending on the configuration.
> What configuration, exactly?
Basically with pessimistic locking or async replication, you have a one-phase prepare command, otherwise you have separate prepare and commit commands.
I've also added a section for use1PcForAutoCommitTransactions, which uses another one-phase commit mode.
>> TODO What happens if a node is a backup owner for two of the keys?
> Hmm? The primary owner sends the PutMapCommand to all nodes that are backup owners of at least one of the keys primary-owned by this node. But you're right that I am not sure whether this command makes the backup node write only
Yeah, the backup doesn't do any check on who forwarded the command so it can update the wrong entries.
>> When a node tries to read a key it does not own, it sends a remote get
> AFAIK the node checks if the entry is in data container, not whether it is owned.
That changed with ISPN-4444.
> Regarding the situation that an entry is lost, I think that you should note somewhere that further attempts to read it will simply return null, not throwing any exceptions (the user could expect that).
Both can happen, depending on the partition handling configuration, so I've added a forward reference.
> I lack explicit resolution on ISPN-4995, regarding reading and writing two (or more) entries and expectation whether any read is already visible.
I've added another paragraph to cover the "after the update is finished, other processes can see only the new value" rule in the ISPN-4995 scenario.
I only had it covered in replicated mode, where minority partition nodes can also read stale values until they enter degraded mode.
> Specify and document cache consistency guarantees
> -------------------------------------------------
>
> Key: ISPN-5016
> URL: https://issues.jboss.org/browse/ISPN-5016
> Project: Infinispan
> Issue Type: Task
> Components: Documentation-Core
> Affects Versions: 7.0.2.Final
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
>
> We can't simply use the consistency model defined by Java Specification and broaden it for whole cache (maybe the expression "can't" is too strong, but we definitely don't want to do that in some cases).
> By consistency guarantees/model I mean mostly in which order are
> writes allowed to be observed: and we can't boil it down to simply
> causal, PRAM or any other consistency model as writes can be observed as non-atomic in Infinispan.
> Infinispan documentation is quite scarce about that, the only trace I've
> found is in Glossarry [2] "Infinispan has traditionally followed ACID
> principles as far as possible, however an eventually consistent mode
> embracing BASE is on the roadmap."
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 11 months
[JBoss JIRA] (ISPN-4810) Local Transactional Cache loses data when eviction is enabled and there are multiple readers and one writer
by Mario Mueller (JIRA)
[ https://issues.jboss.org/browse/ISPN-4810?page=com.atlassian.jira.plugin.... ]
Mario Mueller commented on ISPN-4810:
-------------------------------------
For me this behaviour also shows up with LevelDb and JDBC stores, if that helps in any way.
> Local Transactional Cache loses data when eviction is enabled and there are multiple readers and one writer
> -----------------------------------------------------------------------------------------------------------
>
> Key: ISPN-4810
> URL: https://issues.jboss.org/browse/ISPN-4810
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 6.0.2.Final
> Environment: Windows 7 x64 (NTFS)
> Oracle JDK1.7.0_40
> Apache Maven 3.0.5
> Reporter: Horia Chiorean
> Labels: modeshape
> Attachments: ispn_concurrent.zip
>
>
> Using Infinispan 6.0.2 and a local, transactional cache backed by a <singleFile> store, with eviction enabled and a small {{max-entries}} setting, we have the following scenario:
> * the main thread (i.e. the "writer") starts a transaction, adds a batch of strings into the cache and also appends the same strings into a List cache entry and then commits the transaction
> * after the above has finished (i.e. after tx.commit) it fires a number of reader threads where each reader thread
> ** checks that the string entries were added into the cache and
> ** checks that the entries were correctly appended to the List entry
> * the above steps are repeated a number of times
> On any given run, based on timing, we're seeing that at some point (after some time) some of the reader threads will not see the latest version of the List entry (i.e. will not see the latest elements that were added into the list) but rather an old, stale List (sort of a "lost update" scenario).
> If we either:
> * disable eviction or
> * set the {{max-entries}} to a large enough value (which I suspect has the same effect - not evicting anything) the problem doesn't show up.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 11 months
[JBoss JIRA] (ISPN-5109) Topology hint breaks cross site replication in CacheTopologyControlCommand
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-5109?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration updated ISPN-5109:
------------------------------------------
Bugzilla Update: Perform
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=1177228
> Topology hint breaks cross site replication in CacheTopologyControlCommand
> --------------------------------------------------------------------------
>
> Key: ISPN-5109
> URL: https://issues.jboss.org/browse/ISPN-5109
> Project: Infinispan
> Issue Type: Bug
> Components: Cross-Site Replication
> Environment: JDG 6.4.0.ER8 (Infinispan 6.2.0.ER8)
> Reporter: Osamu Nagano
> Attachments: clustered_xSite-A.xml, clustered_xSite-B.xml
>
>
> When topology hints like site, rack and machine are set under cross site replication configuration, server will keep generating the following WARN log messages.
> {noformat}
> 17:51:03,037 WARN [org.infinispan.topology.CacheTopologyControlCommand] (MSC service thread 1-3) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=namedCache, type=JOIN, sender=svr01/xSite-A ( flags=2 (can_be_sm), site-id=s1, rack-id=r1, machine-id=m1), joinInfo=CacheJoinInfo{consistentHashFactory=org.infinispan.distribution.ch.TopologyAwareConsistentHashFactory@f0d, hashFunction=MurmurHash3, numSegments=80, numOwners=2, timeout=60000, totalOrder=false, distributed=true}, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=0}: java.lang.ClassCastException: org.infinispan.remoting.transport.jgroups.JGroupsAddress cannot be cast to org.infinispan.remoting.transport.TopologyAwareAddress
> at org.infinispan.distribution.topologyaware.TopologyInfo.addTopology(TopologyInfo.java:56) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.distribution.topologyaware.TopologyInfo.<init>(TopologyInfo.java:37) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.distribution.ch.TopologyAwareConsistentHashFactory.addBackupOwners(TopologyAwareConsistentHashFactory.java:27) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.distribution.ch.DefaultConsistentHashFactory.rebalanceBuilder(DefaultConsistentHashFactory.java:126) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.distribution.ch.DefaultConsistentHashFactory.create(DefaultConsistentHashFactory.java:41) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.distribution.ch.DefaultConsistentHashFactory.create(DefaultConsistentHashFactory.java:25) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.topology.ClusterCacheStatus.createInitialCacheTopology(ClusterCacheStatus.java:567) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.topology.ClusterCacheStatus.doJoin(ClusterCacheStatus.java:551) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleJoin(ClusterTopologyManagerImpl.java:131) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:162) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:144) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.topology.LocalTopologyManagerImpl.executeOnCoordinator(LocalTopologyManagerImpl.java:377) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:103) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:109) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_71]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_71]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_71]
> at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_71]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168) [infinispan-commons-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.CacheImpl.start(CacheImpl.java:756) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:581) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:536) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:414) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:428) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:89) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:80) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:104) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:101) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.security.Security.doPrivileged(Security.java:76) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:48) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:109) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:79) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1980) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1913) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71]
> {noformat}
> If such topology hints are removed from udp stack transport element, this issue doesn't happen.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 11 months
[JBoss JIRA] (ISPN-5109) Topology hint breaks cross site replication in CacheTopologyControlCommand
by Osamu Nagano (JIRA)
[ https://issues.jboss.org/browse/ISPN-5109?page=com.atlassian.jira.plugin.... ]
Osamu Nagano updated ISPN-5109:
-------------------------------
Attachment: clustered_xSite-A.xml
clustered_xSite-B.xml
> Topology hint breaks cross site replication in CacheTopologyControlCommand
> --------------------------------------------------------------------------
>
> Key: ISPN-5109
> URL: https://issues.jboss.org/browse/ISPN-5109
> Project: Infinispan
> Issue Type: Bug
> Components: Cross-Site Replication
> Environment: JDG 6.4.0.ER8 (Infinispan 6.2.0.ER8)
> Reporter: Osamu Nagano
> Attachments: clustered_xSite-A.xml, clustered_xSite-B.xml
>
>
> When topology hints like site, rack and machine are set under cross site replication configuration, server will keep generating the following WARN log messages.
> {noformat}
> 17:51:03,037 WARN [org.infinispan.topology.CacheTopologyControlCommand] (MSC service thread 1-3) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=namedCache, type=JOIN, sender=svr01/xSite-A ( flags=2 (can_be_sm), site-id=s1, rack-id=r1, machine-id=m1), joinInfo=CacheJoinInfo{consistentHashFactory=org.infinispan.distribution.ch.TopologyAwareConsistentHashFactory@f0d, hashFunction=MurmurHash3, numSegments=80, numOwners=2, timeout=60000, totalOrder=false, distributed=true}, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=0}: java.lang.ClassCastException: org.infinispan.remoting.transport.jgroups.JGroupsAddress cannot be cast to org.infinispan.remoting.transport.TopologyAwareAddress
> at org.infinispan.distribution.topologyaware.TopologyInfo.addTopology(TopologyInfo.java:56) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.distribution.topologyaware.TopologyInfo.<init>(TopologyInfo.java:37) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.distribution.ch.TopologyAwareConsistentHashFactory.addBackupOwners(TopologyAwareConsistentHashFactory.java:27) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.distribution.ch.DefaultConsistentHashFactory.rebalanceBuilder(DefaultConsistentHashFactory.java:126) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.distribution.ch.DefaultConsistentHashFactory.create(DefaultConsistentHashFactory.java:41) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.distribution.ch.DefaultConsistentHashFactory.create(DefaultConsistentHashFactory.java:25) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.topology.ClusterCacheStatus.createInitialCacheTopology(ClusterCacheStatus.java:567) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.topology.ClusterCacheStatus.doJoin(ClusterCacheStatus.java:551) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleJoin(ClusterTopologyManagerImpl.java:131) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:162) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:144) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.topology.LocalTopologyManagerImpl.executeOnCoordinator(LocalTopologyManagerImpl.java:377) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:103) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:109) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_71]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_71]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_71]
> at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_71]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168) [infinispan-commons-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.CacheImpl.start(CacheImpl.java:756) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:581) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:536) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:414) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:428) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:89) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:80) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:104) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:101) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.security.Security.doPrivileged(Security.java:76) [infinispan-core-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:48) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:109) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:79) [infinispan-server-infinispan-6.2.0.ER8-redhat-1.jar:6.2.0.ER8-redhat-1]
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1980) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1913) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71]
> {noformat}
> If such topology hints are removed from udp stack transport element, this issue doesn't happen.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 11 months