[JBoss JIRA] (ISPN-5208) Avoid invalid topology
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-5208?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-5208:
-----------------------------------------------
Sebastian Łaskawiec <slaskawi(a)redhat.com> changed the Status of [bug 1190001|https://bugzilla.redhat.com/show_bug.cgi?id=1190001] from ASSIGNED to MODIFIED
> Avoid invalid topology
> ----------------------
>
> Key: ISPN-5208
> URL: https://issues.jboss.org/browse/ISPN-5208
> Project: Infinispan
> Issue Type: Enhancement
> Components: Server
> Reporter: Takayoshi Kimura
> Assignee: Galder Zamarreño
> Labels: jdg641
> Fix For: 7.2.0.Final
>
>
> We've seen some invalid topology propagated to client and it causes ArrayIndexOutOfBoundsException:
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at org.infinispan.client.hotrod.impl.transport.tcp.RoundRobinBalancingStrategy.getServerByIndex(RoundRobinBalancingStrategy.java:68) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.transport.tcp.RoundRobinBalancingStrategy.nextServer(RoundRobinBalancingStrategy.java:44) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.nextServer(TcpTransportFactory.java:220) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:194) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.operations.FaultTolerantPingOperation.getTransport(FaultTolerantPingOperation.java:27) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:48) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.ping(RemoteCacheImpl.java:535) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.ping(RemoteCacheManager.java:635) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.createRemoteCache(RemoteCacheManager.java:616) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:527) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:523) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> {noformat}
> {noformat}
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0
> at org.infinispan.client.hotrod.impl.consistenthash.SegmentConsistentHash.getServer(SegmentConsistentHash.java:33)
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:204)
> at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.getTransport(AbstractKeyOperation.java:40)
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:48)
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:237)
> at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:79)
> at sample.Main.main(Main.java:16)
> {noformat}
> It happens on both Hot Rod 2 and 1.3 clients.
> It's really hard to reproduce this state and we don't have a consistent way to reproduce it. However when this happens there is always view change happening so it's related to view change.
> Judging from the stack trace, the client receives numOwners=0 or numSegments=0 topology from the server.
> Also we are unable to find to recover this situation. Rebooting random nodes don't help and keep getting this exceptions on client side.
> Until we can find the root cause, I think it's better to add a guard to avoid this kind invalid topology stored in the server side and propagated to the clients.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years
[JBoss JIRA] (ISPN-5318) JSR-107 Support for cacheResolverFactory annotation property
by Matej Čimbora (JIRA)
Matej Čimbora created ISPN-5318:
-----------------------------------
Summary: JSR-107 Support for cacheResolverFactory annotation property
Key: ISPN-5318
URL: https://issues.jboss.org/browse/ISPN-5318
Project: Infinispan
Issue Type: Enhancement
Components: JCache
Reporter: Matej Čimbora
Currently, there is no mechanism which handles specifying cacheResolverFactory attribute of JSR-107 annotations. Can be workarounded by injecting custom cacheResolver implementation.
Note: Not yet covered by TCK tests.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years
[JBoss JIRA] (ISPN-5174) Transaction cannot be recommitted after ownership changes
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-5174?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-5174:
-----------------------------------------------
Matej Čimbora <mcimbora(a)redhat.com> changed the Status of [bug 1185779|https://bugzilla.redhat.com/show_bug.cgi?id=1185779] from ON_QA to VERIFIED
> Transaction cannot be recommitted after ownership changes
> ---------------------------------------------------------
>
> Key: ISPN-5174
> URL: https://issues.jboss.org/browse/ISPN-5174
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 7.1.0.CR2
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
>
> Once transaction is completed, it cannot commit again. If it should commit more keys since it has become an owner of some new keys modified in this transaction, it just ignores the further commit.
> There is a race with state transfer which can bring an old value (with StateResponseCommand sent before it is commited) but the value is not set by the ongoing transaction either.
> This results with stale value stored on one node.
> In my case, The problematic part is transaction <edg-perf01-62141>:15066 (consisting of 10 modifications) which got prepared and committed on edg-perf04 in topology 25. Before the originator finishes, topology changes and 04 requests ongoing transactions:
> {code}
> 11:06:11,369 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (transport-thread-17) Replication task sending StateRequestCommand{cache=testCache, origin=edg-perf04-35097, type=GET_TRANSACTIONS, topologyId=28, segments=[275, 1, 278, 9, 282, 286, 17, 259, 25, 267, 171, 169, 33, 306, 175, 173, 310, 172, 314, 41, 167, 165, 318, 187, 290, 49, 185, 191, 294, 189, 179, 298, 57, 177, 183, 302, 181, 343, 205, 201, 338, 203, 336, 351, 197, 349, 199, 347, 193, 345, 195, 326, 85, 87, 322, 93, 332, 95, 330, 89, 91, 103, 101, 99, 506, 97, 105, 357, 359, 353, 355, 361]} to single recipient edg-perf01-62141 with response mode GET_ALL
> 11:06:11,495 DEBUG [org.infinispan.statetransfer.StateConsumerImpl] (transport-thread-17) Applying 6 transactions for cache testCache transferred from node edg-perf01-62141
> {code}
> However I don't see how these are applied, since PrepareCommand is not created again - from the code I see only that backup locks are added. Not sure if the transaction is registered at all, since it was already completed on this node (but at that time it did not own key_00000000000002EB).
> After originator stores the entry, it sends one more CommitCommand with topology 28:
> {code}
> 11:06:11,619 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (DefaultStressor-2) Replication task sending CommitCommand {gtx=GlobalTransaction:<edg-perf01-62141>:15066:local, cacheName='testCache', topologyId=28} to addresses [edg-perf03-20530, edg-perf04-35097] with response mode GET_ALL
> {code}
> 04 receives several CommitCommands (both from originator and forwards), but all of them are ignored as the transaction is completed.
> I don't see the logs where state transfer is assembled, but it's probably before the entry is stored on originator as the state transfer contains the old entry:
> {code}
> 11:06:13,449 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (remote-thread-91) Received chunk with keys [key_000000000000065B, key_00000000000006BE, key_FFFFFFFFFFFFE62F, key_0000000000001F42, key_000000000000027B, key_000000000000159D, key_00000000000002EB, key_00000000000002BB] for segment 343 of cache testCache from node edg-perf01-62141
> 11:06:13,454 TRACE [org.infinispan.container.DefaultDataContainer] (remote-thread-91) Store ImmortalCacheEntry{key=key_00000000000002EB, value=[2 #7: 366, 544, 576, 804, 1061, 1181, 1290, ]} in container
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years
[JBoss JIRA] (ISPN-5131) Deploy custom cache store to Infinispan Server
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-5131?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-5131:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 1186857|https://bugzilla.redhat.com/show_bug.cgi?id=1186857] from NEW to ASSIGNED
> Deploy custom cache store to Infinispan Server
> ----------------------------------------------
>
> Key: ISPN-5131
> URL: https://issues.jboss.org/browse/ISPN-5131
> Project: Infinispan
> Issue Type: Feature Request
> Components: Loaders and Stores, Server
> Reporter: Tristan Tarrant
> Assignee: Sebastian Łaskawiec
> Fix For: 7.2.0.Final
>
>
> h2. Overview
> Support the deployment and configuration of a custom cache store.
> h2. Client Perspective
> In order to create a deployable Cache Store the client will have to implement {{AdvancedLoadWriteStoreFactory}} (which will contain factory methods for creating {{AdvancedLoadWriteStore}}). Next, the factory will have to be annotated with {{@NamedFactory}} and placed into a jar together with proper entry of {{META-INF/services/org.infinispan.persistence.AdvancedLoadWriteStoreFactory}}. The last activity is to deploy given jar into Hotrod server.
> h2. Implementation overview
> The implementation is based on Deployable Filters and Converters.
> Currently all writers and loaders are instantiated in {{PersistenceManagerImpl#createLoadersAndWriters}}. This implementation will be modified to use {{CacheStoreFactoryRegistry}}, which will contain a list of {{CacheStoreFactories}}. One of the factories will be added by default - the local one (which will the same mechanism as we do now - {{Util.getInstance(classAnnotation)}}. Other {{CacheStoreFactories}} will be added after deployment scanning.
> h2. Implementation doubts and questions:
> * Should we expose a factory for {{AdvancedLoadWriteStore}} or should we include also {{ExternalStore}} (or even separate factory for {{CacheLoader}} and {{CacheWriter}}?
> ** YES, we should expose all of them.
> * How to ensure that deployment scanning has finished before creating instantiating {{AdvancedLoadWriteStore}}?
> ** Using {{org.infinispan.server.endpoint.subsystem.EndpointSubsystemAdd}}
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years
[JBoss JIRA] (ISPN-5274) Inconsistent data after transaction rollback (with success on originator)
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-5274?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-5274:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 1199490|https://bugzilla.redhat.com/show_bug.cgi?id=1199490] from NEW to ASSIGNED
> Inconsistent data after transaction rollback (with success on originator)
> -------------------------------------------------------------------------
>
> Key: ISPN-5274
> URL: https://issues.jboss.org/browse/ISPN-5274
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Reporter: Matej Čimbora
> Assignee: Dan Berindei
>
> Scenario
> Nodes edg-perf[10-13], partition handling on
> 1.Transaction is started on edg-perf10
> {code}
> 10:24:48,228 TRACE [org.infinispan.transaction.xa.TransactionXaAdapter] (DefaultStressor-6) start called on tx GlobalTransaction:<edg-perf10-20667>:38910:local
> {code}
> 2. Value of key_000000000000065B is updated within the transaction
> {code}
> 10:24:48,405 TRACE [org.radargun.service.InfinispanOperations$Cache] (DefaultStressor-6) PUT cache=testCache key=key_000000000000065B value=[6 #15: 296, 501, 1109, 1119, 1459, 1544, 1999, 2083, 2130, 2257, 2298, 2784, 2941, 3009, 3147, ]
> {code}
> 3. Transaction is successfully prepared on edg-perf11 & edg-perf12
> {code}
> 10:24:48,559 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (DefaultStressor-6) Responses: [sender=edg-perf11-61837, received=true, suspected=false]
> [sender=edg-perf12-7305, received=true, suspected=false]
> {code}
> 4. Transaction commit is issued
> {code}
> 10:24:48,562 TRACE [org.infinispan.transaction.TransactionCoordinator] (DefaultStressor-6) Committing transaction GlobalTransaction:<edg-perf10-20667>:38910:local
> {code}
> 5. Other participating nodes (edg-perf11 & edg-perf12) are suspected...
> {code}
> 10:24:52,705 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (DefaultStressor-6) Target node edg-perf11-61837 left during remote call, ignoring
> 10:24:52,716 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (DefaultStressor-6) Target node edg-perf12-7305 left during remote call, ignoring
> {code}
> ... as they received a new view (without edg-perf10) meanwhile.
> {code}
> 10:24:48,547 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-1,edg-perf12-7305) ISPN000093: Received new, MERGED cluster view: MergeView::[edg-perf12-7305|20] (3) [edg-perf12-7305, edg-perf11-61837, edg-perf13-25187], 2 subgroups: [edg-perf13-25187|8] (1) [edg-perf13-25187], [edg-perf13-25187|19] (1) [edg-perf13-25187]
> {code}
> Still, the transaction is commited on edg-perf10 & updated entry is stored locally
> {code}
> 10:24:52,894 TRACE [org.infinispan.statetransfer.CommitManager] (DefaultStressor-6) Trying to commit. Key=key_000000000000065B. Operation Flag=null, L1 invalidation=false
> 10:24:52,896 TRACE [org.infinispan.statetransfer.CommitManager] (DefaultStressor-6) Committing key=key_000000000000065B. It is a L1 invalidation or a normal put and no tracking is enabled!
> 10:24:52,908 TRACE [org.infinispan.container.DefaultDataContainer] (DefaultStressor-6) Creating new ICE for writing. Existing=ImmortalCacheEntry{key=key_000000000000065B, value=[6 #14: 296, 501, 1109, 1119, 1459, 1544, 1999, 2083, 2130, 2257, 2298, 2784, 2941, 3009, ]}, metadata=EmbeddedMetadata{version=null}, new value=[6 #15: 296, 501, 1109, 1119, 1459, 1544, 1999, 2083, 2130, 2257, 2298, 2784, 2941, 3009, 3147, ]
> {code}
> 6. Other nodes rollback the transaction
> {code}
> 10:24:50,376 DEBUG [org.infinispan.transaction.TransactionTable] (transport-thread-10) Rolling back transaction GlobalTransaction:<edg-perf10-20667>:38910:remote because originator edg-perf10-20667 left the cluster
> {code}
> 7. edg-perf10 receives a new view, containing nodes edg-perf[10,11,13]. Incoming state transfer overwrites the updated value
> {code}
> 10:25:09,614 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,edg-perf10-20667) ISPN000093: Received new, MERGED cluster view: MergeView::[edg-perf10-20667|22] (3) [edg-perf10-20667, edg-perf11-61837, edg-perf13-25187], 4 subgroups: [edg-perf10-20667|15] (2) [edg-perf10-20667, edg-perf11-61837], [edg-perf10-20667|19] (3) [edg-perf10-20667, edg-perf12-7305, edg-perf11-61837], [edg-perf10-20667|18] (1) [edg-perf10-20667], [edg-perf10-20667|21] (2) [edg-perf10-20667, edg-perf13-25187]
> {code}
> 8. get operation returns outdated value
> {code}
> 10:26:21,020 TRACE [org.infinispan.remoting.rpc.RpcManagerImpl] (DefaultStressor-6) Response(s) to ClusteredGetCommand{key=key_000000000000065B, flags=null} is {edg-perf12-7305=SuccessfulResponse{responseValue=ImmortalCacheValue {value=[6 #14: 296, 501, 1109, 1119, 1459, 1544, 1999, 2083, 2130, 2257, 2298, 2784, 2941, 3009, ]}} }
> {code}
> From client perspective, this behavior is not transparent. Provided the transaction ended up successfully, presence of the updated entry can be assumed.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years