[JBoss JIRA] (ISPN-5208) Avoid invalid topology
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-5208?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño updated ISPN-5208:
-----------------------------------
Status: Pull Request Sent (was: Coding In Progress)
Git Pull Request: https://github.com/infinispan/infinispan/pull/3311
> Avoid invalid topology
> ----------------------
>
> Key: ISPN-5208
> URL: https://issues.jboss.org/browse/ISPN-5208
> Project: Infinispan
> Issue Type: Enhancement
> Components: Server
> Reporter: Takayoshi Kimura
> Assignee: Galder Zamarreño
> Labels: jdg641
> Fix For: 7.2.0.Final
>
>
> We've seen some invalid topology propagated to client and it causes ArrayIndexOutOfBoundsException:
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at org.infinispan.client.hotrod.impl.transport.tcp.RoundRobinBalancingStrategy.getServerByIndex(RoundRobinBalancingStrategy.java:68) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.transport.tcp.RoundRobinBalancingStrategy.nextServer(RoundRobinBalancingStrategy.java:44) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.nextServer(TcpTransportFactory.java:220) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:194) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.operations.FaultTolerantPingOperation.getTransport(FaultTolerantPingOperation.java:27) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:48) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.ping(RemoteCacheImpl.java:535) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.ping(RemoteCacheManager.java:635) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.createRemoteCache(RemoteCacheManager.java:616) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:527) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:523) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> {noformat}
> {noformat}
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0
> at org.infinispan.client.hotrod.impl.consistenthash.SegmentConsistentHash.getServer(SegmentConsistentHash.java:33)
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:204)
> at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.getTransport(AbstractKeyOperation.java:40)
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:48)
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:237)
> at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:79)
> at sample.Main.main(Main.java:16)
> {noformat}
> It happens on both Hot Rod 2 and 1.3 clients.
> It's really hard to reproduce this state and we don't have a consistent way to reproduce it. However when this happens there is always view change happening so it's related to view change.
> Judging from the stack trace, the client receives numOwners=0 or numSegments=0 topology from the server.
> Also we are unable to find to recover this situation. Rebooting random nodes don't help and keep getting this exceptions on client side.
> Until we can find the root cause, I think it's better to add a guard to avoid this kind invalid topology stored in the server side and propagated to the clients.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years
[JBoss JIRA] (ISPN-4810) Local Transactional Cache loses data when eviction is enabled and there are multiple readers and one writer
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-4810?page=com.atlassian.jira.plugin.... ]
Dan Berindei resolved ISPN-4810.
--------------------------------
Resolution: Done
> Local Transactional Cache loses data when eviction is enabled and there are multiple readers and one writer
> -----------------------------------------------------------------------------------------------------------
>
> Key: ISPN-4810
> URL: https://issues.jboss.org/browse/ISPN-4810
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 6.0.2.Final
> Environment: Windows 7 x64 (NTFS)
> Oracle JDK1.7.0_40
> Apache Maven 3.0.5
> Reporter: Horia Chiorean
> Assignee: William Burns
> Labels: modeshape
> Fix For: 7.2.0.Beta1, 5.2.11.Final
>
> Attachments: ispn_concurrent.zip
>
>
> Using Infinispan 6.0.2 and a local, transactional cache backed by a <singleFile> store, with eviction enabled and a small {{max-entries}} setting, we have the following scenario:
> * the main thread (i.e. the "writer") starts a transaction, adds a batch of strings into the cache and also appends the same strings into a List cache entry and then commits the transaction
> * after the above has finished (i.e. after tx.commit) it fires a number of reader threads where each reader thread
> ** checks that the string entries were added into the cache and
> ** checks that the entries were correctly appended to the List entry
> * the above steps are repeated a number of times
> On any given run, based on timing, we're seeing that at some point (after some time) some of the reader threads will not see the latest version of the List entry (i.e. will not see the latest elements that were added into the list) but rather an old, stale List (sort of a "lost update" scenario).
> If we either:
> * disable eviction or
> * set the {{max-entries}} to a large enough value (which I suspect has the same effect - not evicting anything) the problem doesn't show up.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years
[JBoss JIRA] (ISPN-5304) WildFly module "/org/infinispan/client/hotrod/jdg-6.4" is different if installed with library or HR-client
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-5304?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration updated ISPN-5304:
------------------------------------------
Bugzilla Update: Perform
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=1202370
> WildFly module "/org/infinispan/client/hotrod/jdg-6.4" is different if installed with library or HR-client
> ----------------------------------------------------------------------------------------------------------
>
> Key: ISPN-5304
> URL: https://issues.jboss.org/browse/ISPN-5304
> Project: Infinispan
> Issue Type: Feature Request
> Affects Versions: 7.2.0.Alpha1, 7.1.1.Final
> Environment: Infinispan as WildFly module
> Reporter: Wolf-Dieter Fink
>
> If Infinispan is used inside WildFly applications there are two WF modules for Infinispan.
> One for Library mode
> One for HotRod remote client mode
> Both modules contain
> modules/org/infinispan/client/hotrod/jdg-6.4/infinispan-client-hotrod-6.2.0.Final-redhat-4.jar
> modules/org/infinispan/client/hotrod/jdg-6.4/module.xml
> BUT
> the module.xml is different:
> < <module name="org.infinispan" slot="jdg-6.4"/>
> ---
> > <module name="org.infinispan.commons" slot="jdg-6.4"/>
> > <module name="org.infinispan.protostream" slot="jdg-6.4"/>
> > <module name="org.infinispan.query.dsl" slot="jdg-6.4"/>
> > <module name="org.infinispan.query.remote.client" slot="jdg-6.4"/>
> This mean it is not possible to use both modules if there are applications with library mode and others with HR remote client mode.
> Also it is odd because it depends on the installation order what dependencies are installed.
> It should be possible to install both modules and use it at the same time from different applicaitons.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years
[JBoss JIRA] (ISPN-5174) Transaction cannot be recommitted after ownership changes
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-5174?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-5174:
-----------------------------------------------
Dan Berindei <dberinde(a)redhat.com> changed the Status of [bug 1185779|https://bugzilla.redhat.com/show_bug.cgi?id=1185779] from ASSIGNED to POST
> Transaction cannot be recommitted after ownership changes
> ---------------------------------------------------------
>
> Key: ISPN-5174
> URL: https://issues.jboss.org/browse/ISPN-5174
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 7.1.0.CR2
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
>
> Once transaction is completed, it cannot commit again. If it should commit more keys since it has become an owner of some new keys modified in this transaction, it just ignores the further commit.
> There is a race with state transfer which can bring an old value (with StateResponseCommand sent before it is commited) but the value is not set by the ongoing transaction either.
> This results with stale value stored on one node.
> In my case, The problematic part is transaction <edg-perf01-62141>:15066 (consisting of 10 modifications) which got prepared and committed on edg-perf04 in topology 25. Before the originator finishes, topology changes and 04 requests ongoing transactions:
> {code}
> 11:06:11,369 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (transport-thread-17) Replication task sending StateRequestCommand{cache=testCache, origin=edg-perf04-35097, type=GET_TRANSACTIONS, topologyId=28, segments=[275, 1, 278, 9, 282, 286, 17, 259, 25, 267, 171, 169, 33, 306, 175, 173, 310, 172, 314, 41, 167, 165, 318, 187, 290, 49, 185, 191, 294, 189, 179, 298, 57, 177, 183, 302, 181, 343, 205, 201, 338, 203, 336, 351, 197, 349, 199, 347, 193, 345, 195, 326, 85, 87, 322, 93, 332, 95, 330, 89, 91, 103, 101, 99, 506, 97, 105, 357, 359, 353, 355, 361]} to single recipient edg-perf01-62141 with response mode GET_ALL
> 11:06:11,495 DEBUG [org.infinispan.statetransfer.StateConsumerImpl] (transport-thread-17) Applying 6 transactions for cache testCache transferred from node edg-perf01-62141
> {code}
> However I don't see how these are applied, since PrepareCommand is not created again - from the code I see only that backup locks are added. Not sure if the transaction is registered at all, since it was already completed on this node (but at that time it did not own key_00000000000002EB).
> After originator stores the entry, it sends one more CommitCommand with topology 28:
> {code}
> 11:06:11,619 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (DefaultStressor-2) Replication task sending CommitCommand {gtx=GlobalTransaction:<edg-perf01-62141>:15066:local, cacheName='testCache', topologyId=28} to addresses [edg-perf03-20530, edg-perf04-35097] with response mode GET_ALL
> {code}
> 04 receives several CommitCommands (both from originator and forwards), but all of them are ignored as the transaction is completed.
> I don't see the logs where state transfer is assembled, but it's probably before the entry is stored on originator as the state transfer contains the old entry:
> {code}
> 11:06:13,449 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (remote-thread-91) Received chunk with keys [key_000000000000065B, key_00000000000006BE, key_FFFFFFFFFFFFE62F, key_0000000000001F42, key_000000000000027B, key_000000000000159D, key_00000000000002EB, key_00000000000002BB] for segment 343 of cache testCache from node edg-perf01-62141
> 11:06:13,454 TRACE [org.infinispan.container.DefaultDataContainer] (remote-thread-91) Store ImmortalCacheEntry{key=key_00000000000002EB, value=[2 #7: 366, 544, 576, 804, 1061, 1181, 1290, ]} in container
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years
[JBoss JIRA] (ISPN-5304) WildFly module "/org/infinispan/client/hotrod/jdg-6.4" is different if installed with library or HR-client
by Wolf-Dieter Fink (JIRA)
Wolf-Dieter Fink created ISPN-5304:
--------------------------------------
Summary: WildFly module "/org/infinispan/client/hotrod/jdg-6.4" is different if installed with library or HR-client
Key: ISPN-5304
URL: https://issues.jboss.org/browse/ISPN-5304
Project: Infinispan
Issue Type: Feature Request
Affects Versions: 7.1.1.Final, 7.2.0.Alpha1
Environment: Infinispan as WildFly module
Reporter: Wolf-Dieter Fink
If Infinispan is used inside WildFly applications there are two WF modules for Infinispan.
One for Library mode
One for HotRod remote client mode
Both modules contain
modules/org/infinispan/client/hotrod/jdg-6.4/infinispan-client-hotrod-6.2.0.Final-redhat-4.jar
modules/org/infinispan/client/hotrod/jdg-6.4/module.xml
BUT
the module.xml is different:
< <module name="org.infinispan" slot="jdg-6.4"/>
---
> <module name="org.infinispan.commons" slot="jdg-6.4"/>
> <module name="org.infinispan.protostream" slot="jdg-6.4"/>
> <module name="org.infinispan.query.dsl" slot="jdg-6.4"/>
> <module name="org.infinispan.query.remote.client" slot="jdg-6.4"/>
This mean it is not possible to use both modules if there are applications with library mode and others with HR remote client mode.
Also it is odd because it depends on the installation order what dependencies are installed.
It should be possible to install both modules and use it at the same time from different applicaitons.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years
[JBoss JIRA] (ISPN-4810) Local Transactional Cache loses data when eviction is enabled and there are multiple readers and one writer
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-4810?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration updated ISPN-4810:
------------------------------------------
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=1196258, https://bugzilla.redhat.com/show_bug.cgi?id=1198302 (was: https://bugzilla.redhat.com/show_bug.cgi?id=1196258)
> Local Transactional Cache loses data when eviction is enabled and there are multiple readers and one writer
> -----------------------------------------------------------------------------------------------------------
>
> Key: ISPN-4810
> URL: https://issues.jboss.org/browse/ISPN-4810
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 6.0.2.Final
> Environment: Windows 7 x64 (NTFS)
> Oracle JDK1.7.0_40
> Apache Maven 3.0.5
> Reporter: Horia Chiorean
> Assignee: William Burns
> Labels: modeshape
> Fix For: 7.2.0.Beta1, 5.2.11.Final
>
> Attachments: ispn_concurrent.zip
>
>
> Using Infinispan 6.0.2 and a local, transactional cache backed by a <singleFile> store, with eviction enabled and a small {{max-entries}} setting, we have the following scenario:
> * the main thread (i.e. the "writer") starts a transaction, adds a batch of strings into the cache and also appends the same strings into a List cache entry and then commits the transaction
> * after the above has finished (i.e. after tx.commit) it fires a number of reader threads where each reader thread
> ** checks that the string entries were added into the cache and
> ** checks that the entries were correctly appended to the List entry
> * the above steps are repeated a number of times
> On any given run, based on timing, we're seeing that at some point (after some time) some of the reader threads will not see the latest version of the List entry (i.e. will not see the latest elements that were added into the list) but rather an old, stale List (sort of a "lost update" scenario).
> If we either:
> * disable eviction or
> * set the {{max-entries}} to a large enough value (which I suspect has the same effect - not evicting anything) the problem doesn't show up.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years