[JBoss JIRA] (ISPN-2410) A command forwarded back to the originator can time out waiting on a key already locked by itself
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2410?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2410:
--------------------------------
Priority: Major (was: Blocker)
> A command forwarded back to the originator can time out waiting on a key already locked by itself
> -------------------------------------------------------------------------------------------------
>
> Key: ISPN-2410
> URL: https://issues.jboss.org/browse/ISPN-2410
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.0.Beta2
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 5.2.0.CR1
>
>
> If a rebalance happens while a prepare command is executing on a remote node, and the originator has become an owner, it makes sense to forward the command back to the originator to lock the keys (or just add them to the backup locks list).
> However, we don't keep the old consistent hashes around, so we don't know if the originator became an owner after invoking the remote command or was already an owner. So if the topology changed, we always forward the prepare back to the owner.
> Back on the originator, minTxTopologyId < currentTopologyId, so the prepare command has to wait for all the backup locks from pending transactions to be released. The problem is that we wait for the current transaction as well, causing a deadlock.
> Seen in OnePhaseXATest:
> {noformat}
> 18:07:46,873 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] NodeA-46125 broadcasting call PrepareCommand {modifications=[PutKeyValueCommand{key=key0, value=value, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:local, cacheName='TestCache', topologyId=-1} to recipient list null
> 18:07:46,873 DEBUG (transport-thread-2,NodeA:TestCache) [LocalTopologyManagerImpl] Updating local consistent hash(es) for cache TestCache: new topology = CacheTopology{id=2, currentCH=ReplicatedConsistentHash{members=[NodeA-46125, NodeB-49450]}, pendingCH=null}
> 18:07:46,894 TRACE (OOB-1,ISPN,NodeB-49450:TestCache) [StateTransferManagerImpl] Forwarding command PrepareCommand {modifications=[PutKeyValueCommand{key=key0, value=value, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:remote, cacheName='TestCache', topologyId=2} to new targets [NodeA-46125]
> 18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [StateTransferInterceptor] handleTopologyAffectedCommand for command PrepareCommand {modifications=[PutKeyValueCommand{key=key0, value=value, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false, gtx=GlobalTransaction:<NodeA-46125>:4353:remote, cacheName='TestCache', topologyId=2}, originLocal=false
> 18:07:46,935 TRACE (OOB-3,ISPN,NodeA-46125:TestCache) [AbstractCacheTransaction] Transaction gtx=GlobalTransaction:<NodeA-46125>:4353:local potentially locks key key0? true
> 18:08:16,874 TRACE (testng-OnePhaseXATest:TestCache) [RpcManagerImpl] replication exception:
> org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to NodeB-49450
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 9 months
[JBoss JIRA] (ISPN-2580) Do not request segments from all nodes at once
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2580?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2580:
--------------------------------
Priority: Blocker (was: Critical)
> Do not request segments from all nodes at once
> ----------------------------------------------
>
> Key: ISPN-2580
> URL: https://issues.jboss.org/browse/ISPN-2580
> Project: Infinispan
> Issue Type: Enhancement
> Components: State transfer
> Affects Versions: 5.2.0.Beta5
> Reporter: Radim Vansa
> Assignee: Adrian Nistor
> Priority: Blocker
> Fix For: 5.2.0.CR1
>
>
> When a new node joins large cluster filled with data, it gets the new CH and REBALANCE_START command, and requests data from all nodes at once (or almost all with even distribution of segments). It may be not able to handle this amount of transfers in parallel even at the JGroups level - this results in data sent to the node and discarded at the receiver, sent again and again. With a heavy congestion the node just buffers fragments of a message from one sender and never passes this up.
> The number of StateRequestCommands(START_STATE_TRANSFER) should be limited so that the node is not congested.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 9 months
[JBoss JIRA] (ISPN-2655) Make HotRod client always read from the main data owner
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-2655?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-2655:
----------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/1551
> Make HotRod client always read from the main data owner
> -------------------------------------------------------
>
> Key: ISPN-2655
> URL: https://issues.jboss.org/browse/ISPN-2655
> Project: Infinispan
> Issue Type: Feature Request
> Affects Versions: 5.2.0.Beta6
> Reporter: Mircea Markus
> Assignee: Tristan Tarrant
> Fix For: 5.2.0.CR1, 5.2.0.Final
>
>
> ISPN-2643 made the java Hot Rod client always write to the main owner. ATM the client picks a random owner for reading though. This read-load-balancing doesn't really help, as assuming the data is evenly spread, the amount of reads would be distributed uniformly across the cluster. OTOH forcing the client to always read from the main owner would guarantee red-consistency for *async* replicated caches. Even so read consistency might still be a problem when a node crashed, but still these is a much stronger guarantee and makes async replication usable in many more scenarios.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 9 months
[JBoss JIRA] (ISPN-2425) Amend the xsite configuration and the xsd
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-2425?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-2425:
----------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> Amend the xsite configuration and the xsd
> -----------------------------------------
>
> Key: ISPN-2425
> URL: https://issues.jboss.org/browse/ISPN-2425
> Project: Infinispan
> Issue Type: Feature Request
> Affects Versions: 5.2.0.Beta2
> Reporter: Mircea Markus
> Assignee: Mircea Markus
> Priority: Blocker
> Fix For: 5.2.0.CR1
>
>
> Change:
> {code:xml}
> <global>
> ...
> <sites local="LON"/>
> ...
> </global>
> {code}
> to
> {code:xml}
> <global>
> ...
> <site name="LON"/>
> ...
> </global>
> {code}
> And:
> {code:xml}
> <backups backupSites="NYC,SFO">
> <backup site="NYC" backupFailurePolicy="WARN" strategy="SYNC" timeout="12000"/>
> <backup site="SFO" backupFailurePolicy="IGNORE" strategy="ASYNC"/>
> <backup site="LON" strategy="SYNC"/>
> </backups>
> {code}
> to:
> {code:xml}
> <backups>
> <backup site="NYC" failurePolicy="WARN" strategy="SYNC" timeout="12000" enabled="true"/>
> <backup site="SFO" failurePolicy="IGNORE" strategy="ASYNC" enabled="true"/>
> <backup site="LON" strategy="SYNC" enabled="false"/>
> </backups>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 9 months
[JBoss JIRA] (ISPN-2550) NoSuchElementException in Hot Rod Encoder
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2550?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2550:
--------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> NoSuchElementException in Hot Rod Encoder
> -----------------------------------------
>
> Key: ISPN-2550
> URL: https://issues.jboss.org/browse/ISPN-2550
> Project: Infinispan
> Issue Type: Bug
> Components: Remote protocols
> Affects Versions: 5.2.0.Beta4
> Reporter: Michal Linhard
> Assignee: Dan Berindei
> Priority: Blocker
> Fix For: 5.2.0.CR1
>
>
> Tomas noticed this a while ago in a specific functional test:
> https://bugzilla.redhat.com/show_bug.cgi?id=875151
> I'm creating a more general JIRA, cause I'm having this in resilience test.
> What I found by quick debug, is that here:
> https://github.com/infinispan/infinispan/blob/master/server/hotrod/src/ma...
> {code}
> for (segmentIdx <- 0 until numSegments) {
> val denormalizedSegmentHashIds = allDenormalizedHashIds(segmentIdx)
> val segmentOwners = ch.locateOwnersForSegment(segmentIdx)
> for (ownerIdx <- 0 until segmentOwners.length) {
> val address = segmentOwners(ownerIdx % segmentOwners.size)
> val serverAddress = members(address)
> val hashId = denormalizedSegmentHashIds(ownerIdx)
> log.tracef("Writing hash id %d for %s:%s", hashId, serverAddress.host, serverAddress.port)
> writeString(serverAddress.host, buf)
> writeUnsignedShort(serverAddress.port, buf)
> buf.writeInt(hashId)
> }
> }
> {code}
> we're trying to obtain serverAddress for nonexistent address and NoSuchElementException is not handled properly.
> It hapens after I kill a node in a resilience test and the exception appears when querying for the node in the members cache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 9 months
[JBoss JIRA] (ISPN-2642) IndexOutOfBoundsException in HotRodDecoder
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2642?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2642:
--------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> IndexOutOfBoundsException in HotRodDecoder
> ------------------------------------------
>
> Key: ISPN-2642
> URL: https://issues.jboss.org/browse/ISPN-2642
> Project: Infinispan
> Issue Type: Bug
> Components: Remote protocols
> Affects Versions: 5.2.0.Beta5
> Reporter: Michal Linhard
> Assignee: Galder Zamarreño
> Fix For: 5.2.0.CR1, 5.2.0.Final
>
>
> In one of the elasticity tests in hyperion I'm getting:
> {code}
> 03:48:04,765 ERROR [org.infinispan.server.hotrod.HotRodDecoder] (HotRodClientMaster-179) ISPN005009: Unexpected error before any request parameters read: java.lang.IndexOutOfBoundsException: 2
> at scala.collection.mutable.ResizableArray$class.apply(ResizableArray.scala:44)
> at scala.collection.mutable.ArrayBuffer.apply(ArrayBuffer.scala:47)
> at org.infinispan.server.hotrod.AbstractTopologyAwareEncoder1x$$anonfun$writeHashTopologyHeader$1$$anonfun$apply$mcVI$sp$1.apply$mcVI$sp(AbstractTopologyAwareEncoder1x.scala:99) [infinispan-server-hotrod-5.2.0.Beta5-redhat-1.jar:5.2.0.Beta5-redhat-1]
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:81)
> at org.infinispan.server.hotrod.AbstractTopologyAwareEncoder1x$$anonfun$writeHashTopologyHeader$1.apply$mcVI$sp(AbstractTopologyAwareEncoder1x.scala:96) [infinispan-server-hotrod-5.2.0.Beta5-redhat-1.jar:5.2.0.Beta5-redhat-1]
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:78)
> at org.infinispan.server.hotrod.AbstractTopologyAwareEncoder1x.writeHashTopologyHeader(AbstractTopologyAwareEncoder1x.scala:93) [infinispan-server-hotrod-5.2.0.Beta5-redhat-1.jar:5.2.0.Beta5-redhat-1]
> at org.infinispan.server.hotrod.AbstractEncoder1x.writeHeader(AbstractEncoder1x.scala:62) [infinispan-server-hotrod-5.2.0.Beta5-redhat-1.jar:5.2.0.Beta5-redhat-1]
> at org.infinispan.server.hotrod.HotRodEncoder.encode(HotRodEncoder.scala:63) [infinispan-server-hotrod-5.2.0.Beta5-redhat-1.jar:5.2.0.Beta5-redhat-1]
> at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:67) [netty-3.5.9.Final.jar:]
> at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:60) [netty-3.5.9.Final.jar:]
> at org.jboss.netty.channel.Channels.write(Channels.java:712) [netty-3.5.9.Final.jar:]
> at org.jboss.netty.channel.Channels.write(Channels.java:679) [netty-3.5.9.Final.jar:]
> at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248) [netty-3.5.9.Final.jar:]
> at org.infinispan.server.core.AbstractProtocolDecoder.writeResponse(AbstractProtocolDecoder.scala:179) [infinispan-server-core-5.2.0.Beta5-redhat-1.jar:5.2.0.Beta5-redhat-1]
> at org.infinispan.server.core.AbstractProtocolDecoder.decodeKey(AbstractProtocolDecoder.scala:115) [infinispan-server-core-5.2.0.Beta5-redhat-1.jar:5.2.0.Beta5-redhat-1]
> at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:71) [infinispan-server-core-5.2.0.Beta5-redhat-1.jar:5.2.0.Beta5-redhat-1]
> at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:47) [infinispan-server-core-5.2.0.Beta5-redhat-1.jar:5.2.0.Beta5-redhat-1]
> at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:502) [netty-3.5.9.Final.jar:]
> at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:437) [netty-3.5.9.Final.jar:]
> at org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:387) [infinispan-server-core-5.2.0.Beta5-redhat-1.jar:5.2.0.Beta5-redhat-1]
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.5.9.Final.jar:]
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.5.9.Final.jar:]
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84) [netty-3.5.9.Final.jar:]
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:472) [netty-3.5.9.Final.jar:]
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:333) [netty-3.5.9.Final.jar:]
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35) [netty-3.5.9.Final.jar:]
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_33]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_33]
> at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_33]
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 9 months
[JBoss JIRA] (ISPN-2534) Some tests are not picked up by Surefire
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-2534?page=com.atlassian.jira.plugin.... ]
Adrian Nistor updated ISPN-2534:
--------------------------------
Fix Version/s: 5.2.0.CR1
(was: 5.2.0.Final)
> Some tests are not picked up by Surefire
> ----------------------------------------
>
> Key: ISPN-2534
> URL: https://issues.jboss.org/browse/ISPN-2534
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite
> Affects Versions: 5.2.0.Beta4
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 5.2.0.CR1
>
>
> Surefire picks up tests named {{\*Test.java}} and {{Test*.java}}, but not {{\*Test\*.java}}.
> There are some tests that don't follow the {{*Test.java}} naming convention, and because of that they don't run in the test suite:
> ./query/src/test/java/org/infinispan/query/config/MultipleCachesTests.java
> ./query/src/test/java/org/infinispan/query/indexedembedded/BooksExampleTests.java
> ./cli/cli-server/src/test/java/org/infinispan/cli/interpreter/ClusteredCLITests.java
> ./cachestore/jdbc/src/test/java/org/infinispan/loaders/jdbc/stringbased/JdbcStringBasedCacheStoreTest2.java
> ./cachestore/jdbc/src/test/java/org/infinispan/loaders/jdbc/stringbased/JdbcStringBasedCacheStoreVamTest2.java
> ./cachestore/jdbc/src/test/java/org/infinispan/loaders/jdbc/mixed/JdbcMixedCacheStoreTest2.java
> ./cachestore/jdbc/src/test/java/org/infinispan/loaders/jdbc/mixed/JdbcMixedCacheStoreVamTest2.java
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 9 months