[JBoss JIRA] (ISPN-3223) HotRod Client recieves stale toplogy view on instance leave
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3223?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-3223:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 988675|https://bugzilla.redhat.com/show_bug.cgi?id=988675] from NEW to ASSIGNED
> HotRod Client recieves stale toplogy view on instance leave
> -----------------------------------------------------------
>
> Key: ISPN-3223
> URL: https://issues.jboss.org/browse/ISPN-3223
> Project: Infinispan
> Issue Type: Bug
> Components: Remote protocols
> Affects Versions: 5.2.4.Final, 5.3.0.CR1
> Reporter: Takayoshi Kimura
> Assignee: Galder Zamarreño
> Fix For: 6.0.0.Final
>
>
> When killed a HotRod server node, HotRod Clinet sometimes recieves a stale toplogy view which includes the dead node and uses it as a latest view. In this case the client keeps trying to connect that node and keeps failing.
> Looks like the AbstractEncoder1x.generateTopologyResponse() takes care of node join but doesn't handle node leave:
> {noformat}
> if (!serverEndpointsMap.keySet.containsAll(cacheMembers)) {
> {noformat}
> For example, serverEndpointsMap.keySet is [A, B, C] and the actual cacheMembers is [A, B].
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-3367) org.infinispan.statetransfer.ClusterTopologyManagerTest.testClusterRecoveryWithRebalance fails randomly
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3367?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-3367:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 988333|https://bugzilla.redhat.com/show_bug.cgi?id=988333] from NEW to ASSIGNED
> org.infinispan.statetransfer.ClusterTopologyManagerTest.testClusterRecoveryWithRebalance fails randomly
> -------------------------------------------------------------------------------------------------------
>
> Key: ISPN-3367
> URL: https://issues.jboss.org/browse/ISPN-3367
> Project: Infinispan
> Issue Type: Bug
> Components: Core API
> Affects Versions: 6.0.0.Alpha1
> Environment: {w2k8r2 OracleJDK1.7}, { RHEL6_x86_64, OracleJDK1.7 and OpenJDK1.7}
> Reporter: Vitalii Chepeliuk
> Assignee: Dan Berindei
> Labels: testsuite_stability
> Fix For: 6.0.0.Final
>
> Attachments: ClusterTopologyManagerTest.log.zip
>
>
> Error Message
> Thread already timed out waiting for event merge
> Stacktrace
> java.lang.IllegalStateException: Thread already timed out waiting for event merge
> at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:131)
> at org.infinispan.test.fwk.CheckPoint.triggerForever(CheckPoint.java:120)
> at org.infinispan.statetransfer.ClusterTopologyManagerTest.testClusterRecoveryWithRebalance(ClusterTopologyManagerTest.java:280)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:715)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Add links to jenkins jobs
> windows, OracleJDK1.7>>>
> https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/JDG/view/FUNC/job/e...
> https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/JDG/view/FUNC/job/e...
> Linux, OracleJDK1.7>>>
> https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/JDG/view/FUNC/job/e...
> Linux, OpenJDK1.7>>>
> https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/JDG/view/FUNC/job/e...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-3357) Insufficient owners with putIfAbsent during node join rebalance
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3357?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-3357:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 989808|https://bugzilla.redhat.com/show_bug.cgi?id=989808] from NEW to ASSIGNED
> Insufficient owners with putIfAbsent during node join rebalance
> ---------------------------------------------------------------
>
> Key: ISPN-3357
> URL: https://issues.jboss.org/browse/ISPN-3357
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.4.Final, 6.0.0.Alpha1
> Reporter: Takayoshi Kimura
> Assignee: Dan Berindei
> Priority: Critical
> Attachments: 7c29bccb.log
>
>
> Here is test scenario:
> * DIST numOwners=2, start with 3 nodes cluster then join 1 node during load
> * HotRod putIfAbsent accesses from 40 threads (1 process, 1 remote cache instance), 40000 entries total
> After the test run, the numberOfEntries on each node are:
> * node1: 20074
> * node2: 19888
> * node3: 20114
> * node4: 18885
> Total is 78961, 1039 entries are missing. No error on HotRod client side so 80000 entries should be there.
> Let's take a look at example missing entry, hash(thread01key151) = 7c29bccb.
> Current CH: owners(7c29bccb) are [node1, node2]
> Pending CH: owners(7c29bccb) are [node1, node2, node4]
> Balanced CH: owners(7c29bccb) are [node1, node4]
> The events sequence is:
> * hotrod -> node1
> * node1 -> node2, node4
> * node2 committed entry
> * node4 performed clustered get before write, got a value from node2 and will not commit the entry because this node thinks it's not changed/created
> * node1 committed entry
> * node2 invalidates the entry because it's no longer an owner
> Result owners(7c29bccb) are only node1 and node4 is missing. This entry may be completely lost by further rebalances when node4 is donor for this segment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-1965) Some entries not available during view change
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-1965?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-1965:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 808623|https://bugzilla.redhat.com/show_bug.cgi?id=808623] from ASSIGNED to MODIFIED
> Some entries not available during view change
> ---------------------------------------------
>
> Key: ISPN-1965
> URL: https://issues.jboss.org/browse/ISPN-1965
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.1.3.FINAL
> Reporter: Michal Linhard
> Assignee: Dan Berindei
>
> In the 4 node, dist mode, num-owners=2, elasticity test
> http://www.qa.jboss.com/~mlinhard/hyperion/run44-elas-dist/
> there is a cca 90 sec period of time where clients get null responses to GET
> requests on entries that should exist in the cache.
> first occurence:
> hyperion1139.log 05:31:01,202 286.409
> last occurence:
> hyperion1135.log 05:32:45,441 390.648
> total occurence count: (in all 19 driver nodes)
> 152241
> (this doesn't mean it happens for 152K keys, because each key is retried after
> erroneous attempt)
> data doesn't seem to be lost, because these errors cease after a while and
> number of entries returns back to normal (see cache_entries.csv)
> this happens approximately in the period between node0001 is killed and cluster
> {node0002 - node0004} is formed (and shortly after).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-2186) Coordinator tries to install new view after graceful shutdown
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-2186?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-2186:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 854665|https://bugzilla.redhat.com/show_bug.cgi?id=854665] from ASSIGNED to MODIFIED
> Coordinator tries to install new view after graceful shutdown
> -------------------------------------------------------------
>
> Key: ISPN-2186
> URL: https://issues.jboss.org/browse/ISPN-2186
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.1.6.FINAL
> Reporter: Michal Linhard
> Assignee: Dan Berindei
> Priority: Minor
> Labels: jdg, jdg6
> Fix For: 5.1.x
>
>
> This is not a serious problem, because so far the only thing I discoverd it causes is a superfluous debug level log message.
> This happened in elasticity tests with radargun,
> In these tests we go from 4 nodes to 8 and back to 4.
> See views installed:
> http://www.qa.jboss.com/~mlinhard/hyperion2/run218-radargun-08-elasticity...
> In this case, each time the node is shutdown it is the coordinator (I'm not sure whether this is accident or coordinators are picked by their seniority in the cluster)
> The shutdown is made gracefully via DefaultCacheManager.stop() and each time this happens I can see an attempt of CacheViewsManagerImpl to install a new view - which doesn't complete because the coordinator shuts down few moments later and the new view is really established by a new coordinator.
> Log from slave on node0003:
> {code}
> 02:37:32,737 INFO [org.radargun.stages.KillStage] (pool-1-thread-1) Tearing down cache wrapper.
> 02:38:02,752 WARN [org.infinispan.transaction.TransactionTable] (pool-1-thread-1) ISPN000100: Stopping, but there are 9 local transactions and 0 remote transactions that did not finish in time.
> 02:38:02,755 DEBUG [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-3,hyperion1098-55173) Installing new view CacheView{viewId=9, members=[hyperion1099-41789, hyperion1097-42149, hyperion1100-42888, hyperion1102-1099, hyperion1101-56019, hyperion1103-38655, hyperion1096-46484]} for cache x
> 02:38:04,279 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-1-thread-1) ISPN000080: Disconnecting and closing JGroups Channel
> 02:38:04,641 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-1-thread-1) ISPN000082: Stopping the RpcDispatcher
> 02:38:04,816 INFO [org.radargun.Slave] (pool-1-thread-1) Finished stage: KillStage {tearDown=true, productName='jdg60', useSmartClassLoading=true, slaveIndex=0, activeSlavesCount=8, totalSlavesCount=8, slaves=[0]}
> 02:38:04,817 INFO [org.radargun.Slave] (main) Ack successfully sent to the master
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-2746) testDistributedPutWithTopologyChanges test randomly fails on all environments for org.infinispan.server.hotrod.HotRodDistributionTest and org.infinispan.server.hotrod.HotRod11DistributionTest
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-2746?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-2746:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 903247|https://bugzilla.redhat.com/show_bug.cgi?id=903247] from ASSIGNED to MODIFIED
> testDistributedPutWithTopologyChanges test randomly fails on all environments for org.infinispan.server.hotrod.HotRodDistributionTest and org.infinispan.server.hotrod.HotRod11DistributionTest
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-2746
> URL: https://issues.jboss.org/browse/ISPN-2746
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 5.2.0.CR2
> Reporter: Anna Manukyan
> Assignee: Dan Berindei
> Labels: testsuite_stability
> Fix For: 5.2.2.Final, 5.3.0.Alpha1
>
>
> The test org.infinispan.server.hotrod.HotRodDistributionTest.testDistributedPutWithTopologyChanges almost always fails on all environments (Solaris, Windows, RHEL). The same issue happens for test org.infinispan.server.hotrod.HotRod11DistributionTest.testDistributedPutWithTopologyChanges with the same error message.
> The error is:
> {code}
> Error Message
> expected [7] but found [6]
> Stacktrace
> java.lang.AssertionError: expected [7] but found [6]
> at org.testng.Assert.fail(Assert.java:89)
> at org.testng.Assert.failNotEquals(Assert.java:489)
> at org.testng.Assert.assertEquals(Assert.java:118)
> at org.testng.Assert.assertEquals(Assert.java:365)
> at org.testng.Assert.assertEquals(Assert.java:375)
> at org.infinispan.server.hotrod.test.HotRodTestingUtil$.assertTopologyId(HotRodTestingUtil.scala:306)
> at org.infinispan.server.hotrod.test.HotRodTestingUtil$.assertHashTopology10Received(HotRodTestingUtil.scala:230)
> at org.infinispan.server.hotrod.test.HotRodTestingUtil$.assertHashTopology10Received(HotRodTestingUtil.scala:217)
> at org.infinispan.server.hotrod.HotRodDistributionTest.testDistributedPutWithTopologyChanges(HotRodDistributionTest.scala:119)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:715)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-3088) Put long values from CLI, raise java.lang.NumberFormatException: For input string: "1000l" on server
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3088?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-3088:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 961334|https://bugzilla.redhat.com/show_bug.cgi?id=961334] from ASSIGNED to MODIFIED
> Put long values from CLI, raise java.lang.NumberFormatException: For input string: "1000l" on server
> ----------------------------------------------------------------------------------------------------
>
> Key: ISPN-3088
> URL: https://issues.jboss.org/browse/ISPN-3088
> Project: Infinispan
> Issue Type: Bug
> Components: CLI
> Affects Versions: 5.2.4.Final
> Environment: ALL
> Reporter: Vitalii Chepeliuk
> Assignee: Tristan Tarrant
> Labels: cli
> Fix For: 5.3.0.CR2
>
>
> From "help put" in CLI we get that a long is identified by a sequence of decimal digits suffixed by 'l', e.g. 1000l
> But when put some data to cache e.g put num 1000l we get follow message on client side------------------------------------------------------------------
> [remoting://localhost:9999/local/default]> put num 1000l
> For input string: "1000l"
> And exception on server side-------------------------------------------------------------------------------------------------------------------
> 14:17:23,179 ERROR [org.infinispan.cli.interpreter.Interpreter] (pool-1-thread-1) ISPN019003: Interpreter error: java.lang.NumberFormatException: For input string: "1000l"
> at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_17]
> at java.lang.Long.parseLong(Long.java:441) [rt.jar:1.7.0_17]
> at java.lang.Long.valueOf(Long.java:540) [rt.jar:1.7.0_17]
> at org.infinispan.cli.interpreter.IspnQLParser.literal(IspnQLParser.java:3791) [infinispan-cli-server-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
> at org.infinispan.cli.interpreter.IspnQLParser.putStatement(IspnQLParser.java:2147) [infinispan-cli-server-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
> at org.infinispan.cli.interpreter.IspnQLParser.statement(IspnQLParser.java:689) [infinispan-cli-server-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
> at org.infinispan.cli.interpreter.IspnQLParser.statements(IspnQLParser.java:237) [infinispan-cli-server-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
> at org.infinispan.cli.interpreter.Interpreter.execute(Interpreter.java:157) [infinispan-cli-server-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_17]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_17]
> at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_17]
> at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:287) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) [rt.jar:1.7.0_17]
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:792) [rt.jar:1.7.0_17]
> at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.invoke(PluggableMBeanServerImpl.java:498) [jboss-as-jmx-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4]
> at org.jboss.as.jmx.PluggableMBeanServerImpl.invoke(PluggableMBeanServerImpl.java:246) [jboss-as-jmx-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4]
> at org.jboss.remotingjmx.protocol.v1.ServerProxy$InvokeHandler.handle(ServerProxy.java:1034)
> at org.jboss.remotingjmx.protocol.v1.ServerProxy$MessageReciever$1.run(ServerProxy.java:215)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_17]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_17]
> at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_17]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months