[JBoss JIRA] (ISPN-4480) Messages sent to leavers can clog the JGroups bundler thread
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-4480?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-4480:
-----------------------------------------------
Alan Field <afield(a)redhat.com> changed the Status of [bug 1116965|https://bugzilla.redhat.com/show_bug.cgi?id=1116965] from ON_QA to VERIFIED
> Messages sent to leavers can clog the JGroups bundler thread
> ------------------------------------------------------------
>
> Key: ISPN-4480
> URL: https://issues.jboss.org/browse/ISPN-4480
> Project: Infinispan
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Components: Core
> Affects Versions: 6.0.2.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
>
> In a stress test that repeatedly kills nodes while performing read/write operations, the TransferQueueBundler thread seems to spend a lot of time waiting for physical addresses:
> {noformat}
> 06:40:10,316 WARN [org.radargun.utils.Utils] (pool-5-thread-1) Stack for thread TransferQueueBundler,default,apex953-14666:
> java.lang.Thread.sleep(Native Method)
> org.jgroups.util.Util.sleep(Util.java:1504)
> org.jgroups.util.Util.sleepRandom(Util.java:1574)
> org.jgroups.protocols.TP.sendToSingleMember(TP.java:1685)
> org.jgroups.protocols.TP.doSend(TP.java:1670)
> org.jgroups.protocols.TP$TransferQueueBundler.sendBundledMessages(TP.java:2476)
> org.jgroups.protocols.TP$TransferQueueBundler.sendMessages(TP.java:2392)
> org.jgroups.protocols.TP$TransferQueueBundler.run(TP.java:2383)
> java.lang.Thread.run(Thread.java:744)
> {noformat}
> There are 2 bugs related to this already fixed in JGroups 3.5.0.Beta2+: JGRP-1814, JGRP-1815
> There is also a special case where the physical address could be removed from the cache too soon, exacerbating the effect of JGRP-1815: JGRP-1858
> We can work around the problem by changing the JGroups configuration:
> * TP.logical_addr_cache_expiration=86400000
> ** Only expire addresses after 1 day
> * TP.physical_addr_max_fetch_attempts=1
> ** Sleep for only 20ms waiting for the physical address (default 3 - 1500ms)
> * UNICAST3_conn_close_timeout=10000
> ** Drop the pending messages to leavers sooner
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
11 years, 9 months
[JBoss JIRA] (ISPN-4512) CacheManagerTest.testCacheManagerRestartReusingConfigurations random failures
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-4512?page=com.atlassian.jira.plugin.... ]
Work on ISPN-4512 started by William Burns.
> CacheManagerTest.testCacheManagerRestartReusingConfigurations random failures
> -----------------------------------------------------------------------------
>
> Key: ISPN-4512
> URL: https://issues.jboss.org/browse/ISPN-4512
> Project: Infinispan
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Components: Core, Test Suite - Core
> Affects Versions: 7.0.0.Alpha4
> Reporter: Dan Berindei
> Assignee: William Burns
> Priority: Blocker
> Labels: testsuite_stability
> Fix For: 7.0.0.Alpha5
>
> Attachments: CacheManagerTest_t_ISPN-4154_failing_elasticity_test_20140707.log.gz
>
>
> When a new cache manager is started with the same configuration, it uses the JGroupsTransport instance. In some rare cases, the JGroupsTransport keeps using the old marshaller, which doesn't work, and the cache fails to start:
> {noformat}
> 23:54:08,203 TRACE (testng-CacheManagerTest:___defaultcache) [JGroupsTransport] dests=[NodeB-24139], command=CacheTopologyControlCommand{cache=___defaultcache, type=JOIN, sender=NodeA-33664, joinInfo=CacheJoinInfo{consistentHashFactory=org.infinispan.distribution.ch.impl.ReplicatedConsistentHashFactory@b8c8791, hashFunction=MurmurHash3, numSegments=60, numOwners=2, timeout=240000, totalOrder=false, distributed=false}, topologyId=0, currentCH=null, pendingCH=null, throwable=null, viewId=3}, mode=SYNCHRONOUS, timeout=240000
> 23:54:08,207 DEBUG (testng-CacheManagerTest:___defaultcache) [VersionAwareMarshaller] Object is not serializable
> java.io.NotSerializableException: org.infinispan.topology.CacheTopologyControlCommand
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:890)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
> at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:73)
> at org.infinispan.marshall.core.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:77)
> at org.infinispan.commons.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:41)
> at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:85)
> at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:23)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:335)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:352)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:165)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:526)
> at org.infinispan.topology.LocalTopologyManagerImpl.executeOnCoordinator(LocalTopologyManagerImpl.java:290)
> at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:100)
> at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:104)
> {noformat}
> The only test that does this is CacheManagerTest.testCacheManagerRestartReusingConfigurations.
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
11 years, 9 months
[JBoss JIRA] (ISPN-4524) ClusteredCacheWithAsyncDirTest.testCombinationOfFilters random failures
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/ISPN-4524?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero commented on ISPN-4524:
---------------------------------------
Thanks Dan.
Since it runs the same tests as the parent, with the sole difference of having an asynchronous CacheStore enabled, I think the next step is to narrow down if the problem is in either of:
- Async being broken again for CacheStores (especially under load).
- The Lucene Directory applying some incorrect expectation regarding instance uniqueness of some of its structures (assumption which fails when a CacheStore is enabled)
Both mistakes happened frequently in the past. Seems like a challenge to keep [~gustavonalle] entertained :-)
> ClusteredCacheWithAsyncDirTest.testCombinationOfFilters random failures
> ------------------------------------------------------------------------
>
> Key: ISPN-4524
> URL: https://issues.jboss.org/browse/ISPN-4524
> Project: Infinispan
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Components: Test Suite - Query
> Affects Versions: 7.0.0.Alpha4
> Reporter: Dan Berindei
> Assignee: Gustavo Fernandes
> Priority: Blocker
> Labels: testsuite_stability
> Fix For: 7.0.0.Alpha5
>
>
> The test fails in the CI build with TRACE enabled almost every time:
> http://ci.infinispan.org/project.html?tab=testDetails&testNameId=64155977...
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
11 years, 9 months
[JBoss JIRA] (ISPN-3594) Implementations of persistence.ParallelIterationTest fail randomly on all environmets
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-3594?page=com.atlassian.jira.plugin.... ]
Work on ISPN-3594 started by Pedro Ruivo.
> Implementations of persistence.ParallelIterationTest fail randomly on all environmets
> -------------------------------------------------------------------------------------
>
> Key: ISPN-3594
> URL: https://issues.jboss.org/browse/ISPN-3594
> Project: Infinispan
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Components: Core, Loaders and Stores
> Affects Versions: 6.0.0.Beta1, 7.0.0.Alpha1, 7.0.0.Alpha2
> Environment: RHEL{5, 6} && {x86_64, x64} && JDK6 && {Windows 2012 64bit && JDK6}
> Reporter: Anna Manukyan
> Assignee: Pedro Ruivo
> Priority: Blocker
> Labels: testsuite_stability
> Fix For: 7.0.0.Alpha5
>
>
> The tests extending from org.infinispan.persistence.ParallelIterationTest fail randomly on RHEL machines for JDK6.
> The error messages are:
> JdbcBinaryStoreParallelIterationTest.testParallelIteration
> {code}
> java.lang.AssertionError: expected [5] but found [2]
> at org.testng.Assert.fail(Assert.java:89)
> at org.testng.Assert.failNotEquals(Assert.java:489)
> at org.testng.Assert.assertEquals(Assert.java:118)
> at org.testng.Assert.assertEquals(Assert.java:365)
> at org.testng.Assert.assertEquals(Assert.java:375)
> at org.infinispan.persistence.ParallelIterationTest.runIterationTest(ParallelIterationTest.java:117)
> at org.infinispan.persistence.ParallelIterationTest.testParallelIteration(ParallelIterationTest.java:58)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:715)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> SingleFileStoreParallelIterationTest.testParallelIteration
> {code}
> java.lang.AssertionError: expected [5] but found [4]
> at org.testng.Assert.fail(Assert.java:89)
> at org.testng.Assert.failNotEquals(Assert.java:489)
> at org.testng.Assert.assertEquals(Assert.java:118)
> at org.testng.Assert.assertEquals(Assert.java:365)
> at org.testng.Assert.assertEquals(Assert.java:375)
> at org.infinispan.persistence.ParallelIterationTest.runIterationTest(ParallelIterationTest.java:117)
> at org.infinispan.persistence.ParallelIterationTest.testParallelIteration(ParallelIterationTest.java:58)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:715)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> LevelDBParallelIterationTest.testParallelIteration (from TestSuite)
> {code}
> expected [5] but found [4]
> Stacktrace
> java.lang.AssertionError: expected [5] but found [4]
> at org.testng.Assert.fail(Assert.java:94)
> at org.testng.Assert.failNotEquals(Assert.java:494)
> at org.testng.Assert.assertEquals(Assert.java:123)
> at org.testng.Assert.assertEquals(Assert.java:370)
> at org.testng.Assert.assertEquals(Assert.java:380)
> at org.infinispan.persistence.ParallelIterationTest.runIterationTest(ParallelIterationTest.java:117)
> at org.infinispan.persistence.ParallelIterationTest.testParallelIteration(ParallelIterationTest.java:58)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
> at java.lang.reflect.Method.invoke(Method.java:619)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:273)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1170)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:640)
> at java.lang.Thread.run(Thread.java:853)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
11 years, 9 months
[JBoss JIRA] (ISPN-4520) JdbcBinaryStoreTest.testLoadAndStoreWithLifespanAndIdle random failures
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-4520?page=com.atlassian.jira.plugin.... ]
Work on ISPN-4520 started by Pedro Ruivo.
> JdbcBinaryStoreTest.testLoadAndStoreWithLifespanAndIdle random failures
> -----------------------------------------------------------------------
>
> Key: ISPN-4520
> URL: https://issues.jboss.org/browse/ISPN-4520
> Project: Infinispan
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Components: Core, Test Suite - Core
> Affects Versions: 7.0.0.Alpha4
> Reporter: Dan Berindei
> Assignee: Pedro Ruivo
> Priority: Blocker
> Labels: testsuite_stability
> Fix For: 7.0.0.Alpha5
>
>
> I think the 1s timeout is a bit small for the CI machine:
> {noformat}
> java.lang.IllegalStateException: Purge has timed out
> at org.infinispan.persistence.BaseStoreTest.purgeExpired(BaseStoreTest.java:268)
> {noformat}
> The other methods using {{BaseStoreTest.purgeExpired}} and the other tests extending BaseStoreTest probably have the same problem.
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
11 years, 9 months