[JBoss JIRA] (ISPN-4846) State transfer keeps trying to fetch transaction data after the cache was stopped
by Dan Berindei (JIRA)
Dan Berindei created ISPN-4846:
----------------------------------
Summary: State transfer keeps trying to fetch transaction data after the cache was stopped
Key: ISPN-4846
URL: https://issues.jboss.org/browse/ISPN-4846
Project: Infinispan
Issue Type: Bug
Components: Core, State Transfer
Affects Versions: 7.0.0.CR1
Reporter: Dan Berindei
Fix For: 7.0.0.CR2
StateConsumerImpl doesn't check if the cache is stopped while fetching transaction data, it only stops when it's no longer able to find providers for transactions.
However, JGroupsTransport throws a generic CacheException when the channel is stopped. The state transfer thread can enter a busy-wait loop, retrying to get the transaction data and immediately getting the CacheException, filling the log with messages like this:
{noformat}
19:32:28,237 WARN (remote-thread-NodeN-p42592-t1:) [StateConsumerImpl] ISPN000209: Failed to retrieve transactions for segments [10, 11, 12, 13, 14, 15, 17, 16, 19, 18, 21, 20, 23, 22, 25, 24, 27, 26, 29, 28, 42, 43, 40, 41, 46, 47, 44, 45, 51, 50, 49, 48, 55, 54, 53, 52, 59, 58, 57, 56] of cache testCache from node NodeM-53416
org.infinispan.commons.CacheException: java.lang.IllegalStateException: channel is not connected
at org.infinispan.commons.util.Util.rewrapAsCacheException(Util.java:655)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:176)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:536)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:290)
at org.infinispan.statetransfer.StateConsumerImpl.getTransactions(StateConsumerImpl.java:766)
at org.infinispan.statetransfer.StateConsumerImpl.requestTransactions(StateConsumerImpl.java:685)
at org.infinispan.statetransfer.StateConsumerImpl.addTransfers(StateConsumerImpl.java:629)
at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:331)
at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:195)
at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:43)
at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:116)
{noformat}
We should check is the cache is stopped before retrying in StateConsumerImpl.requestTransactions. I also think we should change the stop order - it would make sense to stop the remote executor threads and the RpcDispatcher before we stop the channel.
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 5 months
[JBoss JIRA] (ISPN-4841) TopologyAwareConsistentHashFactory is slow for large cluster
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-4841?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-4841:
-----------------------------------------------
Dan Berindei <dberinde(a)redhat.com> changed the Status of [bug 1152934|https://bugzilla.redhat.com/show_bug.cgi?id=1152934] from NEW to ASSIGNED
> TopologyAwareConsistentHashFactory is slow for large cluster
> ------------------------------------------------------------
>
> Key: ISPN-4841
> URL: https://issues.jboss.org/browse/ISPN-4841
> Project: Infinispan
> Issue Type: Enhancement
> Components: Core
> Affects Versions: 7.0.0.CR1
> Reporter: Takayoshi Kimura
>
> A user observed 100% CPU usage for a long time on coordinator node when booting 500 nodes with 500 caches defined.
> It looks like the TopologyAwareConsistentHashFactory performs O(n^2), it has double loop for all Machines. It takes 50 sec to compute rebalance with 1 cache 500 nodes. This calculation is performed on every cache, so it eats 25000 sec CPU times with 500 nodes 500 caches.
> The hprof shows 90% of the time is consumed in the TopologyInfo.computeMaxSegmentsForMachine().
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 5 months
[JBoss JIRA] (ISPN-800) Infinispan inside OSGI
by Ion Savin (JIRA)
[ https://issues.jboss.org/browse/ISPN-800?page=com.atlassian.jira.plugin.s... ]
Ion Savin updated ISPN-800:
---------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> Infinispan inside OSGI
> ----------------------
>
> Key: ISPN-800
> URL: https://issues.jboss.org/browse/ISPN-800
> Project: Infinispan
> Issue Type: Feature Request
> Components: Core
> Reporter: Luca Stancapiano
> Assignee: Ion Savin
>
> We need to import infinispan inside a OSGI repository. Tests are made with Felix.
> I added the configuration to use infinispan inside a osgi repository. We need to ignore all listed dependencies. With this configuration we can install infinispan-core.jar inside OSGI. Its achievement will be as a base installation here: https://github.com/flashboss/infinispan
> I added the Import-Package because you are forced to put manually in Felix all dependencies as jgroups, jboss marshalling, jcip, all apache commons. I've seen infinispan core working by default without all those libraries, so I think the same achievement should be replicated in OSGI.
> Inside the Import-Package tag I excluded those libraries so Infinispan core can be started in default mode without errors. If we want use the replication in OSGI, it is enough add manually the other packages (jgroups.jar etc etc)
> Actually the core bundle can be installed. But to be used it needs theese projects be installed as osgi bundles:
> jboss transaction api 1.0.1.GA
> We patched it. There is a new OSGI version here: https://repository.jboss.org/nexus/content/groups/public/org/jboss/spec/j... )
> jgroups 2.10.1.GA
> (it's a osgi bundle since the 3.x version)
> river 1.2.3.GA
> (opened an issue for marshalling 1.4.0 in JBMAR-118 and https://github.com/flashboss/jboss-marshalling/blob/master/river/pom.xml )
> marshalling-api 1.2.3.GA
> (opened an issue for marshalling 1.4.0 in JBMAR-118 and https://github.com/flashboss/jboss-marshalling/blob/master/api/pom.xml )
> jboss logging spi 2.0.5.GA
> (added a jira issue in JBLOGGING-51 . It could be fixed in the 2.2.0.CR2 version. Fixed in the 3.x version)
> rhq plugin annotations 1.4.0.B01
> (opened a feature request in https://bugzilla.redhat.com/show_bug.cgi?id=657754 )
> i18nlog 1.0.9
> (sent a patch in https://sourceforge.net/projects/i18nlog . It could become a OSGI bundle in the 1.0.10 version. Waiting for a response. Fixed in 1.15)
> log4j 1.2.16
> (that's ok...it is a osgi bundle ;))
> jcip-annotations 1.0
> (I sent a patch via email to brian(a)briangoetz.com and a post in http://tembrel.blogspot.com. Sent the patch in concurrency-interest(a)cs.oswego.edu too. They responded to me. There is a OSGI version with a different artifact name. I changed the dependency in the pom.xml of the parent project)
> We should make sure proper 'Import-Package' property is specified in the MANIFEST.MF so that:
> 1- it fails to load obviously when there's any missing bundles that are essential in using the very core functionality of Infinispan.
> 2 - it does not fail due to the dependency that is not really essential.
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 5 months
[JBoss JIRA] (ISPN-4027) TransactionTable.start() initialize the TxCleanupService thread pool even when the cache is NON_TRANSACTIONAL
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-4027?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration updated ISPN-4027:
------------------------------------------
Bugzilla Update: Perform
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=1152942
> TransactionTable.start() initialize the TxCleanupService thread pool even when the cache is NON_TRANSACTIONAL
> -------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-4027
> URL: https://issues.jboss.org/browse/ISPN-4027
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 6.0.1.Final
> Reporter: Guillermo GARCIA OCHOA
> Assignee: Takayoshi Kimura
> Labels: 630
>
> In the {{TransactionTable.start()}} each cache creates a thread pool and a job is scheduled to clean up completed transactions.
> {code:java}
> private void start() {
> ...
> totalOrder = configuration.transaction().transactionProtocol().isTotalOrder();
> if (!totalOrder) {
> // Periodically run a task to cleanup the transaction table from completed transactions.
> ThreadFactory tf = new ThreadFactory() {
> @Override
> public Thread newThread(Runnable r) {
> String address = rpcManager != null ? rpcManager.getTransport().getAddress().toString() : "local";
> Thread th = new Thread(r, "TxCleanupService," + cacheName + "," + address);
> th.setDaemon(true);
> return th;
> }
> };
> executorService = Executors.newSingleThreadScheduledExecutor(tf);
> long interval = configuration.transaction().reaperWakeUpInterval();
> executorService.scheduleAtFixedRate(new Runnable() {
> @Override
> public void run() {
> cleanupCompletedTransactions();
> }
> }, interval, interval, TimeUnit.MILLISECONDS);
> }
> }
> {code}
> As you can see in the code, even is the cache is {{NON_TRANSACTIONAL}} the job is scheduled, consuming resources to do nothing (the {{completedTransactions}} map is always empty)
> Maybe I'm missing something, but our application profiling is showing us that these threads do nothing but they are consuming precious resources because we have more than 1000 {{NON_TRANSACTIONAL}} caches.
> (i) This can be considered when solving ISPN-3702 too.
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 5 months
[JBoss JIRA] (ISPN-4826) X-Site State transfer values not propagated correctly
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-4826?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-4826:
-------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> X-Site State transfer values not propagated correctly
> ------------------------------------------------------
>
> Key: ISPN-4826
> URL: https://issues.jboss.org/browse/ISPN-4826
> Project: Infinispan
> Issue Type: Bug
> Components: Cross-Site Replication
> Affects Versions: 7.0.0.CR1
> Reporter: Matej Čimbora
> Assignee: Pedro Ruivo
> Fix For: 7.0.0.CR2
>
>
> Used configuration:
> a) SITE1: 2 nodes, cache testCacheSite1
> <backups>
> <backup site="SITE2"/>
> </backups>
> b) SITE2: 3 nodes, cache testCacheSite1_backup – backup cache for testCacheSite1
> <backup-for remote-cache="testCacheSite1" remote-site="SITE1"/>
> When using backup cache with name (testCacheSite1_backup) different from the name of the main cache in SITE1 (testCacheSite1), the data is not propagated to the backup cache completely. The issue seems to be fixed by using the same name for the backup cache (testCacheSite1).
> Scenario
> 1. Start site1 and write data into it (1000 entries)
> 2. Start site2 and invoke XsiteAdminOperations.pushState(“SITE2”)
> 3. Wait 2 minutes
> 4. Check whether the state was transferred to site2 (tested on dist & repl backup cache configs)
> a) distributed mode (numOwners=2) - expected 2000 entries in total, was 648 on site2 master & 0 on other nodes
> b) replicated mode – expected 3000 entries in total, was 1000 on site2 master & 0 on other nodes
>
> Trace log:
> 04:14:39,116 TRACE [org.infinispan.remoting.InboundInvocationHandlerImpl] (OOB-10,edg-perf13-23152) Silently ignoring that testCacheSite1 cache is not defined
> 04:14:39,375 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-10,edg-perf13-23152) Attempting to execute command: SingleRpcCommand{cacheName='testCacheSite1_backup', command=PutKeyValueCommand{key=key_0000000000000001, value=value_key_0000000000000001_SITE1_ORIGINAL@testCacheSite1, flags=[SKIP_REMOTE_LOOKUP, PUT_FOR_X_SITE_STATE_TRANSFER, IGNORE_RETURN_VALUES, SKIP_XSITE_BACKUP], putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedMetadata{version=null}, successful=true}} [sender=edg-perf14-31850]
> 04:14:39,376 TRACE [org.infinispan.statetransfer.StateTransferLockImpl] (OOB-10,edg-perf13-23152) Checking if transaction data was received for topology 4, current topology is 4
> 04:14:39,376 TRACE [org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl] (OOB-10,edg-perf13-23152) Added a new task: 0 task(s) are waiting
> 04:14:39,376 TRACE [org.infinispan.remoting.InboundInvocationHandlerImpl] (remote-thread--p3-t2) Calling perform() on SingleRpcCommand{cacheName='testCacheSite1_backup', command=PutKeyValueCommand{key=key_0000000000000001, value=value_key_0000000000000001_SITE1_ORIGINAL@testCacheSite1, flags=[SKIP_REMOTE_LOOKUP, PUT_FOR_X_SITE_STATE_TRANSFER, IGNORE_RETURN_VALUES, SKIP_XSITE_BACKUP], putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedMetadata{version=null}, successful=true}}
> 04:14:39,378 TRACE [org.infinispan.commands.remote.BaseRpcInvokingCommand] (remote-thread--p3-t2) Invoking command PutKeyValueCommand{key=key_0000000000000001, value=value_key_0000000000000001_SITE1_ORIGINAL@testCacheSite1, flags=[SKIP_REMOTE_LOOKUP, PUT_FOR_X_SITE_STATE_TRANSFER, IGNORE_RETURN_VALUES, SKIP_XSITE_BACKUP], putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedMetadata{version=null}, successful=true}, with originLocal flag set to false
> 04:14:39,378 TRACE [org.infinispan.interceptors.InvocationContextInterceptor] (remote-thread--p3-t2) Invoked with command PutKeyValueCommand{key=key_0000000000000001, value=value_key_0000000000000001_SITE1_ORIGINAL@testCacheSite1, flags=[SKIP_REMOTE_LOOKUP, PUT_FOR_X_SITE_STATE_TRANSFER, IGNORE_RETURN_VALUES, SKIP_XSITE_BACKUP], putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedMetadata{version=null}, successful=true} and InvocationContext [org.infinispan.context.impl.NonTxInvocationContext@266883cb]
> 04:14:39,379 TRACE [org.infinispan.statetransfer.StateTransferInterceptor] (remote-thread--p3-t2) handleNonTxWriteCommand for command PutKeyValueCommand{key=key_0000000000000001, value=value_key_0000000000000001_SITE1_ORIGINAL@testCacheSite1, flags=[SKIP_REMOTE_LOOKUP, PUT_FOR_X_SITE_STATE_TRANSFER, IGNORE_RETURN_VALUES, SKIP_XSITE_BACKUP], putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedMetadata{version=null}, successful=true}
> 04:14:39,380 TRACE [org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor] (remote-thread--p3-t2) Are (edg-perf13-23152) we the lock owners for key 'key_0000000000000001'? false
> 04:14:39,380 TRACE [org.infinispan.interceptors.EntryWrappingInterceptor] (remote-thread--p3-t2) Wrapping entry 'key_0000000000000001'? true
> 04:14:39,380 TRACE [org.infinispan.container.EntryFactoryImpl] (remote-thread--p3-t2) Exists in context? null
> 04:14:39,382 TRACE [org.infinispan.container.EntryFactoryImpl] (remote-thread--p3-t2) Retrieved from container null (isL1Enabled=false, isLocal=true)
> 04:14:39,382 TRACE [org.infinispan.container.EntryFactoryImpl] (remote-thread--p3-t2) Creating new entry.
> 04:14:39,388 TRACE [org.infinispan.container.EntryFactoryImpl] (remote-thread--p3-t2) Wrap key_0000000000000001 for put. Entry=ReadCommittedEntry(197b92bc){key=key_0000000000000001, value=null, oldValue=null, isCreated=true, isChanged=false, isRemoved=false, isValid=true, skipRemoteGet=false, metadata=EmbeddedMetadata{version=null}}
> 04:14:39,390 TRACE [org.infinispan.interceptors.CallInterceptor] (remote-thread--p3-t2) Executing command: PutKeyValueCommand{key=key_0000000000000001, value=value_key_0000000000000001_SITE1_ORIGINAL@testCacheSite1, flags=[SKIP_REMOTE_LOOKUP, PUT_FOR_X_SITE_STATE_TRANSFER, IGNORE_RETURN_VALUES, SKIP_XSITE_BACKUP], putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedMetadata{version=null}, successful=true}.
> 04:14:39,391 TRACE [org.infinispan.interceptors.EntryWrappingInterceptor] (remote-thread--p3-t2) About to commit entry ReadCommittedEntry(197b92bc){key=key_0000000000000001, value=value_key_0000000000000001_SITE1_ORIGINAL@testCacheSite1, oldValue=null, isCreated=true, isChanged=true, isRemoved=false, isValid=true, skipRemoteGet=false, metadata=EmbeddedMetadata{version=null}}
> 04:14:39,392 TRACE [org.infinispan.statetransfer.CommitManager] (remote-thread--p3-t2) Trying to commit. Key=key_0000000000000001. Operation Flag=PUT_FOR_X_SITE_STATE_TRANSFER, L1 invalidation=false
> 04:14:39,392 TRACE [org.infinispan.statetransfer.CommitManager] (remote-thread--p3-t2) Not committing key=key_0000000000000001. It is a state transfer key but no track is enabled!
> 04:14:39,392 TRACE [org.infinispan.interceptors.EntryWrappingInterceptor] (remote-thread--p3-t2) The return value is null
> Suspicious lines:
> 04:14:39,116 TRACE [org.infinispan.remoting.InboundInvocationHandlerImpl] (OOB-10,edg-perf13-23152) Silently ignoring that testCacheSite1 cache is not defined
> 04:14:39,392 TRACE [org.infinispan.statetransfer.CommitManager] (remote-thread--p3-t2) Not committing key=key_0000000000000001. It is a state transfer key but no track is enabled!
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 5 months
[JBoss JIRA] (ISPN-4845) statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus fails randomly
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-4845?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration updated ISPN-4845:
------------------------------------------
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=987461, https://bugzilla.redhat.com/show_bug.cgi?id=1138572 (was: https://bugzilla.redhat.com/show_bug.cgi?id=987461)
> statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus fails randomly
> -------------------------------------------------------------------------------------
>
> Key: ISPN-4845
> URL: https://issues.jboss.org/browse/ISPN-4845
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 5.2.10.Final
> Reporter: Michal Vinkler
> Labels: 5.2.x
>
> Seen with EAP 6.3.0.ER10, Infinispan 5.2.10
> Test org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus randomly fails (seen on Solaris and HP-UX).
> Might be the same as ISPN-4743.
> Stacktraces:
> HP-UX version
> Error Message
> {code}
> Timed out waiting for rebalancing to complete on node ClusterTopologyManagerTest-NodeB-47391, expected member list is [ClusterTopologyManagerTest-NodeB-47391], current member list is [ClusterTopologyManagerTest-NodeB-47391, ClusterTopologyManagerTest-NodeC-55740]!
> {code}
> Stacktrace
> {code}
> java.lang.RuntimeException: Timed out waiting for rebalancing to complete on node ClusterTopologyManagerTest-NodeB-47391, expected member list is [ClusterTopologyManagerTest-NodeB-47391], current member list is [ClusterTopologyManagerTest-NodeB-47391, ClusterTopologyManagerTest-NodeC-55740]!
> at org.infinispan.test.TestingUtil.waitForRehashToComplete(TestingUtil.java:203)
> at org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus(ClusterTopologyManagerTest.java:353)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> Also see standard output:
> https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Infi...
> Solaris version
> Error Message
> {code}
> Thread already timed out waiting for event 3 left
> {code}
> Stacktrace
> {code}
> java.lang.IllegalStateException: Thread already timed out waiting for event 3 left
> at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:150)
> at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:135)
> at org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus(ClusterTopologyManagerTest.java:350)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> Also see standard output:
> https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Infi...
> Might be the same as ISPN-4743.
> Downstream BZ was: https://bugzilla.redhat.com/show_bug.cgi?id=987461
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 5 months
[JBoss JIRA] (ISPN-4027) TransactionTable.start() initialize the TxCleanupService thread pool even when the cache is NON_TRANSACTIONAL
by Takayoshi Kimura (JIRA)
[ https://issues.jboss.org/browse/ISPN-4027?page=com.atlassian.jira.plugin.... ]
Takayoshi Kimura updated ISPN-4027:
-----------------------------------
Summary: TransactionTable.start() initialize the TxCleanupService thread pool even when the cache is NON_TRANSACTIONAL (was: TransactionTable.start() initialize the TxServiceCleanup thread pool even when the cache is NON_TRANSACTIONAL)
> TransactionTable.start() initialize the TxCleanupService thread pool even when the cache is NON_TRANSACTIONAL
> -------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-4027
> URL: https://issues.jboss.org/browse/ISPN-4027
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 6.0.1.Final
> Reporter: Guillermo GARCIA OCHOA
> Assignee: Takayoshi Kimura
> Labels: 630
>
> In the {{TransactionTable.start()}} each cache creates a thread pool and a job is scheduled to clean up completed transactions.
> {code:java}
> private void start() {
> ...
> totalOrder = configuration.transaction().transactionProtocol().isTotalOrder();
> if (!totalOrder) {
> // Periodically run a task to cleanup the transaction table from completed transactions.
> ThreadFactory tf = new ThreadFactory() {
> @Override
> public Thread newThread(Runnable r) {
> String address = rpcManager != null ? rpcManager.getTransport().getAddress().toString() : "local";
> Thread th = new Thread(r, "TxCleanupService," + cacheName + "," + address);
> th.setDaemon(true);
> return th;
> }
> };
> executorService = Executors.newSingleThreadScheduledExecutor(tf);
> long interval = configuration.transaction().reaperWakeUpInterval();
> executorService.scheduleAtFixedRate(new Runnable() {
> @Override
> public void run() {
> cleanupCompletedTransactions();
> }
> }, interval, interval, TimeUnit.MILLISECONDS);
> }
> }
> {code}
> As you can see in the code, even is the cache is {{NON_TRANSACTIONAL}} the job is scheduled, consuming resources to do nothing (the {{completedTransactions}} map is always empty)
> Maybe I'm missing something, but our application profiling is showing us that these threads do nothing but they are consuming precious resources because we have more than 1000 {{NON_TRANSACTIONAL}} caches.
> (i) This can be considered when solving ISPN-3702 too.
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 5 months
[JBoss JIRA] (ISPN-4845) statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus fails randomly
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-4845?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-4845:
-----------------------------------------------
Michal Vinkler <mvinkler(a)redhat.com> changed the Status of [bug 987461|https://bugzilla.redhat.com/show_bug.cgi?id=987461] from NEW to CLOSED
> statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus fails randomly
> -------------------------------------------------------------------------------------
>
> Key: ISPN-4845
> URL: https://issues.jboss.org/browse/ISPN-4845
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 5.2.10.Final
> Reporter: Michal Vinkler
> Labels: 5.2.x
>
> Seen with EAP 6.3.0.ER10, Infinispan 5.2.10
> Test org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus randomly fails (seen on Solaris and HP-UX).
> Might be the same as ISPN-4743.
> Stacktraces:
> HP-UX version
> Error Message
> {code}
> Timed out waiting for rebalancing to complete on node ClusterTopologyManagerTest-NodeB-47391, expected member list is [ClusterTopologyManagerTest-NodeB-47391], current member list is [ClusterTopologyManagerTest-NodeB-47391, ClusterTopologyManagerTest-NodeC-55740]!
> {code}
> Stacktrace
> {code}
> java.lang.RuntimeException: Timed out waiting for rebalancing to complete on node ClusterTopologyManagerTest-NodeB-47391, expected member list is [ClusterTopologyManagerTest-NodeB-47391], current member list is [ClusterTopologyManagerTest-NodeB-47391, ClusterTopologyManagerTest-NodeC-55740]!
> at org.infinispan.test.TestingUtil.waitForRehashToComplete(TestingUtil.java:203)
> at org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus(ClusterTopologyManagerTest.java:353)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> Also see standard output:
> https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Infi...
> Solaris version
> Error Message
> {code}
> Thread already timed out waiting for event 3 left
> {code}
> Stacktrace
> {code}
> java.lang.IllegalStateException: Thread already timed out waiting for event 3 left
> at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:150)
> at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:135)
> at org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus(ClusterTopologyManagerTest.java:350)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> Also see standard output:
> https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Infi...
> Might be the same as ISPN-4743.
> Downstream BZ was: https://bugzilla.redhat.com/show_bug.cgi?id=987461
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 5 months
[JBoss JIRA] (ISPN-4845) statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus fails randomly
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-4845?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration updated ISPN-4845:
------------------------------------------
Bugzilla Update: Perform
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=987461
> statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus fails randomly
> -------------------------------------------------------------------------------------
>
> Key: ISPN-4845
> URL: https://issues.jboss.org/browse/ISPN-4845
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 5.2.10.Final
> Reporter: Michal Vinkler
> Labels: 5.2.x
>
> Seen with EAP 6.3.0.ER10, Infinispan 5.2.10
> Test org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus randomly fails (seen on Solaris and HP-UX).
> Might be the same as ISPN-4743.
> Stacktraces:
> HP-UX version
> Error Message
> {code}
> Timed out waiting for rebalancing to complete on node ClusterTopologyManagerTest-NodeB-47391, expected member list is [ClusterTopologyManagerTest-NodeB-47391], current member list is [ClusterTopologyManagerTest-NodeB-47391, ClusterTopologyManagerTest-NodeC-55740]!
> {code}
> Stacktrace
> {code}
> java.lang.RuntimeException: Timed out waiting for rebalancing to complete on node ClusterTopologyManagerTest-NodeB-47391, expected member list is [ClusterTopologyManagerTest-NodeB-47391], current member list is [ClusterTopologyManagerTest-NodeB-47391, ClusterTopologyManagerTest-NodeC-55740]!
> at org.infinispan.test.TestingUtil.waitForRehashToComplete(TestingUtil.java:203)
> at org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus(ClusterTopologyManagerTest.java:353)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> Also see standard output:
> https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Infi...
> Solaris version
> Error Message
> {code}
> Thread already timed out waiting for event 3 left
> {code}
> Stacktrace
> {code}
> java.lang.IllegalStateException: Thread already timed out waiting for event 3 left
> at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:150)
> at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:135)
> at org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus(ClusterTopologyManagerTest.java:350)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> Also see standard output:
> https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Infi...
> Might be the same as ISPN-4743.
> Downstream BZ was: https://bugzilla.redhat.com/show_bug.cgi?id=987461
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 5 months
[JBoss JIRA] (ISPN-4845) statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus fails randomly
by Michal Vinkler (JIRA)
[ https://issues.jboss.org/browse/ISPN-4845?page=com.atlassian.jira.plugin.... ]
Michal Vinkler updated ISPN-4845:
---------------------------------
Description:
Seen with EAP 6.3.0.ER10, Infinispan 5.2.10
Test org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus randomly fails (seen on Solaris and HP-UX).
Might be the same as ISPN-4743.
Stacktraces:
HP-UX version
Error Message
{code}
Timed out waiting for rebalancing to complete on node ClusterTopologyManagerTest-NodeB-47391, expected member list is [ClusterTopologyManagerTest-NodeB-47391], current member list is [ClusterTopologyManagerTest-NodeB-47391, ClusterTopologyManagerTest-NodeC-55740]!
{code}
Stacktrace
{code}
java.lang.RuntimeException: Timed out waiting for rebalancing to complete on node ClusterTopologyManagerTest-NodeB-47391, expected member list is [ClusterTopologyManagerTest-NodeB-47391], current member list is [ClusterTopologyManagerTest-NodeB-47391, ClusterTopologyManagerTest-NodeC-55740]!
at org.infinispan.test.TestingUtil.waitForRehashToComplete(TestingUtil.java:203)
at org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus(ClusterTopologyManagerTest.java:353)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
at org.testng.TestRunner.privateRun(TestRunner.java:767)
at org.testng.TestRunner.run(TestRunner.java:617)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
{code}
Also see standard output:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Infi...
Solaris version
Error Message
{code}
Thread already timed out waiting for event 3 left
{code}
Stacktrace
{code}
java.lang.IllegalStateException: Thread already timed out waiting for event 3 left
at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:150)
at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:135)
at org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus(ClusterTopologyManagerTest.java:350)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
at org.testng.TestRunner.privateRun(TestRunner.java:767)
at org.testng.TestRunner.run(TestRunner.java:617)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
{code}
Also see standard output:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Infi...
Might be the same as ISPN-4743.
Downstream BZ was: https://bugzilla.redhat.com/show_bug.cgi?id=987461
was:
Seen with EAP 6.3.0.ER10, Infinispan 5.2.10
Test org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus randomly fails (seen on Solaris and HP-UX).
Might be the same as ISPN-4743.
Stacktraces:
HP-UX version
Error Message
{code}
Timed out waiting for rebalancing to complete on node ClusterTopologyManagerTest-NodeB-47391, expected member list is [ClusterTopologyManagerTest-NodeB-47391], current member list is [ClusterTopologyManagerTest-NodeB-47391, ClusterTopologyManagerTest-NodeC-55740]!
{code}
Stacktrace
{code}
java.lang.RuntimeException: Timed out waiting for rebalancing to complete on node ClusterTopologyManagerTest-NodeB-47391, expected member list is [ClusterTopologyManagerTest-NodeB-47391], current member list is [ClusterTopologyManagerTest-NodeB-47391, ClusterTopologyManagerTest-NodeC-55740]!
at org.infinispan.test.TestingUtil.waitForRehashToComplete(TestingUtil.java:203)
at org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus(ClusterTopologyManagerTest.java:353)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
at org.testng.TestRunner.privateRun(TestRunner.java:767)
at org.testng.TestRunner.run(TestRunner.java:617)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
{code}
Also see standard output:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Infi...
Solaris version
Error Message
{code}
Thread already timed out waiting for event 3 left
{code}
Stacktrace
{code}
java.lang.IllegalStateException: Thread already timed out waiting for event 3 left
at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:150)
at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:135)
at org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus(ClusterTopologyManagerTest.java:350)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
at org.testng.TestRunner.privateRun(TestRunner.java:767)
at org.testng.TestRunner.run(TestRunner.java:617)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
{code}
Also see standard output:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Infi...
> statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus fails randomly
> -------------------------------------------------------------------------------------
>
> Key: ISPN-4845
> URL: https://issues.jboss.org/browse/ISPN-4845
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 5.2.10.Final
> Reporter: Michal Vinkler
> Labels: 5.2.x
>
> Seen with EAP 6.3.0.ER10, Infinispan 5.2.10
> Test org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus randomly fails (seen on Solaris and HP-UX).
> Might be the same as ISPN-4743.
> Stacktraces:
> HP-UX version
> Error Message
> {code}
> Timed out waiting for rebalancing to complete on node ClusterTopologyManagerTest-NodeB-47391, expected member list is [ClusterTopologyManagerTest-NodeB-47391], current member list is [ClusterTopologyManagerTest-NodeB-47391, ClusterTopologyManagerTest-NodeC-55740]!
> {code}
> Stacktrace
> {code}
> java.lang.RuntimeException: Timed out waiting for rebalancing to complete on node ClusterTopologyManagerTest-NodeB-47391, expected member list is [ClusterTopologyManagerTest-NodeB-47391], current member list is [ClusterTopologyManagerTest-NodeB-47391, ClusterTopologyManagerTest-NodeC-55740]!
> at org.infinispan.test.TestingUtil.waitForRehashToComplete(TestingUtil.java:203)
> at org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus(ClusterTopologyManagerTest.java:353)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> Also see standard output:
> https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Infi...
> Solaris version
> Error Message
> {code}
> Thread already timed out waiting for event 3 left
> {code}
> Stacktrace
> {code}
> java.lang.IllegalStateException: Thread already timed out waiting for event 3 left
> at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:150)
> at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:135)
> at org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus(ClusterTopologyManagerTest.java:350)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> Also see standard output:
> https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Infi...
> Might be the same as ISPN-4743.
> Downstream BZ was: https://bugzilla.redhat.com/show_bug.cgi?id=987461
--
This message was sent by Atlassian JIRA
(v6.3.1#6329)
11 years, 5 months