[JBoss JIRA] (ISPN-7172) Total order caches can hang during join
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-7172?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-7172:
-------------------------------
Status: Open (was: New)
> Total order caches can hang during join
> ---------------------------------------
>
> Key: ISPN-7172
> URL: https://issues.jboss.org/browse/ISPN-7172
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Core
> Affects Versions: 9.0.0.Alpha4
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: testsuite_stability
> Fix For: 9.0.0.Beta1
>
>
> When two nodes join a cache in quick sequence, it's possible for a {{CacheTopologyControlCommand}} to arrive between {{LocalTopologyManagerImpl}} adding the {{LocalCacheStatus}} to the {{runningCaches}} map, but before it calling {{cacheStatus.getTopologyUpdatesExecutor().executeAsync(() -> joinFuture)}} to block topology updates.
> This exposes a bug in the way {{LimitedExecutor}} wraps a {{WithinThreadExecutor}}, which causes {{SyncPrepareUseSynchronizationTotalOrderTest}} (and possibly other total order tests) to hang:
> {noformat}
> "testng-SyncPrepareUseSynchronizationTotalOrderTest" #15 prio=5 os_prio=0 tid=0x00007fc674b74000 nid=0x2f3f waiting on condition [0x00007fc626682000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000829d03a0> (a java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1687)
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3320)
> at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1767)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1907)
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.executors.LimitedExecutor.execute(LimitedExecutor.java:70)
> at org.infinispan.executors.LimitedExecutor.executeAsync(LimitedExecutor.java:96)
> at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:141)
> at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:121)
> at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:867)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:633)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:622)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:547)
> - locked <0x00000000829d0520> (a org.infinispan.factories.ComponentRegistry)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:231)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:808)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:639)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:590)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:422)
> at org.infinispan.test.MultipleCacheManagersTest.getCaches(MultipleCacheManagersTest.java:248)
> at org.infinispan.test.MultipleCacheManagersTest.waitForClusterToForm(MultipleCacheManagersTest.java:257)
> at org.infinispan.test.MultipleCacheManagersTest.waitForClusterToForm(MultipleCacheManagersTest.java:266)
> at org.infinispan.tx.totalorder.simple.BaseSimpleTotalOrderTest.createCacheManagers(BaseSimpleTotalOrderTest.java:326)
> at org.infinispan.test.MultipleCacheManagersTest.callCreateCacheManagers(MultipleCacheManagersTest.java:109)
> at org.infinispan.test.MultipleCacheManagersTest.createBeforeMethod(MultipleCacheManagersTest.java:119)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 1 month
[JBoss JIRA] (ISPN-7064) RPC to leaver times out instead of finishing immediately
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-7064?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-7064:
----------------------------------
Fix Version/s: 8.2.5.Final
> RPC to leaver times out instead of finishing immediately
> --------------------------------------------------------
>
> Key: ISPN-7064
> URL: https://issues.jboss.org/browse/ISPN-7064
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Core
> Affects Versions: 9.0.0.Alpha4, 8.2.4.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: testsuite_stability
> Fix For: 9.0.0.Beta1, 9.0.0.Final, 8.2.5.Final
>
> Attachments: MultipleNodesLeavingTest_ISPN-5469_async_remote_perform_20160913.log
>
>
> This causes random failures in {{ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus2}}. Node D leaves and nodes E and F start a rebalance. Then node F also leaves, but {{DISCARD}} is enabled first, so the {{CacheTopologyControlCommand(LEAVE)}} synchronous RPC blocks until node F installs a view by itself.
> {noformat}
> 14:05:17,187 INFO (Incoming-1,Test-NodeF-12732:[]) [JGroupsTransport] ISPN000094: Received new cluster view for channel ISPN: [Test-NodeE-18953|3] (2) [Test-NodeE-18953, Test-NodeF-12732]
> 14:05:17,261 TRACE (testng-Test:[]) [JGroupsTransport] dests=[Test-NodeE-18953], command=CacheTopologyControlCommand{cache=testCache, type=LEAVE, sender=Test-NodeF-12732, joinInfo=null, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=3}, mode=SYNCHRONOUS, timeout=240000
> 14:05:22,546 DEBUG (Thread-88606:[]) [GMS] Test-NodeF-12732: installing view [Test-NodeF-12732|4] (1) [Test-NodeF-12732]
> 14:05:22,589 DEBUG (testng-Test:[]) [LocalTopologyManagerImpl] Error sending the leave request for cache testCache to coordinator
> {noformat}
> Node F then cancels its inbound state transfers by sending a synchronous {{StateRequestCommand(CANCEL_STATE_TRANSFER)}} RPC to node F. This RPC sometimes hits JGRP-2103 and it only times out after 5 minutes, causing the test to fail.
> {noformat}
> 14:05:22,608 TRACE (testng-Test:[]) [JGroupsTransport] dests=[Test-NodeE-18953], command=StateRequestCommand{cache=testCache, origin=Test-NodeF-12732, type=CANCEL_STATE_TRANSFER, topologyId=7, segments=[130, 3, ...]}, mode=SYNCHRONOUS_IGNORE_LEAVERS, timeout=240000
> 14:05:22,608 TRACE (testng-Test:[]) [RequestCorrelator] Test-NodeF-12732: invoking multicast RPC [req-id=134161]
> 14:05:22,609 DEBUG (Thread-88606:[]) [JGroupsTransport] New view accepted: [Test-NodeF-12732|4] (1) [Test-NodeF-12732]
> 14:09:22,608 TRACE (timeout-thread-Test-NodeF-p48767-t1:[]) [JGroupsTransport] Responses: Responses{
> Test-NodeE-18953: sender=Test-NodeE-18953, received=false, suspected=true}
> 14:09:22,608 TRACE (timeout-thread-Test-NodeF-p48767-t1:[]) [RpcManagerImpl] Replication exception
> org.infinispan.util.concurrent.TimeoutException: Replication timeout
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$2(JGroupsTransport.java:659) ~[classes/:?]
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) ~[?:1.8.0_101]
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[?:1.8.0_101]
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) [?:1.8.0_101]
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) [?:1.8.0_101]
> at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47) [classes/:?]
> at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:15) [classes/:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_101]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_101]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_101]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
> Suppressed: org.infinispan.remoting.RpcException: Test-NodeE-18953 was suspected
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.addSuppressedExceptions(JGroupsTransport.java:707) ~[classes/:?]
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$2(JGroupsTransport.java:659) ~[classes/:?]
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) ~[?:1.8.0_101]
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[?:1.8.0_101]
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) [?:1.8.0_101]
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) [?:1.8.0_101]
> at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47) [classes/:?]
> at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:15) [classes/:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_101]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_101]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_101]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
> 14:09:22,609 DEBUG (testng-Test:[]) [InboundTransferTask] Caught an exception while cancelling state transfer for segments [130, 3, ...] from Test-NodeE-18953
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 1 month
[JBoss JIRA] (ISPN-7088) Interceptors should not access TransactionManager
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-7088?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-7088:
----------------------------------
Fix Version/s: 8.2.5.Final
> Interceptors should not access TransactionManager
> -------------------------------------------------
>
> Key: ISPN-7088
> URL: https://issues.jboss.org/browse/ISPN-7088
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: testsuite_failure
> Fix For: 9.0.0.Beta1, 9.0.0.Final, 8.2.5.Final
>
>
> With ISPN-5469, we are no longer executing all the interceptor on the user thread, even on the originator. That means trying to read the thread-local transaction with {{TransactionManager.getTransaction()}} sometimes won't work, so interceptors should avoid it.
> E.g. {{InvocationContextInterceptor.markTxForRollbackAndRethrow()}} wants to rollback the current transaction but doesn't find any transaction bound to the current thread, causing failures in {{PessimisticTxPartitionAndMergeDuringRuntimeTest}}.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 1 month
[JBoss JIRA] (ISPN-7067) Cache.evict() sometimes performs a DataContainer.remove()
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-7067?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-7067:
----------------------------------
Fix Version/s: 8.2.5.Final
> Cache.evict() sometimes performs a DataContainer.remove()
> ---------------------------------------------------------
>
> Key: ISPN-7067
> URL: https://issues.jboss.org/browse/ISPN-7067
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.0.0.Alpha4, 8.2.4.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 9.0.0.Beta1, 9.0.0.Final, 8.2.5.Final
>
>
> {{Cache.evict()}} generally uses {{DataContainer.evict()}} to move an entry from the data container to the store.
> However, when {{EntryWrappingInterceptor}} doesn't find the entry in the data container, {{EvictCommand.perform()}} doesn't set the {{EVICTED}} flag on the context entry, and then {{ReadCommittedEntry.commit()}} calls {{DataContainer.remove()}} instead of {{DataContainer.evict()}}.
> If another command activated the entry between the entry wrapping and the commit, this will remove the entry altogether instead of moving it to the store.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 1 month
[JBoss JIRA] (ISPN-6792) NonTxDistributionInterceptor doesn't always change the matcher for retry
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-6792?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-6792:
----------------------------------
Fix Version/s: 8.2.5.Final
> NonTxDistributionInterceptor doesn't always change the matcher for retry
> ------------------------------------------------------------------------
>
> Key: ISPN-6792
> URL: https://issues.jboss.org/browse/ISPN-6792
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.0.0.Alpha2, 8.2.2.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Labels: testsuite_stability
> Fix For: 9.0.0.Alpha3, 8.2.5.Final
>
>
> When invoking a non-transactional write command remotely, {{BaseDistributionInterceptor}} expects to receive an {{OutdatedTopologyException}} wrapped in a {{RemoteException}} if the remote node saw a newer cache topology.
> However, {{OutdatedTopologyException}} is handled differently by {{JGroupsTransport}}, and it is *not* wrapped in a {{RemoteException}}. Because of this, {{BaseDistributionInterceptor}} doesn't update the value matcher, and the retried command may fail when it sees its own value.
> This makes {{NonTxPutIfAbsentDuringLeaveStressTest}} fail randomly with
> {noformat}
> java.lang.AssertionError: expected:<null> but was:<value_7_0>
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
> at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:88)
> at org.infinispan.distribution.rehash.NonTxPutIfAbsentDuringLeaveStressTest.testNodeLeavingDuringPutIfAbsent(NonTxPutIfAbsentDuringLeaveStressTest.java:101)
> {noformat}
> Note that this is different from ISPN-6451: there, the assertion message is {{AssertionError: expected:<value_48_1> but was:<null>}}, and it is most likely caused by ISPN-3918.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 1 month
[JBoss JIRA] (ISPN-7194) Administration console - task history tab is not working
by Roman Macor (JIRA)
Roman Macor created ISPN-7194:
---------------------------------
Summary: Administration console - task history tab is not working
Key: ISPN-7194
URL: https://issues.jboss.org/browse/ISPN-7194
Project: Infinispan
Issue Type: Bug
Components: JMX, reporting and management
Affects Versions: 9.0.0.Alpha4
Reporter: Roman Macor
Click on cache container -> task execution -> TASK_HISTORY
Result: nothing happens, the tab is not active
Expected results: task history should be showed
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 1 month
[JBoss JIRA] (ISPN-7190) Administration console - when adding new cache, base configuration shouldn't show existing caches
by Vladimir Blagojevic (JIRA)
[ https://issues.jboss.org/browse/ISPN-7190?page=com.atlassian.jira.plugin.... ]
Vladimir Blagojevic commented on ISPN-7190:
-------------------------------------------
!Screen Shot 2016-11-13 at 8.05.40 AM.png|thumbnail!
> Administration console - when adding new cache, base configuration shouldn't show existing caches
> -------------------------------------------------------------------------------------------------
>
> Key: ISPN-7190
> URL: https://issues.jboss.org/browse/ISPN-7190
> Project: Infinispan
> Issue Type: Bug
> Components: JMX, reporting and management
> Affects Versions: 9.0.0.Alpha4
> Reporter: Roman Macor
> Assignee: Vladimir Blagojevic
> Attachments: Screenshot from 2016-11-11.png
>
>
> Click on cache container -> Add cache -> expand "base configuration" drop down
> Result: The drop down offers cache configurations and existing caches
> Expected result: The drop down should only offer cache configurations. ("default" and "memcachedCache" shouldn't be there )
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 1 month
[JBoss JIRA] (ISPN-7190) Administration console - when adding new cache, base configuration shouldn't show existing caches
by Vladimir Blagojevic (JIRA)
[ https://issues.jboss.org/browse/ISPN-7190?page=com.atlassian.jira.plugin.... ]
Vladimir Blagojevic updated ISPN-7190:
--------------------------------------
Attachment: Screen Shot 2016-11-13 at 8.05.40 AM.png
> Administration console - when adding new cache, base configuration shouldn't show existing caches
> -------------------------------------------------------------------------------------------------
>
> Key: ISPN-7190
> URL: https://issues.jboss.org/browse/ISPN-7190
> Project: Infinispan
> Issue Type: Bug
> Components: JMX, reporting and management
> Affects Versions: 9.0.0.Alpha4
> Reporter: Roman Macor
> Assignee: Vladimir Blagojevic
> Attachments: Screen Shot 2016-11-13 at 8.05.40 AM.png, Screenshot from 2016-11-11.png
>
>
> Click on cache container -> Add cache -> expand "base configuration" drop down
> Result: The drop down offers cache configurations and existing caches
> Expected result: The drop down should only offer cache configurations. ("default" and "memcachedCache" shouldn't be there )
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 1 month