[Red Hat JIRA] (ISPN-11598) EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead random failures
by Dan Berindei (Jira)
[ https://issues.redhat.com/browse/ISPN-11598?page=com.atlassian.jira.plugi... ]
Dan Berindei reopened ISPN-11598:
---------------------------------
The test is still failing randomly in master:
{noformat}
15:56:45,758 ERROR (testng-Test:[]) [CheckPoint] Timed out waiting for event after_invocation * 1 (available = 0, total = 1)
15:56:45,778 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.eviction.impl.EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead
java.util.concurrent.TimeoutException: Timed out waiting for event after_invocation
at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:50) ~[test-classes/:?]
at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:40) ~[test-classes/:?]
at org.infinispan.eviction.impl.EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead(EvictionWithPassivationAndConcurrentOperationsTest.java:107) ~[test-classes/:?]
{noformat}
> EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead random failures
> ------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-11598
> URL: https://issues.redhat.com/browse/ISPN-11598
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite
> Affects Versions: 11.0.0.Dev03
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 11.0.0.Dev04
>
>
> According to the code comments, the test is supposed to block the {{put(evicted-key)}} operation before releasing the orderer, but because of {{operationCheckPoint.trigger(Mocks.BEFORE_RELEASE)}} it only blocks *after* releasing the orderer. This means the eviction triggered by {{put(other-key)}} and the activation triggered later by {{get(evicted-key)}} are not blocked.
> All three operations eventually call {{operationCheckPoint.awaitStrict(Mocks.BEFORE_RELEASE)}}. The test triggers that event twice, so it unblocks {{put(evicted-key)}} *and* one of {{put(other-key)}}/{{get(evicted-key)}}. The two unblocked operations then compete for a single {{Mocks.AFTER_RELEASE}} event.
> If {{put(evicted-key)}} receives the {{Mocks.AFTER_RELEASE}} event, the test passes, because the other operations can finish without waiting for the activation/passivation to finish. (It just needs to be queued in )
> Because {{CheckPoint}} doesn't guarantee waiters receive events in the queueing order, either {{put(other-key, "value"}}, or {{get(evicted-key)}} could receive the {{AFTER_RELEASE}} event from {{put(evicted-key, "value")}}, making it time out.
> {noformat}
> 19:53:59,795 TRACE (ForkThread-1,Test:[]) [InvocationContextInterceptor] Invoked with command PutKeyValueCommand{key=evicted-key, value=value, flags=[SKIP_CACHE_LOAD], commandInvocationId=CommandInvocation:<local>:37070, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{version=null, lifespan=-1, maxIdle=-1}, successful=true, topologyId=-1} and InvocationContext [SingleKeyNonTxInvocationContext{isLocked=false, key=null, cacheEntry=null, origin=null, lockOwner=CommandInvocation:<local>:37070}]
> 19:53:59,807 TRACE (non-blocking-thread-Test-NodeK-p23146-t1:[]) [DataOperationOrderer] Ordered future java.util.concurrent.CompletableFuture@76d1e6bc[Not completed] is completed for key evicted-key from op WRITE
> 19:53:59,811 TRACE (non-blocking-thread-Test-NodeK-p23146-t1:[]) [CheckPoint] Waiting for event post_released * 1
> {noformat}
> ^ put(evicted-key) is blocked
> {noformat}
> 19:53:59,807 TRACE (testng-Test:[]) [InvocationContextInterceptor] Invoked with command PutKeyValueCommand{key=other-key, value=other-value, flags=[], commandInvocationId=CommandInvocation:<local>:37073, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{version=null, lifespan=-1, maxIdle=-1}, successful=true, topologyId=-1} and InvocationContext [SingleKeyNonTxInvocationContext{isLocked=false, key=null, cacheEntry=null, origin=null, lockOwner=CommandInvocation:<local>:37073}]
> 19:53:59,816 TRACE (non-blocking-thread-Test-NodeK-p23146-t2:[]) [AbstractInternalDataContainer] Store other-key=ImmortalCacheEntry{key=other-key, value=other-value, internalMetadata=null} in container
> 19:53:59,817 TRACE (non-blocking-thread-Test-NodeK-p23146-t2:[]) [DataOperationOrderer] Ordering upcoming future java.util.concurrent.CompletableFuture@32541e18[Not completed] for key evicted-key to run after null
> {noformat}
> ^ put(other-key) is not blocked here
> {noformat}
> 19:53:59,818 TRACE (non-blocking-thread-Test-NodeK-p23146-t2:[]) [PassivationManagerImpl] Passivating entry evicted-key
> 19:53:59,818 TRACE (non-blocking-thread-Test-NodeK-p23146-t2:[]) [PersistenceManagerImpl] Writing to all stores for id 67588
> 19:53:59,841 TRACE (non-blocking-thread-Test-NodeK-p23146-t3:[]) [CompletionStages] Continuing execution of id 67588
> 19:53:59,844 TRACE (non-blocking-thread-Test-NodeK-p23146-t3:[]) [CheckPoint] Waiting for event pre_released * 1
> {noformat}
> ^ put(other-key) blocks in the checkpoint
> {noformat}
> 19:53:59,956 TRACE (testng-Test:[]) [CheckPoint] Triggering event pre_released * 1 (available = 1, total = 2)
> 19:53:59,956 TRACE (testng-Test:[]) [CheckPoint] Triggering event post_released * 1 (available = 1, total = 1)
> 19:53:59,956 TRACE (non-blocking-thread-Test-NodeK-p23146-t3:[]) [CheckPoint] Received event pre_released * 1 (available = 0, total = 2)
> {noformat}
> ^ put(other-key) is unblocked, put(evicted-key) stays blocked
> {noformat}
> 19:54:09,998 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.eviction.impl.EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead
> java.util.concurrent.TimeoutException: null
> at java.util.concurrent.FutureTask.get(FutureTask.java:204) ~[?:?]
> at org.infinispan.eviction.impl.EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead(EvictionWithPassivationAndConcurrentOperationsTest.java:99) ~[test-classes/:?]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
4 years, 10 months
[Red Hat JIRA] (ISPN-11565) Handle Unauthorized actions in the console
by Katia Aresti (Jira)
[ https://issues.redhat.com/browse/ISPN-11565?page=com.atlassian.jira.plugi... ]
Katia Aresti resolved ISPN-11565.
---------------------------------
Fix Version/s: 12.0.0.Final
(was: 12.1.0.Final)
Resolution: Done
> Handle Unauthorized actions in the console
> ------------------------------------------
>
> Key: ISPN-11565
> URL: https://issues.redhat.com/browse/ISPN-11565
> Project: Infinispan
> Issue Type: Feature Request
> Components: Console
> Affects Versions: 10.1.5.Final
> Reporter: Katia Aresti
> Assignee: Katia Aresti
> Priority: Major
> Labels: console
> Fix For: 12.0.0.Final
>
>
> When the user is not authorized to perfom some actions, an error should be displayed.
> The first quick version should just display the errors in the console. Authz based console should be implemented
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
4 years, 10 months
[Red Hat JIRA] (ISPN-12679) AsyncBackupExpirationTest.testExpiredAccess random failures
by Dan Berindei (Jira)
Dan Berindei created ISPN-12679:
-----------------------------------
Summary: AsyncBackupExpirationTest.testExpiredAccess random failures
Key: ISPN-12679
URL: https://issues.redhat.com/browse/ISPN-12679
Project: Infinispan
Issue Type: Bug
Components: Core, Cross-Site Replication, Expiration, Test Suite
Affects Versions: 12.0.0.Final
Reporter: Dan Berindei
Fix For: 12.1.0.Final
Sometimes the backup {{IracPutKeyValueCommand}} is delayed on NodeD and it stores the value only after the test advanced the time service and invokes the get operation that's supposed to expire the entry, causing the entry expiration to fail in the {{NYC}} site.
First {{IracPutKeyValueCommand}} arrives at NodeD and locks the key:
{noformat}
15:59:07,907 TRACE (jgroups-7,Test-NodeB:[]) [InvocationContextInterceptor] Invoked with command PrepareCommand {modifications=[PutKeyValueCommand{key=MagicKey{7FD/E1260FE1/147@Test-NodeB}, value=v, flags=[], commandInvocationId=CommandInvocation:local:0, ...
15:59:07,910 TRACE (jgroups-5,bridge-org.infinispan.xsite.Test,_Test-NodeC:NYC-2:[]) [InvocationContextInterceptor] Invoked with command IracPutKeyValueCommand{key=MagicKey{7FD/E1260FE1/147@Test-NodeB}, value=v, metadata=EmbeddedLifespanExpirableMetadata{lifespan=1000, version=null}, privateMetadata=PrivateMetadata{iracMetadata=IracMetadata{site='LON-1', version=(LON-1=(6:1))}, entryVersion=null}, successful=true, commandInvocationId=CommandInvocation:Test-NodeC:45679} and InvocationContext [SingleKeyNonTxInvocationContext{isLocked=false, key=null, cacheEntry=null, origin=null, lockOwner=CommandInvocation:Test-NodeC:45679}]
15:59:07,912 TRACE (jgroups-8,Test-NodeD:[]) [InvocationContextInterceptor] Invoked with command IracPutKeyValueCommand{key=MagicKey{7FD/E1260FE1/147@Test-NodeB}, value=v, metadata=EmbeddedLifespanExpirableMetadata{lifespan=1000, version=null}, privateMetadata=PrivateMetadata{iracMetadata=IracMetadata{site='LON-1', version=(LON-1=(6:1))}, entryVersion=null}, successful=true, commandInvocationId=CommandInvocation:Test-NodeC:45679} and InvocationContext [SingleKeyNonTxInvocationContext{isLocked=false, key=null, cacheEntry=null, origin=Test-NodeC, lockOwner=CommandInvocation:Test-NodeC:45679}]
15:59:07,912 TRACE (jgroups-8,Test-NodeD:[]) [DefaultLockManager] Lock key=MagicKey{7FD/E1260FE1/147@Test-NodeB} for owner=CommandInvocation:Test-NodeC:45679. timeout=10000 (MILLISECONDS)
15:59:07,912 TRACE (jgroups-8,Test-NodeD:[]) [InfinispanLock] LockPlaceHolder{lockState=ACQUIRED, owner=CommandInvocation:Test-NodeC:45679} successfully acquired the lock.
{noformat}
Then {{RemoteExpiredCommand}} arrives at NodeD, tries to lock the key, and fails:
{noformat}
15:59:07,917 TRACE (testng-Test:[]) [InvocationContextInterceptor] Invoked with command RemoveExpiredCommand{key=MagicKey{7FD/E1260FE1/147@Test-NodeB}, value=v, lifespan=1000, maxIde=false, internalMetadata=null} and InvocationContext [SingleKeyNonTxInvocationContext{isLocked=false, key=null, cacheEntry=null, origin=null, lockOwner=CommandInvocation:local:0}]
15:59:07,918 TRACE (jgroups-7,Test-NodeD:[]) [InvocationContextInterceptor] Invoked with command RemoveExpiredCommand{key=MagicKey{7FD/E1260FE1/147@Test-NodeB}, value=v, lifespan=1000, maxIde=false, internalMetadata=null} and InvocationContext [SingleKeyNonTxInvocationContext{isLocked=false, key=null, cacheEntry=null, origin=Test-NodeC, lockOwner=CommandInvocation:local:0}]
15:59:07,918 TRACE (jgroups-7,Test-NodeD:[]) [DefaultLockManager] Lock key=MagicKey{7FD/E1260FE1/147@Test-NodeB} for owner=CommandInvocation:local:0. timeout=0 (MILLISECONDS)
15:59:07,918 TRACE (jgroups-7,Test-NodeD:[]) [InfinispanLock] Unable to acquire. Lock is held.
{noformat}
Next {{IracPutKeyValueCommand}} stores the value and releases the lock:
{noformat}
15:59:07,919 TRACE (jgroups-8,Test-NodeD:[]) [AbstractInternalDataContainer] Store MagicKey{7FD/E1260FE1/147@Test-NodeB}=MortalCacheEntry{key=MagicKey{7FD/E1260FE1/147@Test-NodeB}, value=v, internalMetadata=PrivateMetadata{iracMetadata=IracMetadata{site='LON-1', version=(LON-1=(6:1))}, entryVersion=null}, created=1002000, lifespan=1000} in container
15:59:07,919 TRACE (jgroups-8,Test-NodeD:[]) [DefaultLockManager] Release locks for keys=[MagicKey{7FD/E1260FE1/147@Test-NodeB}]. owner=CommandInvocation:Test-NodeC:45679
{noformat}
But it's too late for the {{RemoveExpiredCommand}}, which gives up:
{noformat}
15:59:07,920 ERROR (jgroups-7,Test-NodeD:[]) [InvocationContextInterceptor] ISPN000136: Error executing command RemoveExpiredCommand on Cache 'defaultcache', writing keys [MagicKey{7FD/E1260FE1/147@Test-NodeB}]
org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 0 milliseconds for key MagicKey{7FD/E1260FE1/147@Test-NodeB} and requestor CommandInvocation:local:0. Lock is held by null
at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.get(DefaultLockManager.java:291) ~[classes/:?]
at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.get(DefaultLockManager.java:221) ~[classes/:?]
at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.checkState(InfinispanLock.java:439) ~[classes/:?]
at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.toInvocationStage(InfinispanLock.java:411) ~[classes/:?]
at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.toInvocationStage(DefaultLockManager.java:251) ~[classes/:?]
at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.toInvocationStage(DefaultLockManager.java:275) ~[classes/:?]
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAndRecord(AbstractLockingInterceptor.java:287) ~[classes/:?]
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:137) ~[classes/:?]
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitRemoveExpiredCommand(AbstractLockingInterceptor.java:107) ~[classes/:?]
at org.infinispan.commands.write.RemoveExpiredCommand.acceptVisitor(RemoveExpiredCommand.java:46) ~[classes/:?]
at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) ~[classes/:?]
at org.infinispan.interceptors.impl.TxInterceptor.visitRemoveExpiredCommand(TxInterceptor.java:238) ~[classes/:?]
15:59:07,926 TRACE (jgroups-7,Test-NodeD:[]) [CallInterceptor] Cannot remove entry due to it not being expired - this can be caused by different clocks on nodes or a concurrent write
15:59:07,926 TRACE (jgroups-7,Test-NodeD:[]) [BaseDistributionInterceptor] Skipping the replication of the conditional command as it did not succeed on primary owner (RemoveExpiredCommand{key=MagicKey{7FD/E1260FE1/147@Test-NodeB}, value=v, lifespan=1000, maxIde=false, internalMetadata=null}).
{noformat}
And because the entry was not expired, the test fails:
{noformat}
15:59:07,929 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.xsite.AsyncBackupExpirationTest.testExpiredAccess[LON=PESSIMISTIC_TX, NYC=OPTIMISTIC_TX_RC]([true, false])
java.lang.AssertionError:
at org.testng.AssertJUnit.fail(AssertJUnit.java:59) ~[testng-6.14.3.jar:?]
at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24) ~[testng-6.14.3.jar:?]
at org.testng.AssertJUnit.assertNull(AssertJUnit.java:282) ~[testng-6.14.3.jar:?]
at org.testng.AssertJUnit.assertNull(AssertJUnit.java:274) ~[testng-6.14.3.jar:?]
at org.infinispan.xsite.AsyncBackupExpirationTest.testExpiredAccess(AsyncBackupExpirationTest.java:182) ~[test-classes/:?]
{noformat}
I believe this can happen in production code just as in the test, if the entry lifespan was shorter.
So we shouldn't just add a delay in the test, we should make the {{RemoveExpiredCommand}} wait for the {{IracPutKeyValueCommand}} to finish writing the value and release the lock.
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
4 years, 10 months
[Red Hat JIRA] (ISPN-12657) Cross-Site automatic state transfer
by Pedro Ruivo (Jira)
[ https://issues.redhat.com/browse/ISPN-12657?page=com.atlassian.jira.plugi... ]
Pedro Ruivo updated ISPN-12657:
-------------------------------
Sprint: DataGrid Sprint #56
> Cross-Site automatic state transfer
> -----------------------------------
>
> Key: ISPN-12657
> URL: https://issues.redhat.com/browse/ISPN-12657
> Project: Infinispan
> Issue Type: Feature Request
> Components: Cross-Site Replication
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
> Priority: Major
> Fix For: 12.1.0.Final
>
>
> With conflict resolution enable for IRAC state transfer, is now possible to achieve automatic state transfer for it.
> It needs a new toggle in the configuration & a runtime toggle well. Users may want to disable at runtime while performing some maintenance (for example).
> Events from JGroups (site up event) will be the trigger.
> Note: it only affects asynchronous strategy.
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
4 years, 10 months
[Red Hat JIRA] (ISPN-12678) infinispan 6.0.2 升级到9.4.21 后无法query
by Yuemei Qiu (Jira)
[ https://issues.redhat.com/browse/ISPN-12678?page=com.atlassian.jira.plugi... ]
Yuemei Qiu updated ISPN-12678:
------------------------------
Affects Version/s: 9.4.21.Final
> infinispan 6.0.2 升级到9.4.21 后无法query
> -----------------------------------
>
> Key: ISPN-12678
> URL: https://issues.redhat.com/browse/ISPN-12678
> Project: Infinispan
> Issue Type: Component Upgrade
> Affects Versions: 9.4.21.Final
> Reporter: Yuemei Qiu
> Priority: Major
>
> org.infinispan.commons.CacheException: ISPN014038: infinispan-query.jar module is in the classpath but has not been properly initialised!
> at org.infinispan.query.Search.getQueryFactory(Search.java:74)
> at com.citi.gravity.saas.persister.processor.QuerySkynetMessageProcessor.process(QuerySkynetMessageProcessor.java:57)
> at org.apache.camel.impl.ProcessorEndpoint.onExchange(ProcessorEndpoint.java:103)
> at org.apache.camel.impl.ProcessorEndpoint$1.process(ProcessorEndpoint.java:71)
> at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
> at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148)
> at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
> at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
> at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
> at org.apache.camel.component.seda.SedaConsumer.sendToConsumers(SedaConsumer.java:298)
> at org.apache.camel.component.seda.SedaConsumer.doRun(SedaConsumer.java:210)
> at org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:155)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
4 years, 10 months