[Red Hat JIRA] (ISPN-10292) It should be possible to add protobuf files directly to the server installation
by Tristan Tarrant (Jira)
[ https://issues.redhat.com/browse/ISPN-10292?page=com.atlassian.jira.plugi... ]
Tristan Tarrant updated ISPN-10292:
-----------------------------------
Description:
[https://github.com/infinispan/infinispan-designs/pull/24]
Definition files (protobuf) for queries and indexing need to be added with client requests during runtime.
It will be stored in an internal cache and there is a persistence with a filestore to ensure it will survive restarts.
But this is not always ensured as there might be catastropic failures or unclean shutdown which can cause inconsistence or complete loss.
The behaviour in such cases is not deterministic and heavy to track or solve.
For this reason it should be possible to add the protobuf definitions directly to the server installation which is picked up during startup.
It might be a local cache as the expectation is that all nodes will have the same, but a repl cache might be possible as well.
was:
Definition files (protobuf) for queries and indexing need to be added with client requests during runtime.
It will be stored in an internal cache and there is a persistence with a filestore to ensure it will survive restarts.
But this is not always ensured as there might be catastropic failures or unclean shutdown which can cause inconsistence or complete loss.
The behaviour in such cases is not deterministic and heavy to track or solve.
For this reason it should be possible to add the protobuf definitions directly to the server installation which is picked up during startup.
It might be a local cache as the expectation is that all nodes will have the same, but a repl cache might be possible as well.
> It should be possible to add protobuf files directly to the server installation
> -------------------------------------------------------------------------------
>
> Key: ISPN-10292
> URL: https://issues.redhat.com/browse/ISPN-10292
> Project: Infinispan
> Issue Type: Feature Request
> Components: Server
> Reporter: Wolf-Dieter Fink
> Assignee: Gustavo Fernandes
> Priority: Major
> Labels: protobuf, query
>
> [https://github.com/infinispan/infinispan-designs/pull/24]
>
> Definition files (protobuf) for queries and indexing need to be added with client requests during runtime.
> It will be stored in an internal cache and there is a persistence with a filestore to ensure it will survive restarts.
> But this is not always ensured as there might be catastropic failures or unclean shutdown which can cause inconsistence or complete loss.
> The behaviour in such cases is not deterministic and heavy to track or solve.
> For this reason it should be possible to add the protobuf definitions directly to the server installation which is picked up during startup.
> It might be a local cache as the expectation is that all nodes will have the same, but a repl cache might be possible as well.
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
3 years, 2 months
[Red Hat JIRA] (ISPN-11598) EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead random failures
by Dan Berindei (Jira)
[ https://issues.redhat.com/browse/ISPN-11598?page=com.atlassian.jira.plugi... ]
Dan Berindei commented on ISPN-11598:
-------------------------------------
The reason the test is still failing is because it assumes that {{DataOperationOrderer.completeOperation()}} will unblock the pending operation, but will not use the same thread for running it.
That assumption is wrong. E.g. here the test triggers the {{before_release}} event for the {{put(other-key)}} operation's {{evicted-key}} orderer, and expects the {{evicted-key}} activation for {{get(evicted-key)}} to happen in another thread. But because the activation is too fast, the {{get(evict-key)}} operation resumes on the same thread and waits for another {{before_release}} event instead of triggering the {{after_invocation}} event that the main test thread is waiting for:
{noformat}
15:56:35,757 TRACE (testng-Test:[]) [CheckPoint] Waiting for event after_invocation * 1
15:56:35,764 TRACE (non-blocking-thread-Test-NodeK-p13520-t4:[]) [CheckPoint] Received event before_release * 1 (available = 0, total = 2)
15:56:35,764 TRACE (non-blocking-thread-Test-NodeK-p13520-t4:[]) [DataOperationOrderer] Ordered future java.util.concurrent.CompletableFuture@6d34aaac[Not completed, 1 dependents] is completed for key evicted-key from op READ
15:56:35,764 TRACE (non-blocking-thread-Test-NodeK-p13520-t4:[]) [CacheLoaderInterceptor] Loading entry for key evicted-key
15:56:35,764 TRACE (non-blocking-thread-Test-NodeK-p13520-t4:[]) [BlockingManagerImpl] Submitting blocking supply operation StoreAdapter-load-5 to blocking thread
15:56:35,765 TRACE (non-blocking-thread-Test-NodeK-p13520-t4:[]) [BlockingManagerImpl] Stage for StoreAdapter-load-5 was already completed, returning in same thread
15:56:35,766 TRACE (non-blocking-thread-Test-NodeK-p13520-t4:[]) [DataOperationOrderer] Ordering upcoming future java.util.concurrent.CompletableFuture@7e1e1607[Not completed] for key other-key to run after null
15:56:35,766 TRACE (non-blocking-thread-Test-NodeK-p13520-t4:[]) [PassivationManagerImpl] Passivating entry other-key
15:56:35,768 TRACE (non-blocking-thread-Test-NodeK-p13520-t4:[]) [DataOperationOrderer] Ordered future java.util.concurrent.CompletableFuture@7e1e1607[Not completed] is completed for key other-key from op READ
15:56:35,768 TRACE (non-blocking-thread-Test-NodeK-p13520-t4:[]) [PassivationCacheLoaderInterceptor] Activating key: evicted-key - not waiting for response
15:56:35,768 TRACE (non-blocking-thread-Test-NodeK-p13520-t4:[]) [PersistenceManagerImpl] Deleting entry for key evicted-key from stores
15:56:35,770 TRACE (non-blocking-thread-Test-NodeK-p13520-t4:[]) [CheckPoint] Triggering event before_invocation * 1 (available = 1, total = 3)
15:56:35,770 TRACE (non-blocking-thread-Test-NodeK-p13520-t4:[]) [CheckPoint] Waiting for event before_release * 1
{noformat}
> EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead random failures
> ------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-11598
> URL: https://issues.redhat.com/browse/ISPN-11598
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite
> Affects Versions: 11.0.0.Dev03, 12.0.0.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 12.1.0.Final
>
>
> According to the code comments, the test is supposed to block the {{put(evicted-key)}} operation before releasing the orderer, but because of {{operationCheckPoint.trigger(Mocks.BEFORE_RELEASE)}} it only blocks *after* releasing the orderer. This means the eviction triggered by {{put(other-key)}} and the activation triggered later by {{get(evicted-key)}} are not blocked.
> All three operations eventually call {{operationCheckPoint.awaitStrict(Mocks.BEFORE_RELEASE)}}. The test triggers that event twice, so it unblocks {{put(evicted-key)}} *and* one of {{put(other-key)}}/{{get(evicted-key)}}. The two unblocked operations then compete for a single {{Mocks.AFTER_RELEASE}} event.
> If {{put(evicted-key)}} receives the {{Mocks.AFTER_RELEASE}} event, the test passes, because the other operations can finish without waiting for the activation/passivation to finish. (It just needs to be queued in )
> Because {{CheckPoint}} doesn't guarantee waiters receive events in the queueing order, either {{put(other-key, "value"}}, or {{get(evicted-key)}} could receive the {{AFTER_RELEASE}} event from {{put(evicted-key, "value")}}, making it time out.
> {noformat}
> 19:53:59,795 TRACE (ForkThread-1,Test:[]) [InvocationContextInterceptor] Invoked with command PutKeyValueCommand{key=evicted-key, value=value, flags=[SKIP_CACHE_LOAD], commandInvocationId=CommandInvocation:<local>:37070, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{version=null, lifespan=-1, maxIdle=-1}, successful=true, topologyId=-1} and InvocationContext [SingleKeyNonTxInvocationContext{isLocked=false, key=null, cacheEntry=null, origin=null, lockOwner=CommandInvocation:<local>:37070}]
> 19:53:59,807 TRACE (non-blocking-thread-Test-NodeK-p23146-t1:[]) [DataOperationOrderer] Ordered future java.util.concurrent.CompletableFuture@76d1e6bc[Not completed] is completed for key evicted-key from op WRITE
> 19:53:59,811 TRACE (non-blocking-thread-Test-NodeK-p23146-t1:[]) [CheckPoint] Waiting for event post_released * 1
> {noformat}
> ^ put(evicted-key) is blocked
> {noformat}
> 19:53:59,807 TRACE (testng-Test:[]) [InvocationContextInterceptor] Invoked with command PutKeyValueCommand{key=other-key, value=other-value, flags=[], commandInvocationId=CommandInvocation:<local>:37073, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{version=null, lifespan=-1, maxIdle=-1}, successful=true, topologyId=-1} and InvocationContext [SingleKeyNonTxInvocationContext{isLocked=false, key=null, cacheEntry=null, origin=null, lockOwner=CommandInvocation:<local>:37073}]
> 19:53:59,816 TRACE (non-blocking-thread-Test-NodeK-p23146-t2:[]) [AbstractInternalDataContainer] Store other-key=ImmortalCacheEntry{key=other-key, value=other-value, internalMetadata=null} in container
> 19:53:59,817 TRACE (non-blocking-thread-Test-NodeK-p23146-t2:[]) [DataOperationOrderer] Ordering upcoming future java.util.concurrent.CompletableFuture@32541e18[Not completed] for key evicted-key to run after null
> {noformat}
> ^ put(other-key) is not blocked here
> {noformat}
> 19:53:59,818 TRACE (non-blocking-thread-Test-NodeK-p23146-t2:[]) [PassivationManagerImpl] Passivating entry evicted-key
> 19:53:59,818 TRACE (non-blocking-thread-Test-NodeK-p23146-t2:[]) [PersistenceManagerImpl] Writing to all stores for id 67588
> 19:53:59,841 TRACE (non-blocking-thread-Test-NodeK-p23146-t3:[]) [CompletionStages] Continuing execution of id 67588
> 19:53:59,844 TRACE (non-blocking-thread-Test-NodeK-p23146-t3:[]) [CheckPoint] Waiting for event pre_released * 1
> {noformat}
> ^ put(other-key) blocks in the checkpoint
> {noformat}
> 19:53:59,956 TRACE (testng-Test:[]) [CheckPoint] Triggering event pre_released * 1 (available = 1, total = 2)
> 19:53:59,956 TRACE (testng-Test:[]) [CheckPoint] Triggering event post_released * 1 (available = 1, total = 1)
> 19:53:59,956 TRACE (non-blocking-thread-Test-NodeK-p23146-t3:[]) [CheckPoint] Received event pre_released * 1 (available = 0, total = 2)
> {noformat}
> ^ put(other-key) is unblocked, put(evicted-key) stays blocked
> {noformat}
> 19:54:09,998 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.eviction.impl.EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead
> java.util.concurrent.TimeoutException: null
> at java.util.concurrent.FutureTask.get(FutureTask.java:204) ~[?:?]
> at org.infinispan.eviction.impl.EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead(EvictionWithPassivationAndConcurrentOperationsTest.java:99) ~[test-classes/:?]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
3 years, 2 months
[Red Hat JIRA] (ISPN-11598) EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead random failures
by Dan Berindei (Jira)
[ https://issues.redhat.com/browse/ISPN-11598?page=com.atlassian.jira.plugi... ]
Dan Berindei updated ISPN-11598:
--------------------------------
Fix Version/s: 12.1.0.Final
(was: 11.0.0.Dev04)
> EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead random failures
> ------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-11598
> URL: https://issues.redhat.com/browse/ISPN-11598
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite
> Affects Versions: 11.0.0.Dev03, 12.0.0.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 12.1.0.Final
>
>
> According to the code comments, the test is supposed to block the {{put(evicted-key)}} operation before releasing the orderer, but because of {{operationCheckPoint.trigger(Mocks.BEFORE_RELEASE)}} it only blocks *after* releasing the orderer. This means the eviction triggered by {{put(other-key)}} and the activation triggered later by {{get(evicted-key)}} are not blocked.
> All three operations eventually call {{operationCheckPoint.awaitStrict(Mocks.BEFORE_RELEASE)}}. The test triggers that event twice, so it unblocks {{put(evicted-key)}} *and* one of {{put(other-key)}}/{{get(evicted-key)}}. The two unblocked operations then compete for a single {{Mocks.AFTER_RELEASE}} event.
> If {{put(evicted-key)}} receives the {{Mocks.AFTER_RELEASE}} event, the test passes, because the other operations can finish without waiting for the activation/passivation to finish. (It just needs to be queued in )
> Because {{CheckPoint}} doesn't guarantee waiters receive events in the queueing order, either {{put(other-key, "value"}}, or {{get(evicted-key)}} could receive the {{AFTER_RELEASE}} event from {{put(evicted-key, "value")}}, making it time out.
> {noformat}
> 19:53:59,795 TRACE (ForkThread-1,Test:[]) [InvocationContextInterceptor] Invoked with command PutKeyValueCommand{key=evicted-key, value=value, flags=[SKIP_CACHE_LOAD], commandInvocationId=CommandInvocation:<local>:37070, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{version=null, lifespan=-1, maxIdle=-1}, successful=true, topologyId=-1} and InvocationContext [SingleKeyNonTxInvocationContext{isLocked=false, key=null, cacheEntry=null, origin=null, lockOwner=CommandInvocation:<local>:37070}]
> 19:53:59,807 TRACE (non-blocking-thread-Test-NodeK-p23146-t1:[]) [DataOperationOrderer] Ordered future java.util.concurrent.CompletableFuture@76d1e6bc[Not completed] is completed for key evicted-key from op WRITE
> 19:53:59,811 TRACE (non-blocking-thread-Test-NodeK-p23146-t1:[]) [CheckPoint] Waiting for event post_released * 1
> {noformat}
> ^ put(evicted-key) is blocked
> {noformat}
> 19:53:59,807 TRACE (testng-Test:[]) [InvocationContextInterceptor] Invoked with command PutKeyValueCommand{key=other-key, value=other-value, flags=[], commandInvocationId=CommandInvocation:<local>:37073, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{version=null, lifespan=-1, maxIdle=-1}, successful=true, topologyId=-1} and InvocationContext [SingleKeyNonTxInvocationContext{isLocked=false, key=null, cacheEntry=null, origin=null, lockOwner=CommandInvocation:<local>:37073}]
> 19:53:59,816 TRACE (non-blocking-thread-Test-NodeK-p23146-t2:[]) [AbstractInternalDataContainer] Store other-key=ImmortalCacheEntry{key=other-key, value=other-value, internalMetadata=null} in container
> 19:53:59,817 TRACE (non-blocking-thread-Test-NodeK-p23146-t2:[]) [DataOperationOrderer] Ordering upcoming future java.util.concurrent.CompletableFuture@32541e18[Not completed] for key evicted-key to run after null
> {noformat}
> ^ put(other-key) is not blocked here
> {noformat}
> 19:53:59,818 TRACE (non-blocking-thread-Test-NodeK-p23146-t2:[]) [PassivationManagerImpl] Passivating entry evicted-key
> 19:53:59,818 TRACE (non-blocking-thread-Test-NodeK-p23146-t2:[]) [PersistenceManagerImpl] Writing to all stores for id 67588
> 19:53:59,841 TRACE (non-blocking-thread-Test-NodeK-p23146-t3:[]) [CompletionStages] Continuing execution of id 67588
> 19:53:59,844 TRACE (non-blocking-thread-Test-NodeK-p23146-t3:[]) [CheckPoint] Waiting for event pre_released * 1
> {noformat}
> ^ put(other-key) blocks in the checkpoint
> {noformat}
> 19:53:59,956 TRACE (testng-Test:[]) [CheckPoint] Triggering event pre_released * 1 (available = 1, total = 2)
> 19:53:59,956 TRACE (testng-Test:[]) [CheckPoint] Triggering event post_released * 1 (available = 1, total = 1)
> 19:53:59,956 TRACE (non-blocking-thread-Test-NodeK-p23146-t3:[]) [CheckPoint] Received event pre_released * 1 (available = 0, total = 2)
> {noformat}
> ^ put(other-key) is unblocked, put(evicted-key) stays blocked
> {noformat}
> 19:54:09,998 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.eviction.impl.EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead
> java.util.concurrent.TimeoutException: null
> at java.util.concurrent.FutureTask.get(FutureTask.java:204) ~[?:?]
> at org.infinispan.eviction.impl.EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead(EvictionWithPassivationAndConcurrentOperationsTest.java:99) ~[test-classes/:?]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
3 years, 2 months
[Red Hat JIRA] (ISPN-11598) EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead random failures
by Dan Berindei (Jira)
[ https://issues.redhat.com/browse/ISPN-11598?page=com.atlassian.jira.plugi... ]
Dan Berindei updated ISPN-11598:
--------------------------------
Affects Version/s: 12.0.0.Final
> EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead random failures
> ------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-11598
> URL: https://issues.redhat.com/browse/ISPN-11598
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite
> Affects Versions: 11.0.0.Dev03, 12.0.0.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 11.0.0.Dev04
>
>
> According to the code comments, the test is supposed to block the {{put(evicted-key)}} operation before releasing the orderer, but because of {{operationCheckPoint.trigger(Mocks.BEFORE_RELEASE)}} it only blocks *after* releasing the orderer. This means the eviction triggered by {{put(other-key)}} and the activation triggered later by {{get(evicted-key)}} are not blocked.
> All three operations eventually call {{operationCheckPoint.awaitStrict(Mocks.BEFORE_RELEASE)}}. The test triggers that event twice, so it unblocks {{put(evicted-key)}} *and* one of {{put(other-key)}}/{{get(evicted-key)}}. The two unblocked operations then compete for a single {{Mocks.AFTER_RELEASE}} event.
> If {{put(evicted-key)}} receives the {{Mocks.AFTER_RELEASE}} event, the test passes, because the other operations can finish without waiting for the activation/passivation to finish. (It just needs to be queued in )
> Because {{CheckPoint}} doesn't guarantee waiters receive events in the queueing order, either {{put(other-key, "value"}}, or {{get(evicted-key)}} could receive the {{AFTER_RELEASE}} event from {{put(evicted-key, "value")}}, making it time out.
> {noformat}
> 19:53:59,795 TRACE (ForkThread-1,Test:[]) [InvocationContextInterceptor] Invoked with command PutKeyValueCommand{key=evicted-key, value=value, flags=[SKIP_CACHE_LOAD], commandInvocationId=CommandInvocation:<local>:37070, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{version=null, lifespan=-1, maxIdle=-1}, successful=true, topologyId=-1} and InvocationContext [SingleKeyNonTxInvocationContext{isLocked=false, key=null, cacheEntry=null, origin=null, lockOwner=CommandInvocation:<local>:37070}]
> 19:53:59,807 TRACE (non-blocking-thread-Test-NodeK-p23146-t1:[]) [DataOperationOrderer] Ordered future java.util.concurrent.CompletableFuture@76d1e6bc[Not completed] is completed for key evicted-key from op WRITE
> 19:53:59,811 TRACE (non-blocking-thread-Test-NodeK-p23146-t1:[]) [CheckPoint] Waiting for event post_released * 1
> {noformat}
> ^ put(evicted-key) is blocked
> {noformat}
> 19:53:59,807 TRACE (testng-Test:[]) [InvocationContextInterceptor] Invoked with command PutKeyValueCommand{key=other-key, value=other-value, flags=[], commandInvocationId=CommandInvocation:<local>:37073, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{version=null, lifespan=-1, maxIdle=-1}, successful=true, topologyId=-1} and InvocationContext [SingleKeyNonTxInvocationContext{isLocked=false, key=null, cacheEntry=null, origin=null, lockOwner=CommandInvocation:<local>:37073}]
> 19:53:59,816 TRACE (non-blocking-thread-Test-NodeK-p23146-t2:[]) [AbstractInternalDataContainer] Store other-key=ImmortalCacheEntry{key=other-key, value=other-value, internalMetadata=null} in container
> 19:53:59,817 TRACE (non-blocking-thread-Test-NodeK-p23146-t2:[]) [DataOperationOrderer] Ordering upcoming future java.util.concurrent.CompletableFuture@32541e18[Not completed] for key evicted-key to run after null
> {noformat}
> ^ put(other-key) is not blocked here
> {noformat}
> 19:53:59,818 TRACE (non-blocking-thread-Test-NodeK-p23146-t2:[]) [PassivationManagerImpl] Passivating entry evicted-key
> 19:53:59,818 TRACE (non-blocking-thread-Test-NodeK-p23146-t2:[]) [PersistenceManagerImpl] Writing to all stores for id 67588
> 19:53:59,841 TRACE (non-blocking-thread-Test-NodeK-p23146-t3:[]) [CompletionStages] Continuing execution of id 67588
> 19:53:59,844 TRACE (non-blocking-thread-Test-NodeK-p23146-t3:[]) [CheckPoint] Waiting for event pre_released * 1
> {noformat}
> ^ put(other-key) blocks in the checkpoint
> {noformat}
> 19:53:59,956 TRACE (testng-Test:[]) [CheckPoint] Triggering event pre_released * 1 (available = 1, total = 2)
> 19:53:59,956 TRACE (testng-Test:[]) [CheckPoint] Triggering event post_released * 1 (available = 1, total = 1)
> 19:53:59,956 TRACE (non-blocking-thread-Test-NodeK-p23146-t3:[]) [CheckPoint] Received event pre_released * 1 (available = 0, total = 2)
> {noformat}
> ^ put(other-key) is unblocked, put(evicted-key) stays blocked
> {noformat}
> 19:54:09,998 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.eviction.impl.EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead
> java.util.concurrent.TimeoutException: null
> at java.util.concurrent.FutureTask.get(FutureTask.java:204) ~[?:?]
> at org.infinispan.eviction.impl.EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead(EvictionWithPassivationAndConcurrentOperationsTest.java:99) ~[test-classes/:?]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
3 years, 2 months
[Red Hat JIRA] (ISPN-11598) EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead random failures
by Dan Berindei (Jira)
[ https://issues.redhat.com/browse/ISPN-11598?page=com.atlassian.jira.plugi... ]
Dan Berindei reopened ISPN-11598:
---------------------------------
The test is still failing randomly in master:
{noformat}
15:56:45,758 ERROR (testng-Test:[]) [CheckPoint] Timed out waiting for event after_invocation * 1 (available = 0, total = 1)
15:56:45,778 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.eviction.impl.EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead
java.util.concurrent.TimeoutException: Timed out waiting for event after_invocation
at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:50) ~[test-classes/:?]
at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:40) ~[test-classes/:?]
at org.infinispan.eviction.impl.EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead(EvictionWithPassivationAndConcurrentOperationsTest.java:107) ~[test-classes/:?]
{noformat}
> EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead random failures
> ------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-11598
> URL: https://issues.redhat.com/browse/ISPN-11598
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite
> Affects Versions: 11.0.0.Dev03
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 11.0.0.Dev04
>
>
> According to the code comments, the test is supposed to block the {{put(evicted-key)}} operation before releasing the orderer, but because of {{operationCheckPoint.trigger(Mocks.BEFORE_RELEASE)}} it only blocks *after* releasing the orderer. This means the eviction triggered by {{put(other-key)}} and the activation triggered later by {{get(evicted-key)}} are not blocked.
> All three operations eventually call {{operationCheckPoint.awaitStrict(Mocks.BEFORE_RELEASE)}}. The test triggers that event twice, so it unblocks {{put(evicted-key)}} *and* one of {{put(other-key)}}/{{get(evicted-key)}}. The two unblocked operations then compete for a single {{Mocks.AFTER_RELEASE}} event.
> If {{put(evicted-key)}} receives the {{Mocks.AFTER_RELEASE}} event, the test passes, because the other operations can finish without waiting for the activation/passivation to finish. (It just needs to be queued in )
> Because {{CheckPoint}} doesn't guarantee waiters receive events in the queueing order, either {{put(other-key, "value"}}, or {{get(evicted-key)}} could receive the {{AFTER_RELEASE}} event from {{put(evicted-key, "value")}}, making it time out.
> {noformat}
> 19:53:59,795 TRACE (ForkThread-1,Test:[]) [InvocationContextInterceptor] Invoked with command PutKeyValueCommand{key=evicted-key, value=value, flags=[SKIP_CACHE_LOAD], commandInvocationId=CommandInvocation:<local>:37070, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{version=null, lifespan=-1, maxIdle=-1}, successful=true, topologyId=-1} and InvocationContext [SingleKeyNonTxInvocationContext{isLocked=false, key=null, cacheEntry=null, origin=null, lockOwner=CommandInvocation:<local>:37070}]
> 19:53:59,807 TRACE (non-blocking-thread-Test-NodeK-p23146-t1:[]) [DataOperationOrderer] Ordered future java.util.concurrent.CompletableFuture@76d1e6bc[Not completed] is completed for key evicted-key from op WRITE
> 19:53:59,811 TRACE (non-blocking-thread-Test-NodeK-p23146-t1:[]) [CheckPoint] Waiting for event post_released * 1
> {noformat}
> ^ put(evicted-key) is blocked
> {noformat}
> 19:53:59,807 TRACE (testng-Test:[]) [InvocationContextInterceptor] Invoked with command PutKeyValueCommand{key=other-key, value=other-value, flags=[], commandInvocationId=CommandInvocation:<local>:37073, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{version=null, lifespan=-1, maxIdle=-1}, successful=true, topologyId=-1} and InvocationContext [SingleKeyNonTxInvocationContext{isLocked=false, key=null, cacheEntry=null, origin=null, lockOwner=CommandInvocation:<local>:37073}]
> 19:53:59,816 TRACE (non-blocking-thread-Test-NodeK-p23146-t2:[]) [AbstractInternalDataContainer] Store other-key=ImmortalCacheEntry{key=other-key, value=other-value, internalMetadata=null} in container
> 19:53:59,817 TRACE (non-blocking-thread-Test-NodeK-p23146-t2:[]) [DataOperationOrderer] Ordering upcoming future java.util.concurrent.CompletableFuture@32541e18[Not completed] for key evicted-key to run after null
> {noformat}
> ^ put(other-key) is not blocked here
> {noformat}
> 19:53:59,818 TRACE (non-blocking-thread-Test-NodeK-p23146-t2:[]) [PassivationManagerImpl] Passivating entry evicted-key
> 19:53:59,818 TRACE (non-blocking-thread-Test-NodeK-p23146-t2:[]) [PersistenceManagerImpl] Writing to all stores for id 67588
> 19:53:59,841 TRACE (non-blocking-thread-Test-NodeK-p23146-t3:[]) [CompletionStages] Continuing execution of id 67588
> 19:53:59,844 TRACE (non-blocking-thread-Test-NodeK-p23146-t3:[]) [CheckPoint] Waiting for event pre_released * 1
> {noformat}
> ^ put(other-key) blocks in the checkpoint
> {noformat}
> 19:53:59,956 TRACE (testng-Test:[]) [CheckPoint] Triggering event pre_released * 1 (available = 1, total = 2)
> 19:53:59,956 TRACE (testng-Test:[]) [CheckPoint] Triggering event post_released * 1 (available = 1, total = 1)
> 19:53:59,956 TRACE (non-blocking-thread-Test-NodeK-p23146-t3:[]) [CheckPoint] Received event pre_released * 1 (available = 0, total = 2)
> {noformat}
> ^ put(other-key) is unblocked, put(evicted-key) stays blocked
> {noformat}
> 19:54:09,998 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.eviction.impl.EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead
> java.util.concurrent.TimeoutException: null
> at java.util.concurrent.FutureTask.get(FutureTask.java:204) ~[?:?]
> at org.infinispan.eviction.impl.EvictionWithPassivationAndConcurrentOperationsTest.testEvictionDuringWriteWithConcurrentRead(EvictionWithPassivationAndConcurrentOperationsTest.java:99) ~[test-classes/:?]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
3 years, 2 months