[infinispan-dev] Need to understand, ClusteredCacheWithElasticsearchIndexManagerIT

Thu May 11 09:49:50 EDT 2017

On Thu, May 11, 2017 at 1:54 PM, Katia Aresti <karesti at redhat.com> wrote:

> Hi Gustavo, thanks for the help !
>
> Indeed, I stopped using the rv variable because I realised it was not
> always the compute command result, and somehow I decided I could not take
> for granted in the code that this is the compute command result value. But
> I think this the case only when the PrepareCommand is called on tx mode for
> this particular case.
>
> What confused me a lot in this matter are commands like RemoveCommand, for
> example.
>
> private void processRemoveCommand(final RemoveCommand command, final InvocationContext
> ctx, final Object valueRemoved, TransactionContext transactionContext)
>
> When this method is called from the visitPrepareCommand method, we indeed pass the previous value found just before the call in the cache.
>
> But after, when the same method is called from the visitRemoveCommand we passe the rv parameter. Which is the Remove Command perfom method answer.
>
>
> So, having a look to the RemoveCommand, I can see sometimes we indeed return the prev value. But it might happen depending to return a boolean instead.
>
>
> remove("key", "value")cache2.remove("newGoat");
>
> *LOGS*
>
> Thread => 79
> ctx.getOrigin() => ClusteredCacheWithElasticsearchIndexManagerIT-NodeA-15125
> valueRemoved => Person{name='newGoat', blurb='eats something', age=42, dateOfGraduation=null}
> Thread => 38
> ctx.getOrigin() => ClusteredCacheWithElasticsearchIndexManagerIT-NodeB-23459
> valueRemoved => Person{name='newGoat', blurb='eats something', age=42, dateOfGraduation=null}
> Thread => 47
> ctx.getOrigin() => null
> valueRemoved => Person{name='newGoat', blurb='eats something', age=42, dateOfGraduation=null}
> *removeFromIndexes method is called !!!*
>
>
> But if we call remove with specific value, the method removeFromIndexes is never called.
>
> cache2.remove("newGoat", person4);
>
> *LOGS*
>
> Thread => 79
> ctx.getOrigin() => ClusteredCacheWithElasticsearchIndexManagerIT-NodeA-22063
> valueRemoved => true
> Thread => 38
> ctx.getOrigin() => ClusteredCacheWithElasticsearchIndexManagerIT-NodeB-47249
> valueRemoved => true
> Thread => 47
> ctx.getOrigin() => null
> valueRemoved => true
>
> But in both cases the remove seems to be working, because this assertions work.
>
> found = searchManager.<Person>getQuery(allQuery, Person.class).list();
>
> assertEquals(0, found.size());
>
>
> I'm a bit confused about all this, but we can chat o IRC or bluejeans. It might be some bugs concerning this interceptor, Radim has already opened an issue on this matter.
>
>
It seems to be working, since queries results are correct, but the value is
not actually removed from the indexes.

https://issues.jboss.org/browse/ISPN-7825

Gustavo

> Katia
>
>
> On Wed, May 10, 2017 at 3:33 PM, Gustavo Fernandes <gustavo at infinispan.org
> > wrote:
>
>> The test fails every time NodeB (cache2) happens to *not be* the primary
>> owner of 'newGoat', and needs to forward the command to NodeA. Sequence of
>> events:
>>
>> 1) [NodeB] Compute is called, key is 'newGoat'
>> 2) [NodeB] Command gets visited by the QueryInterceptor, that suspends
>> the execution
>> 3) [NodeB] Owner for 'newGoat' is nodeA, so nodeB forwards the command to
>> nodeA
>>
>> 4) [NodeA] Command gets visited by the QueryInterceptor, that suspends
>> the execution
>> 5) [NodeA] perform() is called in the Compute command
>> 6) [NodeA] Command is then sent to NodeB, which is a backup owner
>>
>> 7) [NodeB] Command gets visited by the QueryInterceptor, that suspends
>> the execution
>> 8) [NodeB] perform() is called in the compute command
>> 9) [NodeB] QueryInterceptor resumes execution. Since command was
>> originated remotely, no indexing is done (due to Index.LOCAL)
>>
>> 9) [NodeA] Receive response from the call done on 6)
>> 10)[NodeA] resumes execution from the QueryInterceptor from 4)
>> 11)[NodeA] Since command was originated remotely, no indexing is done
>> (due to Index.LOCAL)
>>
>> 12)[NodeB] receives response from 3). At this point *the computed value
>> is available* as the return type of the remote invocation
>> 13)[NodeB] resumes the QueryInterceptor invocation from 2)
>> 14)[NodeB] processComputes is then executed, but since the computedValue
>> is not available in the command itself nor in the context, indexing is
>> skipped since there is no value to index or remove
>>
>>
>> Looking at the method visitComputCommand, the variable "rv" stores the
>> return value from the command, but it's not being used, instead the
>> stateBeforeCompute is used which is always null in this scenario,
>> because it is evaluated on 2) which is before the the key exists in the
>> data container:
>>
>> return invokeNextThenAccept(ctx, command, (rCtx, rCommand, rv) -> processComputeCommand(((ComputeCommand) rCommand), rCtx, stateBeforeCompute, null));
>>
>>
>> Gustavo
>>
>>
>> On Tue, May 9, 2017 at 2:21 PM, Katia Aresti <karesti at redhat.com> wrote:
>>
>>> Hi all,
>>>
>>> I'm really struggling with something in order to finish the compute
>>> methods.
>>>
>>> I added a test in *ClusteredCacheWithElasticsearchIndexManagerIT*
>>>
>>> public void testToto() throws Exception {
>>>     SearchManager searchManager = Search.getSearchManager(cache2);
>>>     QueryBuilder queryBuilder = searchManager
>>>           .buildQueryBuilderForClass(Person.class)
>>>           .get();
>>>     Query allQuery = queryBuilder.all().createQuery();
>>>
>>>     String key = "newGoat";
>>>     Person person4 = new Person(key, "eats something", 42);
>>>
>>>     cache2.putIfAbsent(key, person4);
>>>     StaticTestingErrorHandler.assertAllGood(cache1, cache2);
>>>
>>>     List<Person> found = searchManager.<Person>getQuery(allQuery, Person.class).list();
>>>     assertEquals(1, found.size());
>>>     assertTrue(found.contains(person4));
>>> }
>>>
>>> I put some logs in the processPutKeyValueCommand method in the
>>> *QueryInterceptor* to explain what is happening.
>>>
>>> *2 threads*
>>> Sometimes two threads get involved.
>>>
>>> = Thread 72 First (or second) call
>>> It happens from a non local Node. The so the shouldModifyIndexes says
>>> "no, you should not modify any index" because the
>>> IndexModificationStrategy is set to "LOCAL ONLY". [1]
>>>
>>> 72 ctx.getOrigin() = ClusteredCacheWithElasticsearc
>>> hIndexManagerIT-NodeB-19565
>>> 72 should modify false
>>> 72 previousValue null
>>> 72 putValue Person{name='newGoat', blurb='eats something', age=42,
>>> dateOfGraduation=null} // value in the command
>>> 72 contextValue Person{name='newGoat', blurb='eats something', age=42,
>>> dateOfGraduation=null} //value in the invocation context
>>>
>>> = Thread 48 Second (or first) call
>>> the origin is null, and this is considered as a LOCAL in the
>>> SingleKeyNonTxInvocationContext. [2] In this case, the index is
>>> modified correctly, the value in the context has already been set up by the
>>> PutKeyValueCommand and the index get's correctly updated.
>>>
>>> 48 ctx.getOrigin() = null
>>> 48 should modify true
>>> 48 previousValue null
>>> 48 putValue Person{name='newGoat', blurb='eats something', age=42,
>>> dateOfGraduation=null}
>>> 48 contextValue Person{name='newGoat', blurb='eats something', age=42,
>>> dateOfGraduation=null}
>>>
>>> And everything is ok. Everything is fine too in the case of a compute
>>> method instead of the put method.
>>>
>>> But sometimes, this is not executed like that.
>>>
>>> *3 threads*
>>>
>>> What is a bit more weird to me is this second scenario where the
>>> commands are executed both from non local nodes (A and B). And so the index
>>> is not updated.
>>> But  just later, another thread get's involved and calls the
>>> QueryInterceptor with a invocation context where the command has not been
>>> executed (the value is not inside the context and the debugger does not
>>> enter in the perform method, this has happened just twice before). This
>>> call is coming like from a callback? in the QueueAsyncInvocationStage.
>>>
>>> 80 ctx.getOrigin() = ClusteredCacheWithElasticsearc
>>> hIndexManagerIT-NodeA-65110
>>> 80 should modify false
>>> 80 prev null
>>> 80 putValue Person{name='newGoat', blurb='eats something', age=42,
>>> dateOfGraduation=null}
>>> 80 contextValue Person{name='newGoat', blurb='eats something', age=42,
>>> dateOfGraduation=null}
>>>
>>> 38 ctx.getOrigin() = ClusteredCacheWithElasticsearc
>>> hIndexManagerIT-NodeB-35919
>>> 38 should modify false
>>> 38 prev null
>>> 38 putValue Person{name='newGoat', blurb='eats something', age=42,
>>> dateOfGraduation=null}
>>> 38 contextValue Person{name='newGoat', blurb='eats something', age=42,
>>> dateOfGraduation=null}
>>>
>>> 48 ctx.getOrigin() = null
>>> 48 should modify true
>>> 48 prev null
>>> 48 putValue Person{name='newGoat', blurb='eats something', age=42,
>>> dateOfGraduation=null}
>>> 48 contextValue null
>>>
>>>
>>> This execution works perfectly with PutKeyValueCommand. But don't work
>>> wth compute.
>>>
>>> The "computed value" is not inside the Command like put, replace or
>>> others. It is computed in the perform method (if needed). So, the first
>>> time the command is executed in A, the computed value is in the context,
>>> but the index is not updated. Second call, executed in B, value in context,
>>> but the index is not updated. The magic callback is executed, but the
>>> computed value is nowhere because the command is not executed a third time,
>>> so the context is null.
>>>
>>> Can somebody please give me some light on this and explain to me what am
>>> I missing ? Other tests are failing for the same problem,
>>> like org.infinispan.query.blackbox.ClusteredCacheWithInfinis
>>> panDirectoryTest
>>>
>>> Thank you very much for your help !
>>>
>>> Katia
>>>
>>> [1] https://github.com/infinispan/infinispan/blob/master/que
>>> ry/src/main/java/org/infinispan/query/backend/IndexModificat
>>> ionStrategy.java#L50
>>> [2] https://github.com/infinispan/infinispan/blob/master/cor
>>> e/src/main/java/org/infinispan/context/SingleKeyNonTxInvocat
>>> ionContext.java#L39
>>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20170511/b878bf97/attachment-0001.html 