[JBoss JIRA] (ISPN-3780) CLONE - InvalidateL1Command during ST should not cancel writing the entry by ST in nontx
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-3780?page=com.atlassian.jira.plugin.... ]
William Burns commented on ISPN-3780:
-------------------------------------
Radim, do you happen to have a link to the entire trace?
In the cases I am finding the put is retried properly with the new topology and everything is fine. I have tried a few slight variations of the following case you originally stated.
{quote}
1. D executes PUT -> sends command to A
2. A executes the PUT, sends invalidation to C
3. ST from A to C begins
4. C receives InvalidateL1 and puts this key into updateKeys
5. C receives state but ignores the updated key
{quote}
In this case the put on A should end up throwing a OutdatedTopologyException after it tries to commit which would cause a retry to occur from node D with the updated topology which would put the correct value into C.
I wonder though if the L1 invalidation came from the L1LastChanceInterceptor though as that would not cause a retry. It could be that the commit completes, then the topology is updated and then last chance fires, which is why it would have the correct topology id on the command. I will try to get that test up and running. The complete log would help show me if that matches your case or not.
Either way I wouldn't expect the value to be in L1 still. I would expect the value to be removed from the invalidation, can you confirm if that is true? Looking at the log it looks like it was already null "Entry for key key_0000000000003154 is null"
> CLONE - InvalidateL1Command during ST should not cancel writing the entry by ST in nontx
> ----------------------------------------------------------------------------------------
>
> Key: ISPN-3780
> URL: https://issues.jboss.org/browse/ISPN-3780
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.3.0.Final
> Reporter: Radim Vansa
> Assignee: William Burns
> Priority: Critical
> Labels: 620
> Fix For: 6.1.0.Final
>
>
> When a node which is about to receive the entries for some segment receives InvalidateL1Command, this puts the key into StateConsumer.updatedKeys. After the entries for ST are received, the entry which was invalidated is not updated -> this result in losing the entry.
> Btw., in EntryWrappingInterceptor.visitInvalidateL1Command the trace logs all looked up entries for each entry - this causes some performance problems when tracing is on and there are many invalidated entries. Please, do this only once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 6 months
[JBoss JIRA] (ISPN-3780) CLONE - InvalidateL1Command during ST should not cancel writing the entry by ST in nontx
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/ISPN-3780?page=com.atlassian.jira.plugin.... ]
Radim Vansa commented on ISPN-3780:
-----------------------------------
{code}
edg-perf03:07:00:02,044 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (SharedLogChecker-6) Replication task sending ClusteredGetCommand{key=key_0000000000003154, flags=null} to addresses [edg-perf02-17030, edg-perf01-24254] with response mode GET_FIRST
edg-perf03:07:00:02,045 TRACE [org.infinispan.remoting.rpc.RpcManagerImpl] (SharedLogChecker-6) Response(s) to ClusteredGetCommand{key=key_0000000000003154, flags=null} is {edg-perf02-17030=SuccessfulResponse{responseValue=null} }
...
edg-perf02: 07:00:02,445 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-36,edg-perf02-17030) Attempting to execute command: SingleRpcCommand{cacheName='testCache', command=PutKeyValueCommand{key=key_0000000000003154, value=[ #1: 39~552, ], flags=null, putIfAbsent=true, metadata=EmbeddedMetadata{version=null}, successful=true, ignorePreviousValue=false}} [sender=edg-perf04-38684]
...
edg-perf02:07:00:04,011 TRACE [org.infinispan.commands.remote.BaseRpcInvokingCommand] (remote-thread-22) Invoking command PutKeyValueCommand{key=key_0000000000003154, value=[ #1: 39~552, ], flags=null, putIfAbsent=true, metadata=EmbeddedMetadata{version=null}, successful=true, ignorePreviousValue=false}, with originLocal flag set to false
...
edg-perf02:07:00:04,013 TRACE [org.infinispan.distribution.L1ManagerImpl] (remote-thread-22) Invalidating keys [key_0000000000003154] on nodes [edg-perf03-38968]. Use multicast? true
...
edg-perf03:07:00:04,014 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-79,edg-perf03-38968) Attempting to execute command: SingleRpcCommand{cacheName='testCache', command=InvalidateL1Command{num keys=1, forRehash=false, origin=edg-perf04-38684}} [sender=edg-perf02-17030]
...
edg-perf03:07:00:04,014 TRACE [org.infinispan.commands.remote.BaseRpcInvokingCommand] (remote-thread-27) Invoking command InvalidateL1Command{num keys=1, forRehash=false, origin=edg-perf04-38684}, with originLocal flag set to false
...
edg-perf03:07:00:04,015 TRACE [org.infinispan.container.EntryFactoryImpl] (remote-thread-27) Retrieved from container null (isL1Enabled=true, isLocal=true)
...
edg-perf03:07:00:04,015 TRACE [org.infinispan.commands.write.InvalidateL1Command] (remote-thread-27) Preparing to invalidate keys [key_0000000000003154]
edg-perf03:07:00:04,015 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (remote-thread-27) Key key_0000000000003154 modified by a regular tx, state transfer will ignore it
edg-perf03:07:00:04,015 TRACE [org.infinispan.interceptors.EntryWrappingInterceptor] (remote-thread-27) Entry for key key_0000000000003154 is null : not calling commitUpdate
...
edg-perf02:07:00:04,046 TRACE [org.infinispan.remoting.rpc.RpcManagerImpl] (transport-thread-15) edg-perf02-17030 invoking StateResponseCommand{cache=testCache, origin=edg-perf02-17030, topologyId=12} to recipient list [edg-perf03-38968] with options RpcOptions{timeout=600000, unit=MILLISECONDS, fifoOrder=true, totalOrder=false, responseFilter=null, responseMode=SYNCHRONOUS, skipReplicationQueue=false}
...
edg-perf03:07:00:04,082 TRACE [org.infinispan.remoting.InboundInvocationHandlerImpl] (remote-thread-33) Calling perform() on StateResponseCommand{cache=testCache, origin=edg-perf02-17030, topologyId=12}
...
edg-perf03:07:00:04,817 TRACE [org.infinispan.interceptors.CallInterceptor] (remote-thread-33) Executing command: PutKeyValueCommand{key=key_0000000000003154, value=[ #1: 39~552, ], flags=[CACHE_MODE_LOCAL, SKIP_REMOTE_LOOKUP, PUT_FOR_STATE_TRANSFER, SKIP_SHARED_CACHE_STORE, SKIP_OWNERSHIP_CHECK, IGNORE_RETURN_VALUES, SKIP_XSITE_BACKUP], putIfAbsent=false, metadata=EmbeddedMetadata{version=null}, successful=true, ignorePreviousValue=false}.
edg-perf03:07:00:04,817 TRACE [org.infinispan.interceptors.EntryWrappingInterceptor] (remote-thread-33) State transfer will not write key/value key_0000000000003154/[ #1: 39~552, ] because it was already updated by somebody else
{code}
> CLONE - InvalidateL1Command during ST should not cancel writing the entry by ST in nontx
> ----------------------------------------------------------------------------------------
>
> Key: ISPN-3780
> URL: https://issues.jboss.org/browse/ISPN-3780
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.3.0.Final
> Reporter: Radim Vansa
> Assignee: William Burns
> Priority: Critical
> Labels: 620
> Fix For: 6.1.0.Final
>
>
> When a node which is about to receive the entries for some segment receives InvalidateL1Command, this puts the key into StateConsumer.updatedKeys. After the entries for ST are received, the entry which was invalidated is not updated -> this result in losing the entry.
> Btw., in EntryWrappingInterceptor.visitInvalidateL1Command the trace logs all looked up entries for each entry - this causes some performance problems when tracing is on and there are many invalidated entries. Please, do this only once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 6 months
[JBoss JIRA] (ISPN-3780) CLONE - InvalidateL1Command during ST should not cancel writing the entry by ST in nontx
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-3780?page=com.atlassian.jira.plugin.... ]
William Burns commented on ISPN-3780:
-------------------------------------
Radim, I have converted over the existing tx tests we had for this issue and created a couple new ones, but I still haven't been able to reproduce this yet for non tx. Is there any way you could get logs for when this happens? I can keep adding some more to see if I can find it, but I think the logs here would help the most.
Thanks!
> CLONE - InvalidateL1Command during ST should not cancel writing the entry by ST in nontx
> ----------------------------------------------------------------------------------------
>
> Key: ISPN-3780
> URL: https://issues.jboss.org/browse/ISPN-3780
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.3.0.Final
> Reporter: Radim Vansa
> Assignee: William Burns
> Priority: Critical
> Labels: 620
> Fix For: 6.1.0.Final
>
>
> When a node which is about to receive the entries for some segment receives InvalidateL1Command, this puts the key into StateConsumer.updatedKeys. After the entries for ST are received, the entry which was invalidated is not updated -> this result in losing the entry.
> Btw., in EntryWrappingInterceptor.visitInvalidateL1Command the trace logs all looked up entries for each entry - this causes some performance problems when tracing is on and there are many invalidated entries. Please, do this only once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 6 months
[JBoss JIRA] (ISPN-3767) MassIndexer breaks search feature with one node cluster
by Romain Pelisse (JIRA)
[ https://issues.jboss.org/browse/ISPN-3767?page=com.atlassian.jira.plugin.... ]
Romain Pelisse commented on ISPN-3767:
--------------------------------------
Hi Adrian,
I think I created a bit too many test cases, as end up numeroting them :)
Cool to hear that the fix are on the way (no idea what you meant by PR, but sounds good anyway :p). At this point, I just need one thing from you, that you double check the project I published on github and called [ISPN-3767|https://github.com/rpelisse/ISPN-3767], and confirm to me if such set up *can work* with ISPN 5.2.6 or not.
As you mentioned above, adding more than 2 nodes to the grid allowed the Mass Indexer to works properly, and the query to work also (ie not "index corruption" or "invalid state"). I've built this project in order to test that, and I still run into the same issue (import is quick, mass indexing runs smoothly also, but any query fails afterwards, returning always zero).
If there is no chance to have that working in ISPN 5.2.6 - which I find weird, because the unit test seems to work, I'll switch to 6.0.0. Otherwise, I'll be happy to find out what is wrong in my setup.
> MassIndexer breaks search feature with one node cluster
> -------------------------------------------------------
>
> Key: ISPN-3767
> URL: https://issues.jboss.org/browse/ISPN-3767
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.2.4.Final
> Reporter: Romain Pelisse
> Assignee: Adrian Nistor
> Fix For: 6.1.0.Final
>
>
> Hi,
> Trying to cope with the extreme slowliness of put() operation with indexing [1], I've tried to use the MassIndexer, to create the index AFTER adding all the data in the grid. This actually works pretty well, but, when running in a "single node" grid, it "breaks" the search, which always returns 0 result to any kind of query.
> I've modified one of the test suite of InfiniSpan to reproduce the issue:
> https://github.com/rpelisse/infinispan/tree/mass-indexer-breaks-search-wi...
> Once this branch is checked out, just run :
> $ cd ./query
> $ mvn clean -Dtest=org.infinispan.query.distributed.DistributedMassIndexingTest test
> Note: MassIndexer being implemented using the Map/Reduce algorithm, it requires to run within a cluster (ie several instances of ISPN).
> [1] http://stackoverflow.com/questions/10090361/infinispan-very-slow-for-load...
> If run within a single node cluster, the MassIndexer
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 6 months
[JBoss JIRA] (ISPN-3767) MassIndexer breaks search feature with one node cluster
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-3767?page=com.atlassian.jira.plugin.... ]
Adrian Nistor commented on ISPN-3767:
-------------------------------------
Oops, you added a third branch today and I noticed only now. I checked it now but I get a build failure. Should I try harder? Please note the mass indexing problem is solved and the PR is ready to be integrated. Could you try to run your test case #3 against my branch?
> MassIndexer breaks search feature with one node cluster
> -------------------------------------------------------
>
> Key: ISPN-3767
> URL: https://issues.jboss.org/browse/ISPN-3767
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.2.4.Final
> Reporter: Romain Pelisse
> Assignee: Adrian Nistor
> Fix For: 6.1.0.Final
>
>
> Hi,
> Trying to cope with the extreme slowliness of put() operation with indexing [1], I've tried to use the MassIndexer, to create the index AFTER adding all the data in the grid. This actually works pretty well, but, when running in a "single node" grid, it "breaks" the search, which always returns 0 result to any kind of query.
> I've modified one of the test suite of InfiniSpan to reproduce the issue:
> https://github.com/rpelisse/infinispan/tree/mass-indexer-breaks-search-wi...
> Once this branch is checked out, just run :
> $ cd ./query
> $ mvn clean -Dtest=org.infinispan.query.distributed.DistributedMassIndexingTest test
> Note: MassIndexer being implemented using the Map/Reduce algorithm, it requires to run within a cluster (ie several instances of ISPN).
> [1] http://stackoverflow.com/questions/10090361/infinispan-very-slow-for-load...
> If run within a single node cluster, the MassIndexer
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 6 months
[JBoss JIRA] (ISPN-3767) MassIndexer breaks search feature with one node cluster
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-3767?page=com.atlassian.jira.plugin.... ]
Adrian Nistor commented on ISPN-3767:
-------------------------------------
I examined the second test you supplied. Ok, indexing is slow :) This issue is already tracked by ISPN-3586 (no ETA yet).
> MassIndexer breaks search feature with one node cluster
> -------------------------------------------------------
>
> Key: ISPN-3767
> URL: https://issues.jboss.org/browse/ISPN-3767
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.2.4.Final
> Reporter: Romain Pelisse
> Assignee: Adrian Nistor
> Fix For: 6.1.0.Final
>
>
> Hi,
> Trying to cope with the extreme slowliness of put() operation with indexing [1], I've tried to use the MassIndexer, to create the index AFTER adding all the data in the grid. This actually works pretty well, but, when running in a "single node" grid, it "breaks" the search, which always returns 0 result to any kind of query.
> I've modified one of the test suite of InfiniSpan to reproduce the issue:
> https://github.com/rpelisse/infinispan/tree/mass-indexer-breaks-search-wi...
> Once this branch is checked out, just run :
> $ cd ./query
> $ mvn clean -Dtest=org.infinispan.query.distributed.DistributedMassIndexingTest test
> Note: MassIndexer being implemented using the Map/Reduce algorithm, it requires to run within a cluster (ie several instances of ISPN).
> [1] http://stackoverflow.com/questions/10090361/infinispan-very-slow-for-load...
> If run within a single node cluster, the MassIndexer
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 6 months
[JBoss JIRA] (ISPN-3767) MassIndexer breaks search feature with one node cluster
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-3767?page=com.atlassian.jira.plugin.... ]
Adrian Nistor updated ISPN-3767:
--------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/2259
> MassIndexer breaks search feature with one node cluster
> -------------------------------------------------------
>
> Key: ISPN-3767
> URL: https://issues.jboss.org/browse/ISPN-3767
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.2.4.Final
> Reporter: Romain Pelisse
> Assignee: Adrian Nistor
> Fix For: 6.1.0.Final
>
>
> Hi,
> Trying to cope with the extreme slowliness of put() operation with indexing [1], I've tried to use the MassIndexer, to create the index AFTER adding all the data in the grid. This actually works pretty well, but, when running in a "single node" grid, it "breaks" the search, which always returns 0 result to any kind of query.
> I've modified one of the test suite of InfiniSpan to reproduce the issue:
> https://github.com/rpelisse/infinispan/tree/mass-indexer-breaks-search-wi...
> Once this branch is checked out, just run :
> $ cd ./query
> $ mvn clean -Dtest=org.infinispan.query.distributed.DistributedMassIndexingTest test
> Note: MassIndexer being implemented using the Map/Reduce algorithm, it requires to run within a cluster (ie several instances of ISPN).
> [1] http://stackoverflow.com/questions/10090361/infinispan-very-slow-for-load...
> If run within a single node cluster, the MassIndexer
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 6 months
[JBoss JIRA] (ISPN-3767) MassIndexer breaks search feature with one node cluster
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-3767?page=com.atlassian.jira.plugin.... ]
Adrian Nistor commented on ISPN-3767:
-------------------------------------
Any chance to migrate your app to use 6.x? We stopped backporting fixes to 5.2.x.
Also, regarding map-reduce not working in local mode, that was fixed in 5.3 (ISPN-2812).
> MassIndexer breaks search feature with one node cluster
> -------------------------------------------------------
>
> Key: ISPN-3767
> URL: https://issues.jboss.org/browse/ISPN-3767
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.2.4.Final
> Reporter: Romain Pelisse
> Assignee: Adrian Nistor
> Priority: Minor
>
> Hi,
> Trying to cope with the extreme slowliness of put() operation with indexing [1], I've tried to use the MassIndexer, to create the index AFTER adding all the data in the grid. This actually works pretty well, but, when running in a "single node" grid, it "breaks" the search, which always returns 0 result to any kind of query.
> I've modified one of the test suite of InfiniSpan to reproduce the issue:
> https://github.com/rpelisse/infinispan/tree/mass-indexer-breaks-search-wi...
> Once this branch is checked out, just run :
> $ cd ./query
> $ mvn clean -Dtest=org.infinispan.query.distributed.DistributedMassIndexingTest test
> Note: MassIndexer being implemented using the Map/Reduce algorithm, it requires to run within a cluster (ie several instances of ISPN).
> [1] http://stackoverflow.com/questions/10090361/infinispan-very-slow-for-load...
> If run within a single node cluster, the MassIndexer
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 6 months