I expect our Update routines to still work. I don't know what the code you're showing does though, as those are not our APIs. If you could share a test using our APIs I'll look at it.

The only thing I can think of is that now there's an explicit FLUSH type of Work, if you miss that you might hit stale (deleted) values with a Query.. but this mistake should only be possible if you're driving the backend directly as our public APIs generate them as needed.

>> I expect our Update routines to still work. I don't know what the code >> you're showing does though, as those are not our APIs. >> >> If you could share a test using our APIs I'll look at it. > > What about if you run our test? :-) > (it should be easy -- once you have all CD projects checked-out and built) Sorry I won't have time this week, I'll be on holidays next week, and then probably have to catch up the week after. If you're happy to wait a month, ok.

>> The only thing I can think of is that now there's an explicit FLUSH >> type of Work, if you miss that you might hit stale (deleted) values >> with a Query.. but this mistake should only be possible if you're >> driving the backend directly as our public APIs generate them as >> needed. > > Do we need to call this flush explicitly? Yes

On 02 Apr 2015, at 21:00, Ales Justin <ales.justin(a)gmail.com&gt; wrote: When debugging, I see this -- it get "bbb" twice (old and new, where I would actually expect stale "ccc"): results = {java.util.ArrayList@17380} size = 3 0 = {java.lang.Object[3]@17384} 0 = {com.google.appengine.api.datastore.Key@17395} "test(1)" 1 = {java.lang.String@17396} "#Thu Apr 02 20:53:33 CEST 2015\nprop=STRING\n" 2 = {java.lang.String@17397} "040:aaa" 1 = {java.lang.Object[3]@17385} 0 = {com.google.appengine.api.datastore.Key@17391} "test(1)" 1 = {java.lang.String@17392} "#Thu Apr 02 20:51:24 CEST 2015\nprop=STRING\n" 2 = {java.lang.String@17393} "040:bbb" 2 = {java.lang.Object[3]@17386} 0 = {com.google.appengine.api.datastore.Key@17387} "test(1)" 1 = {java.lang.String@17388} "#Thu Apr 02 20:53:33 CEST 2015\nprop=STRING\n" 2 = {java.lang.String@17389} "040:bbb" But still no idea on why dup "bbb" is there ... -Ales > On 01 Apr 2015, at 11:59, Ales Justin <ales.justin(a)gmail.com <mailto:ales.justin@gmail.com>> wrote: > > Did entity updating changed in HS5 / Lucene4? > (aka Document updating) > > Since this used to work for us (with HS4, Lucene3): > > --- > @Test > public void testProjectionQueriesHandleEntityModificationProperly() throws Exception { > Entity e = createEntity("test", 1) > .withProperty("prop", Arrays.asList("aaa", "bbb", "ccc")) > .store(); > > Query query = new Query("test") > .addProjection(new PropertyProjection("prop", String.class)) > .addSort("prop"); > > assertEquals(3, service.prepare(query).asList(withDefaults()).size()); > > e = createEntity(e.getKey()) > .withProperty("prop", Arrays.asList("aaa", "bbb")) > .store(); > > assertEquals(2, service.prepare(query).asList(withDefaults()).size()); > > service.delete(e.getKey()); > > assertEquals(0, service.prepare(query).asList(withDefaults()).size()); > } > > --- > > Where we add each list/collection value under same field: > > if (propertyValue instanceof Collection) { > Collection collection = (Collection) propertyValue; > for (Object element : collection) { > if (PropertyUtils.isIndexedProperty(element)) { > final Bridge inner = BridgeUtils.matchBridge(element); > luceneOptions.addFieldToDocument(propertyName, inner.objectToString(element), document); > } > } > > --- > > As it looks like the update still keeps the old values around: > > java.lang.AssertionError: expected:<2> but was:<3> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:542) > at org.jboss.test.capedwarf.datastore.test.QueryOptimizationsTest.testProjectionQueriesHandleEntityModificationProperly(QueryOptimizationsTest.java:510) > > --- > > Any way to force / fix this? > > -Ales >

On 03 Apr 2015, at 12:38, Ales Justin <ales.justin(a)gmail.com&gt; wrote: OK, found the culprit ... It is this "weird" rule -- why not remove if the value class is the same?! private boolean shouldRemove(Object value, Object previousValue) { return !(value == null || previousValue == null) && !value.getClass().equals(previousValue.getClass()); } in --- private void processPutKeyValueCommand(final PutKeyValueCommand command, final InvocationContext ctx, final Object previousValue, TransactionContext transactionContext) { final boolean usingSkipIndexCleanupFlag = usingSkipIndexCleanup(command); //whatever the new type, we might still need to cleanup for the previous value (and schedule removal first!) Object value = extractValue(command.getValue()); if (!usingSkipIndexCleanupFlag && updateKnownTypesIfNeeded(previousValue) && shouldRemove(value, previousValue)) { if (shouldModifyIndexes(command, ctx)) { transactionContext = transactionContext == null ? makeTransactionalEventContext() : transactionContext; removeFromIndexes(previousValue, extractValue(command.getKey()), transactionContext); --- -Ales > On 03 Apr 2015, at 12:18, Ales Justin <ales.justin(a)gmail.com <mailto:ales.justin@gmail.com>> wrote: > > I think the problem is in our SearchWorkCreator, and how HS doesn't provide old value when doing update (while doing delete). > > * https://github.com/capedwarf/capedwarf-blue/blob/master/datastore/src/mai... <https://github.com/capedwarf/capedwarf-blue/blob/master/datastore/src/mai... > > We "explode" original value with multiple "similar" values -- due to how projections and indexing works in GAE. > But since we don't get old value, we cannot fully re-construct previous "similar" values. > > This used to work. > What changed in HS? > > Or how to get old value to do proper delete? > e.g. > Grabbing the old value from cache (is it really still there? the old one that is ...), > and creating proper DELETE manually, where I can then change UPDATE to ADD. > > -Ales > >> On 02 Apr 2015, at 21:00, Ales Justin <ales.justin(a)gmail.com <mailto:ales.justin@gmail.com>> wrote: >> >> When debugging, I see this -- it get "bbb" twice (old and new, where I would actually expect stale "ccc"): >> >> results = {java.util.ArrayList@17380} size = 3 >> 0 = {java.lang.Object[3]@17384} >> 0 = {com.google.appengine.api.datastore.Key@17395} "test(1)" >> 1 = {java.lang.String@17396} "#Thu Apr 02 20:53:33 CEST 2015\nprop=STRING\n" >> 2 = {java.lang.String@17397} "040:aaa" >> 1 = {java.lang.Object[3]@17385} >> 0 = {com.google.appengine.api.datastore.Key@17391} "test(1)" >> 1 = {java.lang.String@17392} "#Thu Apr 02 20:51:24 CEST 2015\nprop=STRING\n" >> 2 = {java.lang.String@17393} "040:bbb" >> 2 = {java.lang.Object[3]@17386} >> 0 = {com.google.appengine.api.datastore.Key@17387} "test(1)" >> 1 = {java.lang.String@17388} "#Thu Apr 02 20:53:33 CEST 2015\nprop=STRING\n" >> 2 = {java.lang.String@17389} "040:bbb" >> >> >> But still no idea on why dup "bbb" is there ... >> >> -Ales >> >>> On 01 Apr 2015, at 11:59, Ales Justin <ales.justin(a)gmail.com <mailto:ales.justin@gmail.com>> wrote: >>> >>> Did entity updating changed in HS5 / Lucene4? >>> (aka Document updating) >>> >>> Since this used to work for us (with HS4, Lucene3): >>> >>> --- >>> @Test >>> public void testProjectionQueriesHandleEntityModificationProperly() throws Exception { >>> Entity e = createEntity("test", 1) >>> .withProperty("prop", Arrays.asList("aaa", "bbb", "ccc")) >>> .store(); >>> >>> Query query = new Query("test") >>> .addProjection(new PropertyProjection("prop", String.class)) >>> .addSort("prop"); >>> >>> assertEquals(3, service.prepare(query).asList(withDefaults()).size()); >>> >>> e = createEntity(e.getKey()) >>> .withProperty("prop", Arrays.asList("aaa", "bbb")) >>> .store(); >>> >>> assertEquals(2, service.prepare(query).asList(withDefaults()).size()); >>> >>> service.delete(e.getKey()); >>> >>> assertEquals(0, service.prepare(query).asList(withDefaults()).size()); >>> } >>> >>> --- >>> >>> Where we add each list/collection value under same field: >>> >>> if (propertyValue instanceof Collection) { >>> Collection collection = (Collection) propertyValue; >>> for (Object element : collection) { >>> if (PropertyUtils.isIndexedProperty(element)) { >>> final Bridge inner = BridgeUtils.matchBridge(element); >>> luceneOptions.addFieldToDocument(propertyName, inner.objectToString(element), document); >>> } >>> } >>> >>> --- >>> >>> As it looks like the update still keeps the old values around: >>> >>> java.lang.AssertionError: expected:<2> but was:<3> >>> at org.junit.Assert.fail(Assert.java:88) >>> at org.junit.Assert.failNotEquals(Assert.java:743) >>> at org.junit.Assert.assertEquals(Assert.java:118) >>> at org.junit.Assert.assertEquals(Assert.java:555) >>> at org.junit.Assert.assertEquals(Assert.java:542) >>> at org.jboss.test.capedwarf.datastore.test.QueryOptimizationsTest.testProjectionQueriesHandleEntityModificationProperly(QueryOptimizationsTest.java:510) >>> >>> --- >>> >>> Any way to force / fix this? >>> >>> -Ales >>> >> >