Empty result on query while data is still present ------------------------------------------------- Key: ISPN-11025 URL: https://issues.jboss.org/browse/ISPN-11025 Project: Infinispan Issue Type: Bug Components: Remote Querying Affects Versions: 10.0.1.Final Environment: 12 node Infinispan Cluster, OpenShift 4.2 Reporter: Jens Reimann Priority: Major In a loop like this: 1) Query 100 next entries 2) For each entry: delete entry by key The query first returns a full set of 100 entries. Then starts to return non-full entries (while the total number found indicates that there are more than 100 entries still left). At some point it starts returning results with 0 entries, still showing that a the total number of entries is above zero: {code} 12:33:15.874 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 100 entries in this iteration. Total remaining: 295419. 12:33:16.648 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 68 entries in this iteration. Total remaining: 295351. 12:33:17.535 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 62 entries in this iteration. Total remaining: 295289. 12:33:18.206 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 38 entries in this iteration. Total remaining: 295251. 12:33:19.054 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 62 entries in this iteration. Total remaining: 295189. 12:33:19.766 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 38 entries in this iteration. Total remaining: 295151. 12:33:20.593 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 62 entries in this iteration. Total remaining: 295089. 12:33:21.336 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 38 entries in this iteration. Total remaining: 295051. 12:33:22.209 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 62 entries in this iteration. Total remaining: 294989. 12:33:22.458 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 38 entries in this iteration. Total remaining: 294951. 12:33:23.672 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 100 entries in this iteration. Total remaining: 294851. 12:33:24.946 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 100 entries in this iteration. Total remaining: 294751. 12:33:25.065 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 21 entries in this iteration. Total remaining: 294730. 12:33:25.114 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 0 entries in this iteration. Total remaining: 294730. 12:33:25.168 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 0 entries in this iteration. Total remaining: 294730. 12:33:25.224 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 0 entries in this iteration. Total remaining: 294730. 12:33:25.274 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 0 entries in this iteration. Total remaining: 294730. 12:33:25.326 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 0 entries in this iteration. Total remaining: 294730. 12:33:25.387 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 0 entries in this iteration. Total remaining: 294730. 12:33:25.442 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 0 entries in this iteration. Total remaining: 294730. 12:33:25.494 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 0 entries in this iteration. Total remaining: 294730. 12:33:25.547 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 0 entries in this iteration. Total remaining: 294730. 12:33:25.606 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 0 entries in this iteration. Total remaining: 294730. 12:33:25.659 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 0 entries in this iteration. Total remaining: 294730. 12:33:26.044 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 100 entries in this iteration. Total remaining: 294630. 12:33:26.079 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 0 entries in this iteration. Total remaining: 294630. 12:33:26.121 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 0 entries in this iteration. Total remaining: 294630. 12:33:26.154 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 0 entries in this iteration. Total remaining: 294630. 12:33:26.192 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 0 entries in this iteration. Total remaining: 294630. 12:33:26.236 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 0 entries in this iteration. Total remaining: 294630. 12:33:26.288 [main] INFO i.e.i.t.c.InfinispanTenantCleaner - Deleted 0 entries in this iteration. Total remaining: 294630. … {code} Executing the following code (while the deletion process is still running): {code} oc -n enmasse-infra rsh $HOST curl "http://app:test12@$HOST:11222/rest/v2/caches/devices?action=search&query=from+io.enmasse.iot.infinispan.device.DeviceInformation+d&max_results=1" oc -n enmasse-infra rsh $HOST curl "http://app:test12@$HOST:11222/rest/v2/caches/devices?action=search&query=from+io.enmasse.iot.infinispan.device.DeviceInformation+d" oc -n enmasse-infra rsh $HOST curl "http://app:test12@$HOST:11222/rest/v2/caches/devices?action=size" {code} Shows: {code:json} { "total_results" : 237484, "hits" : [ ] } { "total_results" : 237384, "hits" : [ ] } 237184 {code} Stopping the deletion process and re-executing the script above provides the proper information.