[JBoss JIRA] (ISPN-9020) Remote query: disabling indexing for a type with @Indexed(false) still creates empty documents in index
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-9020?page=com.atlassian.jira.plugin.... ]
Adrian Nistor updated ISPN-9020:
--------------------------------
Description: When adding to cache protobuf encoded entries of types that are specifically annotated as @Indexed(false) we would expect that nothing is added to the index. But a document with no fields is added instead, cluttering the index. This does not impact the correctness of query results but it impacts performance because every put operation touches the index even if not required. (was: When adding to cache types that are specifically marked as @Indexed(false) we would expect that nothing is added to the index. A document with no fields is added instead cluttering the index. This does not impact the result of queries.)
> Remote query: disabling indexing for a type with @Indexed(false) still creates empty documents in index
> -------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9020
> URL: https://issues.jboss.org/browse/ISPN-9020
> Project: Infinispan
> Issue Type: Bug
> Components: Remote Querying
> Affects Versions: 7.0.0.Final
> Reporter: Adrian Nistor
> Assignee: Adrian Nistor
> Fix For: 9.3.0.Final
>
>
> When adding to cache protobuf encoded entries of types that are specifically annotated as @Indexed(false) we would expect that nothing is added to the index. But a document with no fields is added instead, cluttering the index. This does not impact the correctness of query results but it impacts performance because every put operation touches the index even if not required.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (ISPN-9003) Clustered maxIdle expiration
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-9003?page=com.atlassian.jira.plugin.... ]
William Burns edited comment on ISPN-9003 at 3/29/18 11:04 AM:
---------------------------------------------------------------
Talking with [~NadirX] we believe the best approach may be a consensus based removal of maxIdle entries. That is it will do the following things
1. Read occurs on node where maxIdle is expired and sends request to primary owner (blocks the read)
2. Primary owner locks the key
3. Primary sends message to all nodes asking if entry is expired due to maxIdle (also updates the updated time on all keys - just in case it isn't expired)
4. Primary receives responses to decide if it is expired
5a. Primary responds to read node and tells them if it was expired
5b. Primary removes from all nodes if it was expired
6. Cluster wide expiration occurs if necessary allowing for listeners to be invoked (including cluster listeners - which we didn't do before)
The above approach gives no overhead for reads when the entry is not present and is not expired. The only overhead is if a node finds that the entry was expired via maxIdle and then must block to confirm if it should be removed. However the chance of this occurring seems a bit low depending on what maxIdle was set to. If the expiration reaper finds the expired entry first it will properly update its timestamps or remove as needed.
Another side effect of this is that maxIdle may be refreshed for all entries that are transferred via state transfer since I don't think the access time is currently replicated when state transfer occurs.
was (Author: william.burns):
Talking with [~NadirX] we believe the best approach may be a consensus based removal of maxIdle entries. That is it will do the following things
1. Read occurs on node where maxIdle is expired and sends request to primary owner (blocks the read)
2. Primary owner locks the key
3. Primary sends message to all nodes asking if entry is expired due to maxIdle (also updates the updated time on all keys - just in case it isn't expired)
4. Primary receives responses and if all say it is expired it will remove the entry
5a. Primary responds to read node and tells them if it was expired
5b. Primary removes from all nodes if it was expired
6. Cluster wide expiration occurs if necessary allowing for listeners to be invoked (including cluster listeners - which we didn't do before)
The above approach gives no overhead for reads when the entry is not present and is not expired. The only overhead is if a node finds that the entry was expired via maxIdle and then must block to confirm if it should be removed. However the chance of this occurring seems a bit low depending on what maxIdle was set to. If the expiration reaper finds the expired entry first it will properly update its timestamps or remove as needed.
Another side effect of this is that maxIdle may be refreshed for all entries that are transferred via state transfer since I don't think the access time is currently replicated when state transfer occurs.
> Clustered maxIdle expiration
> ----------------------------
>
> Key: ISPN-9003
> URL: https://issues.jboss.org/browse/ISPN-9003
> Project: Infinispan
> Issue Type: Enhancement
> Reporter: Tristan Tarrant
> Assignee: William Burns
>
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (ISPN-9003) Clustered maxIdle expiration
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-9003?page=com.atlassian.jira.plugin.... ]
William Burns commented on ISPN-9003:
-------------------------------------
Talking with [~NadirX] we believe the best approach may be a consensus based removal of maxIdle entries. That is it will do the following things
1. Read occurs on node where maxIdle is expired and sends request to primary owner (blocks the read)
2. Primary owner locks the key
3. Primary sends message to all nodes asking if entry is expired due to maxIdle (also updates the updated time on all keys - just in case it isn't expired)
4. Primary receives responses and if all say it is expired it will remove the entry
5a. Primary responds to read node and tells them if it was expired
5b. Primary removes from all nodes if it was expired
6. Cluster wide expiration occurs if necessary allowing for listeners to be invoked (including cluster listeners - which we didn't do before)
The above approach gives no overhead for reads when the entry is not present and is not expired. The only overhead is if a node finds that the entry was expired via maxIdle and then must block to confirm if it should be removed. However the chance of this occurring seems a bit low depending on what maxIdle was set to. If the expiration reaper finds the expired entry first it will properly update its timestamps or remove as needed.
Another side effect of this is that maxIdle may be refreshed for all entries that are transferred via state transfer since I don't think the access time is currently replicated when state transfer occurs.
> Clustered maxIdle expiration
> ----------------------------
>
> Key: ISPN-9003
> URL: https://issues.jboss.org/browse/ISPN-9003
> Project: Infinispan
> Issue Type: Enhancement
> Reporter: Tristan Tarrant
> Assignee: William Burns
>
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (ISPN-6161) Indexes should be created only for annotated message types
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-6161?page=com.atlassian.jira.plugin.... ]
Adrian Nistor commented on ISPN-6161:
-------------------------------------
This change is too big for a 9.x, should wait for 10. Until then I propose to have an option to disable the old behaviour but keep the default the same: this is implemented as ISPN-9021
> Indexes should be created only for annotated message types
> ----------------------------------------------------------
>
> Key: ISPN-6161
> URL: https://issues.jboss.org/browse/ISPN-6161
> Project: Infinispan
> Issue Type: Task
> Components: Remote Querying
> Reporter: Tristan Tarrant
> Assignee: Adrian Nistor
> Fix For: 9.3.0.Final
>
>
> Modify behaviour of indexing for protobuf entities: Current behaviour is: if the message types is not annotated then we index all its fields; if some fields are annotated then index those fields only. New behaviour: index only the annotated fields.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (ISPN-9023) Eviction profile for “memory” type is different from “count” type
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-9023?page=com.atlassian.jira.plugin.... ]
William Burns commented on ISPN-9023:
-------------------------------------
COUNT and MEMORY (when using binary or object storage) both use the same eviction algorithm under the covers (which is controlled by Caffeine). The only mode that doesn't use Caffeine is when using off heap storage in which case we use a simple LRU cache.
>From the sound of it you are running into something regarding weights affecting which entries it evicts. Are you able to write a simple reproducer? If so Ben Manes who wrote Caffeine should be able to help figure out what is going on exactly, as he mentioned on the forum post.
> Eviction profile for “memory” type is different from “count” type
> -----------------------------------------------------------------
>
> Key: ISPN-9023
> URL: https://issues.jboss.org/browse/ISPN-9023
> Project: Infinispan
> Issue Type: Bug
> Components: Eviction
> Affects Versions: 9.1.1.Final
> Environment: * Sun JDK 8 (1.8.0_92)
> * Linux x64
> Reporter: Dmitry Katsubo
> Priority: Minor
>
> I would like to use Infinispan as cache for image binary data (Java type {{byte[]}}). I assume that Infinispan default strategy is LIFO or similar strategy that prefers to keep most recently used/added cache entity.
> In my loadtest scenario I make four calls in the round to retrieve the same entity. It is assumed that first call never hits the cache (because each round a unique entity is requested), but following three always hit the cache. So the expected profile should look like this:
> {code}
> #1: MISS-HIT-HIT-HIT
> #2: MISS-HIT-HIT-HIT
> ...
> {code}
> It works perfectly (exactly as expected above) when I configure Infinispan with COUNT eviction type with some number of entities:
> {code}
> <local-cache name="imagesCache" statistics="true">
> <!-- lifespan="30 min" max-idle="30 min" interval="1 min" -->
> <expiration lifespan="1800000" max-idle="1800000" interval="60000" />
> <memory>
> <binary eviction="COUNT" size="500" />
> </memory>
> </local-cache>
> {code}
> Cache hit ratio based on numbers I capture in loadtest: {{(2952-738)/2952 = 0.75}} and that matches the stats I observe via JMX.
> When I change the number of entities to keep in the memory (<binary eviction="COUNT" size="100" />), hit ratio does not change (as expected).
> After that I have restarted the application with only this small change in cache configuration:
> {code}
> <memory>
> <binary eviction="MEMORY" size="1000000" />
> </memory>
> {code}
> I would expect that Infinispan performance has the same profile, however it turns out that once the given amount of memory is fully allocated, newly added entities don't evict old entities but get immediately evicted. It means that roughly after 100 entities are added, all four requests to cache result cache miss (the ratio is now 0.58 instead of 0.75):
> {code}
> #1: MISS-HIT-HIT-HIT
> #2: MISS-HIT-HIT-HIT
> ...
> #101: MISS-MISS-MISS-MISS
> ...
> {code}
> If I increase the memory, indeed hit ratio comes closer to 0.75 however I would like hit ratio to be the same irrespective the memory size (provided that memory can fit at least few entities).
> Once I configure the passivation to file, memory-based eviction policy starts to work as expected:
> {code}
> <local-cache name="imagesCache" statistics="true">
> <expiration lifespan="1800000" max-idle="1800000" interval="60000" />
> <persistence passivation="true">
> <file-store path="/var/cache/infinispan" purge="true">
> <write-behind thread-pool-size="5" />
> </file-store>
> </persistence>
> <memory>
> <binary eviction="MEMORY" size="1000000" />
> </memory>
> </local-cache>
> {code}
> but I would like to force the profile I need without passivation enabled.
> Additional information is provided [on the forum|https://stackoverflow.com/questions/48420712/eviction-for-memory-ty...].
> Bottomline: If underlying component supports different eviction "modes", please expose this setting via XML so that the user of the library can control the mode.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (ISPN-8691) Infinispan rejects to read cache file bigger than 2147483647 (Integer.MAX_VALUE)
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-8691?page=com.atlassian.jira.plugin.... ]
William Burns commented on ISPN-8691:
-------------------------------------
SingleFileStore currently shouldn't be able to write an entry larger than Integer.MAX_VALUE. My guess is that we are just missing a check on the write side to error sooner rather than later. Unfortunately our SPI for CacheWriter uses byte[] which means we will be bound to Integer.MAX_VALUE until that is fixed :(
We should make sure that if a user tries to persist an entry larger than MAX_VALUE to throw an error though, which this issue can encompass.
> Infinispan rejects to read cache file bigger than 2147483647 (Integer.MAX_VALUE)
> --------------------------------------------------------------------------------
>
> Key: ISPN-8691
> URL: https://issues.jboss.org/browse/ISPN-8691
> Project: Infinispan
> Issue Type: Enhancement
> Components: Loaders and Stores
> Affects Versions: 9.1.1.Final
> Reporter: Dmitry Katsubo
> Priority: Minor
>
> In my scenario the cache file size created by {{SingleFileStore}} is 3.054.196.342 bytes. When this file is tried to be loaded, it fails with the following exception:
> {code}
> Caused by: org.infinispan.persistence.spi.PersistenceException: ISPN000279: Failed to read stored entries from file. Error in file /work/search-service-layer_data/infinispan/cache_test_dk83146/markupCache.dat at offset 4
> at org.infinispan.persistence.file.SingleFileStore.rebuildIndex(SingleFileStore.java:182)
> at org.infinispan.persistence.file.SingleFileStore.start(SingleFileStore.java:127)
> ... 155 more
> {code}
> Cache file content:
> {code}
> 0000000000: 46 43 53 31 80 B1 89 47 │ 00 00 00 00 00 00 00 00 FCS1?+%G
> 0000000010: 00 00 00 00 FF FF FF FF │ FF FF FF FF 02 15 4E 06 yyyyyyyy☻§N♠
> 0000000020: 05 03 04 09 00 00 00 2F │ 6F 72 67 2E 73 70 72 69 ♣♥♦○ /org.spri
> 0000000030: 6E 67 66 72 61 6D 65 77 │ 6F 72 6B 2E 63 61 63 68 ngframework.cach
> 0000000040: 65 2E 69 6E 74 65 72 63 │ 65 70 74 6F 72 2E 53 69 e.interceptor.Si
> 0000000050: 6D 70 6C 65 4B 65 79 4C │ 0A 57 03 6B 6D 93 D8 00 mpleKeyL◙W♥km"O
> 0000000060: 00 00 02 00 00 00 08 68 │ 61 73 68 43 6F 64 65 23 ☻ ◘hashCode#
> 0000000070: 00 00 00 00 06 70 61 72 │ 61 6D 73 16 00 16 15 E6 ♠params▬ ▬§?
> {code}
> The problem is that integer value 0x80B18947 is treated as signed integer in line {{SingleFileStore:181}}, hence in expression
> {code}
> if (fe.size < KEY_POS + fe.keyLen + fe.dataLen + fe.metadataLen) {
> throw log.errorReadingFileStore(file.getPath(), filePos);
> }
> {code}
> {{fe.size}} is negative and equal to -2135848633.
> I have tried to configure the persistence storage so that it gets purged on start:
> {code}
> <persistence passivation="true">
> <file-store path="/var/cache/infinispan" purge="true">
> <write-behind thread-pool-size="5" />
> </file-store>
> </persistence>
> {code}
> however this does not help as storage is first read and then purged (see also ISPN-7186).
> It is expected that {{SingleFileStore}} either does not allow to write such big entries to the cache, or handles them correctly.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years