[JBoss JIRA] (ISPN-8385) Off-heap container entries contain addional bytes
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-8385?page=com.atlassian.jira.plugin.... ]
Work on ISPN-8385 started by William Burns.
-------------------------------------------
> Off-heap container entries contain addional bytes
> -------------------------------------------------
>
> Key: ISPN-8385
> URL: https://issues.jboss.org/browse/ISPN-8385
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.1.1.Final, 9.2.0.Alpha1
> Reporter: Vojtech Juranek
> Assignee: William Burns
>
> Off-heap container entries contain some additional bytes (ASCII 1,2,1,4), which probably shouldn't be there. Besides additional consumption of memory, it can cause problem as entry put into the cache with cache API is not accessible via container API (which is available to the use), i.e. something like this will fail:
> {noformat}
> cache.put(key, value);
> container = AbstractDelegatingCache.unwrapCache(cache).getAdvancedCache().getDataContainer();
> InternalCacheEntry entry = container.get(key);
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 2 months
[JBoss JIRA] (ISPN-8385) Off-heap container entries contain addional bytes
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-8385?page=com.atlassian.jira.plugin.... ]
William Burns updated ISPN-8385:
--------------------------------
Status: Open (was: New)
> Off-heap container entries contain addional bytes
> -------------------------------------------------
>
> Key: ISPN-8385
> URL: https://issues.jboss.org/browse/ISPN-8385
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.1.1.Final, 9.2.0.Alpha1
> Reporter: Vojtech Juranek
> Assignee: William Burns
>
> Off-heap container entries contain some additional bytes (ASCII 1,2,1,4), which probably shouldn't be there. Besides additional consumption of memory, it can cause problem as entry put into the cache with cache API is not accessible via container API (which is available to the use), i.e. something like this will fail:
> {noformat}
> cache.put(key, value);
> container = AbstractDelegatingCache.unwrapCache(cache).getAdvancedCache().getDataContainer();
> InternalCacheEntry entry = container.get(key);
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 2 months
[JBoss JIRA] (ISPN-8478) Reduce build memory usage
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-8478?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-8478:
------------------------------------
Status: Open (was: New)
> Reduce build memory usage
> -------------------------
>
> Key: ISPN-8478
> URL: https://issues.jboss.org/browse/ISPN-8478
> Project: Infinispan
> Issue Type: Task
> Components: Build process
> Affects Versions: 9.2.0.Alpha2
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 9.2.0.Beta1
>
>
> The CI Maven process runs without a max heap parameter, so it can use up to 1/4 of agent RAM. It should run with a specific limit and so should all the other JVMs it spawns: the surefire fork, Karaf containers, WildFly servers etc.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 2 months
[JBoss JIRA] (ISPN-8478) Reduce build memory usage
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-8478?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-8478:
------------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> Reduce build memory usage
> -------------------------
>
> Key: ISPN-8478
> URL: https://issues.jboss.org/browse/ISPN-8478
> Project: Infinispan
> Issue Type: Task
> Components: Build process
> Affects Versions: 9.2.0.Alpha2
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 9.2.0.Beta1
>
>
> The CI Maven process runs without a max heap parameter, so it can use up to 1/4 of agent RAM. It should run with a specific limit and so should all the other JVMs it spawns: the surefire fork, Karaf containers, WildFly servers etc.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 2 months
[JBoss JIRA] (ISPN-8385) Off-heap container entries contain addional bytes
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-8385?page=com.atlassian.jira.plugin.... ]
William Burns commented on ISPN-8385:
-------------------------------------
Talking with [~gustavonalle] the user can do `cache.getAdvancedCache().getKeyDataConversion().toStorage(key)` to get the underlying key as the storage requires. I will add some documentation to the `DataContainer` to say this shouldn't be used directly and the user should use the above method to convert if the underlying cache isn't stored as is.
> Off-heap container entries contain addional bytes
> -------------------------------------------------
>
> Key: ISPN-8385
> URL: https://issues.jboss.org/browse/ISPN-8385
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.1.1.Final, 9.2.0.Alpha1
> Reporter: Vojtech Juranek
> Assignee: William Burns
>
> Off-heap container entries contain some additional bytes (ASCII 1,2,1,4), which probably shouldn't be there. Besides additional consumption of memory, it can cause problem as entry put into the cache with cache API is not accessible via container API (which is available to the use), i.e. something like this will fail:
> {noformat}
> cache.put(key, value);
> container = AbstractDelegatingCache.unwrapCache(cache).getAdvancedCache().getDataContainer();
> InternalCacheEntry entry = container.get(key);
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 2 months
[JBoss JIRA] (ISPN-8385) Off-heap container entries contain addional bytes
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-8385?page=com.atlassian.jira.plugin.... ]
William Burns edited comment on ISPN-8385 at 11/8/17 11:19 AM:
---------------------------------------------------------------
That is exactly what it is. Also to note this is only an issue with embedded off heap, server is perfectly fine since the serialization occurs on the client.
Looking at this there are 2 possible solutions:
# We could wrap the byte[] differently, thus preventing the GlobalMarshaller from occurring. Unfortunately we would then have to store an extra byte with the key and value to signify that it is a byte[] and not a serialized object. So this still has the same issue as reported. This also would then add an additional byte overhead to non byte[] entries, which when running embedded should be the most commonly used entries by far.
# If we pay attention to the upcoming cache type value and if it was _application/octet-stream_ we would know the values will always have to be byte[]. This isn't doable currently though and would only work under specific configuration settings.
# We could wrap the data container retrieved from the cache and automatically perform the encoding and wrapping for the user. This is probably the cleanest method, but I am not sure if we want to do this or not. [~gustavonalle] have an opinion? Virtually all of the components you can retrieve don't do this currently. Do we then have to go and change them all? Sounds nasty to me.
Therefore I am suggesting that this is not a bug and is just intended. If the user wants to use the data container directly they should retrieve the encoder and wrapper from the cache and do the operation themselves. We can document the data container though to highlight this to the user.
was (Author: william.burns):
That is exactly what it is. Also to note this is only an issue with embedded off heap, server is perfectly fine since the serialization occurs on the client.
Looking at this there are 2 possible solutions:
# We could wrap the byte[] differently, thus preventing the GlobalMarshaller from occurring. Unfortunately we would then have to store an extra byte with the key and value to signify that it is a byte[] and not a serialized object. So this still has the same issue as reported. This also would then add an additional byte overhead to non byte[] entries, which when running embedded should be the most commonly used entries by far.
# If we pay attention to the upcoming cache type value and if it was _application/octet-stream_ we would know the values will always have to be byte[]. This isn't doable currently though and would only work under specific configuration settings.
# We could wrap the data container retrieved from the cache and automatically perform the encoding and wrapping for the user. This is probably the cleanest method, but I am not sure if we want to do this or not. [~gustavonalle] have an opinion? Virtually all of the components you can retrieve don't do this currently. Do we then have to go and change them all? Sounds nasty to me.
Therefore I am suggesting that this is not a bug and is just intended. If the user wants to use the data container directly they should retrieve the encoder and wrapper from the cache and do the operation themselves.
> Off-heap container entries contain addional bytes
> -------------------------------------------------
>
> Key: ISPN-8385
> URL: https://issues.jboss.org/browse/ISPN-8385
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.1.1.Final, 9.2.0.Alpha1
> Reporter: Vojtech Juranek
> Assignee: William Burns
>
> Off-heap container entries contain some additional bytes (ASCII 1,2,1,4), which probably shouldn't be there. Besides additional consumption of memory, it can cause problem as entry put into the cache with cache API is not accessible via container API (which is available to the use), i.e. something like this will fail:
> {noformat}
> cache.put(key, value);
> container = AbstractDelegatingCache.unwrapCache(cache).getAdvancedCache().getDataContainer();
> InternalCacheEntry entry = container.get(key);
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 2 months
[JBoss JIRA] (ISPN-8385) Off-heap container entries contain addional bytes
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-8385?page=com.atlassian.jira.plugin.... ]
William Burns edited comment on ISPN-8385 at 11/8/17 11:18 AM:
---------------------------------------------------------------
That is exactly what it is. Also to note this is only an issue with embedded off heap, server is perfectly fine since the serialization occurs on the client.
Looking at this there are 2 possible solutions:
# We could wrap the byte[] differently, thus preventing the GlobalMarshaller from occurring. Unfortunately we would then have to store an extra byte with the key and value to signify that it is a byte[] and not a serialized object. So this still has the same issue as reported. This also would then add an additional byte overhead to non byte[] entries, which when running embedded should be the most commonly used entries by far.
# If we pay attention to the upcoming cache type value and if it was _application/octet-stream_ we would know the values will always have to be byte[]. This isn't doable currently though and would only work under specific configuration settings.
# We could wrap the data container retrieved from the cache and automatically perform the encoding and wrapping for the user. This is probably the cleanest method, but I am not sure if we want to do this or not. [~gustavonalle] have an opinion? Virtually all of the components you can retrieve don't do this currently. Do we then have to go and change them all? Sounds nasty to me.
Therefore I am suggesting that this is not a bug and is just intended. If the user wants to use the data container directly they should retrieve the encoder and wrapper from the cache and do the operation themselves.
was (Author: william.burns):
That is exactly what it is. Also to note this is only an issue with embedded off heap, server is perfectly fine since the serialization occurs on the client.
Looking at this there are 2 possible solutions:
# We could wrap the byte[] differently, thus preventing the GlobalMarshaller from occurring. Unfortunately we would then have to store an extra byte with the key and value to signify that it is a byte[] and not a serialized object. So this still has the same issue as reported. This also would then add an additional byte overhead to non byte[] entries, which when running embedded should be the most commonly used entries by far.
# If we pay attention to the upcoming cache type value and if it was _application/octet-stream_ we would know the values will always have to be byte[]. This isn't doable currently though and would only work under specific configuration settings.
# We could wrap the data container retrieved from the cache and automatically perform the encoding and wrapping for the user. This is probably the cleanest method, but I am not sure if we want to do this or not. [~gustavonalle] have an opinion? Virtually all of the components you can retrieve don't do this currently. Do we then have to go and change them all? Sounds nasty to me.
Therefore I am saying that this
> Off-heap container entries contain addional bytes
> -------------------------------------------------
>
> Key: ISPN-8385
> URL: https://issues.jboss.org/browse/ISPN-8385
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.1.1.Final, 9.2.0.Alpha1
> Reporter: Vojtech Juranek
> Assignee: William Burns
>
> Off-heap container entries contain some additional bytes (ASCII 1,2,1,4), which probably shouldn't be there. Besides additional consumption of memory, it can cause problem as entry put into the cache with cache API is not accessible via container API (which is available to the use), i.e. something like this will fail:
> {noformat}
> cache.put(key, value);
> container = AbstractDelegatingCache.unwrapCache(cache).getAdvancedCache().getDataContainer();
> InternalCacheEntry entry = container.get(key);
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 2 months
[JBoss JIRA] (ISPN-8385) Off-heap container entries contain addional bytes
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-8385?page=com.atlassian.jira.plugin.... ]
William Burns commented on ISPN-8385:
-------------------------------------
That is exactly what it is. Also to note this is only an issue with embedded off heap, server is perfectly fine since the serialization occurs on the client.
Looking at this there are 2 possible solutions:
# We could wrap the byte[] differently, thus preventing the GlobalMarshaller from occurring. Unfortunately we would then have to store an extra byte with the key and value to signify that it is a byte[] and not a serialized object. So this still has the same issue as reported. This also would then add an additional byte overhead to non byte[] entries, which when running embedded should be the most commonly used entries by far.
# If we pay attention to the upcoming cache type value and if it was _application/octet-stream_ we would know the values will always have to be byte[]. This isn't doable currently though and would only work under specific configuration settings.
# We could wrap the data container retrieved from the cache and automatically perform the encoding and wrapping for the user. This is probably the cleanest method, but I am not sure if we want to do this or not. [~gustavonalle] have an opinion? Virtually all of the components you can retrieve don't do this currently. Do we then have to go and change them all? Sounds nasty to me.
Therefore I am saying that this
> Off-heap container entries contain addional bytes
> -------------------------------------------------
>
> Key: ISPN-8385
> URL: https://issues.jboss.org/browse/ISPN-8385
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.1.1.Final, 9.2.0.Alpha1
> Reporter: Vojtech Juranek
> Assignee: William Burns
>
> Off-heap container entries contain some additional bytes (ASCII 1,2,1,4), which probably shouldn't be there. Besides additional consumption of memory, it can cause problem as entry put into the cache with cache API is not accessible via container API (which is available to the use), i.e. something like this will fail:
> {noformat}
> cache.put(key, value);
> container = AbstractDelegatingCache.unwrapCache(cache).getAdvancedCache().getDataContainer();
> InternalCacheEntry entry = container.get(key);
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 2 months
[JBoss JIRA] (ISPN-8396) Add interceptor preventing going out of memory
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-8396?page=com.atlassian.jira.plugin.... ]
William Burns commented on ISPN-8396:
-------------------------------------
So here is my initial thoughts of how to do this in the core, which then could be used further up the stack
We would add a new "eviction" type currently we have (COUNT and MEMORY) - I can't think of a good name, maybe MEMORY-EXCEPTION. This would then not actually add a new interceptor to the stack towards the top that only takes a WrappedBytes and checks the length and the OffHeapMemoryAllocator to see if it can allocate that much memory. If not it would throw an exception. This allows off heap to use the non bounded data container which is substantially faster than the bounded one. The only drawback is this wouldn't work with on heap.
We could add on heap (OBJECT storage would still not be supported), but it would require an additional wrapper around the DefaultDataContainer to count the byte[] coming in and out when using BINARY.
So the only issue I found with this approach is that nodes can vary on whether they think they can insert values or not. So if a node was to write a given entry, a backup might not have room but the primary does. This could cause an issue with the write in nontx mode as it could have been updated on some of the nodes. Maybe we always write if the primary says we should?
> Add interceptor preventing going out of memory
> ----------------------------------------------
>
> Key: ISPN-8396
> URL: https://issues.jboss.org/browse/ISPN-8396
> Project: Infinispan
> Issue Type: Feature Request
> Components: Cloud Integrations, Core
> Reporter: Sebastian Łaskawiec
> Assignee: William Burns
>
> We need an interceptor which will calculate the amount of required memory for PUT and report an error if that put will cause going out of memory.
> Note that this is strictly connected to eviction mechanism (we might want to evict some entries on write)
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 2 months