[infinispan-issues] [JBoss JIRA] (ISPN-8540) Investigate off-heap and eviction memory usage
Sebastian Łaskawiec created ISPN-8540:
-----------------------------------------
Summary: Investigate off-heap and eviction memory usage
Key: ISPN-8540
URL: https://issues.jboss.org/browse/ISPN-8540
Project: Infinispan
Issue Type: Task
Components: Cloud Integrations
Reporter: Sebastian Łaskawiec
Assignee: William Burns
With the [latest
changes|https://github.com/jboss-container-images/jboss-dataservices-imag...],
online services calibrate eviction as follows:
* Set {{Xmx}} to 100 MB
* Set Additional JVM overhead to 200
* Calculate {{-XX:MaxRAM}} as sum of the above
* Add 30% of the container capacity overhead
* Calculate eviction size as Container size - Xmx - Additional JVM overhead - 30% of
container capacity.
This renders the following configuration (this is just an example):
{code}
<subsystem xmlns="urn:infinispan:server:core:8.5">
<cache-container name="clustered"
default-cache="default" statistics="true">
<transport lock-timeout="60000"/>
<global-state/>
<async-operations-thread-pool min-threads="1"/>
<transport-thread-pool min-threads="1"/>
<distributed-cache-configuration name="caching-service"
owners="1">
<memory>
<off-heap size="65011712"
eviction="MEMORY"/>
</memory>
<partition-handling when-split="ALLOW_READ_WRITES"
merge-policy="REMOVE_ALL"/>
</distributed-cache-configuration>
<distributed-cache name="default"
configuration="caching-service"/>
<distributed-cache name="memcachedCache"
configuration="caching-service"/>
<distributed-cache-configuration name="default"/>
<distributed-cache-configuration name="memcachedCache"/>
</cache-container>
</subsystem>
{code}
The configuration above doesn't seem to work very well. The results spreadsheet might
be found
[here|https://docs.google.com/spreadsheets/d/1HbOhwze_L4EQi06R1sq9brK5zlom...].
In particular when calibrated to 512 MB container it leaves 22 MB of free space in the
container, similar configuration for 430 MB container leaves 33 MB. This trend goes even
worse when it comes to smaller containers.
What's even more surprising, the value calculated for total overhead (Xmx + JVM
Overhead + 30% of container capacity) doesn't seem to be linear. It's quadratic
(sort of). This results in very low amount of data we can actually store in the grid.
Those figures seem quite low and before moving investing more time into calibrating this
thing I would like to make sure there is no bug there and eviction works as expected.
This can also be reproduced using local standalone Infinispan/JDG installation. All you
need to do is to calculate values manually and figure out Xmx/MaxRAM parameters (full list
of parameters used by the container might be found
[here|https://gist.github.com/slaskawi/5c270d1ba3ec3e1752faf18bfd00a70c#fi...].
After booting up the server, run 1K key/val load against it (JMH works quite nicely). On
the second terminal window, check the {{pmem -x}} output. The second column is RSS, which
is quite similar to what CGroups see. Depending what you're testing, going beyond
512MB/1G at any point of time would result in killing a container in real world.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)