[ https://issues.jboss.org/browse/MODCLUSTER-288?page=com.atlassian.jira.pl... ] Michal Babacek updated MODCLUSTER-288: -------------------------------------- Summary: SystemMemoryUsageLoadMetric is not correct on Linux/Unix (was: SystemMemoryUsageLoadMetric is of no use to anyone) Environment: Linux/Unix operating systems with standard memory management Description: I am convinced that the [SystemMemoryUsageLoadMetric.java|https://github.com/modcluster/mod_cluste...] is of no use to anyone. I have tested it by allocating some memory outside the JVM control with [NativeMemoryStress.c|https://github.com/Karm/clusterbench/blob/system-mem...]. The results from Mod_cluster are such that the system is almost fully loaded all the time despite the fact that I work with 8GB RAM box with almost nothing running on it. This [picture|http://hudson.qa.jboss.com/hudson/view/mod_cluster-QE/view/mod_cl...] displays time on X axis and (100-(Load from Mod_cluster)) on Y axis. As you may observe, all 4 nodes appear to be under full load all the time and there is a spike on each of them. This spike represents a period of time when the aforementioned [NativeMemoryStress.c|https://github.com/Karm/clusterbench/blob/system-mem...] was called with the following setting: {noformat} megabytes 500; milliseconds 60000; {noformat} What had happened is that kernel has cleared some cache and garbage data from the RAM in order to accommodate new demands, so by the time I freed the memory, there was all of sudden actually more free RAM then before. The actual problem with the current [SystemMemoryUsageLoadMetric.java|https://github.com/modcluster/mod_cluste...] implementation using [OperatingSystemMXBean|http://docs.oracle.com/javase/6/docs/jre/api/manage...] is that this code: {code:lang=java} +++ public static final String FREE_MEMORY = "FreePhysicalMemorySize"; public static final String TOTAL_MEMORY = "TotalPhysicalMemorySize"; +++ double free = ((Number) this.server.getAttribute(name, FREE_MEMORY)).doubleValue(); double total = ((Number) this.server.getAttribute(name, TOTAL_MEMORY)).doubleValue(); return (total - free) / total; +++ {code} Is somewhat similar to this (on 8GB RAM machine with almost nothing running on at the moment): {noformat} [karm@killerbox karm]$ free -m total used free shared buffers cached Mem: 7933 6755 1177 0 44 141 -/+ buffers/cache: 6569 1363 Swap: 16383 58 16325 {noformat} Obviously, this does not mean that I could get only 1177MB of RAM if I want to. The "free" actually means "wasted". The majority of RAM is being used as cache. If I ask for e.g. 2GB, I am gonna get them, because kernel will simply sacrifice some cache, clears some garbage from RAM and free up space so as to accommodate my demands. So, according to my opinion, on a sane, well-behaving caching operating system, the [SystemMemoryUsageLoadMetric.java|https://github.com/modcluster/mod_cluste...] will always show ~ 98% RAM usage. Due to this fact, this metric is unlikely to be of some use to anyone on Linux/UNIX. What do you think? was: I am convinced that the [SystemMemoryUsageLoadMetric.java|https://github.com/modcluster/mod_cluste...] is of no use to anyone. I have tested it by allocating some memory outside the JVM control with [NativeMemoryStress.c|https://github.com/Karm/clusterbench/blob/system-mem...]. The results from Mod_cluster are such that the system is almost fully loaded all the time despite the fact that I work with 8GB RAM box with almost nothing running on it. This [picture|http://hudson.qa.jboss.com/hudson/view/mod_cluster-QE/view/mod_cl...] displays time on X axis and (100-(Load from Mod_cluster)) on Y axis. As you may observe, all 4 nodes appear to be under full load all the time and there is a spike on each of them. This spike represents a period of time when the aforementioned [NativeMemoryStress.c|https://github.com/Karm/clusterbench/blob/system-mem...] was called with the following setting: {noformat} megabytes 500; milliseconds 60000; {noformat} What had happened is that kernel has cleared some cache and garbage data from the RAM in order to accommodate new demands, so by the time I freed the memory, there was all of sudden actually more free RAM then before. The actual problem with the current [SystemMemoryUsageLoadMetric.java|https://github.com/modcluster/mod_cluste...] implementation using [OperatingSystemMXBean|http://docs.oracle.com/javase/6/docs/jre/api/manage...] is that this code: {code:lang=java} +++ public static final String FREE_MEMORY = "FreePhysicalMemorySize"; public static final String TOTAL_MEMORY = "TotalPhysicalMemorySize"; +++ double free = ((Number) this.server.getAttribute(name, FREE_MEMORY)).doubleValue(); double total = ((Number) this.server.getAttribute(name, TOTAL_MEMORY)).doubleValue(); return (total - free) / total; +++ {code} Is somewhat similar to this (on 8GB RAM machine with almost nothing running on at the moment): {noformat} [karm@killerbox karm]$ free -m total used free shared buffers cached Mem: 7933 6755 1177 0 44 141 -/+ buffers/cache: 6569 1363 Swap: 16383 58 16325 {noformat} Obviously, this does not mean that I could get only 1177MB of RAM if I want to. The "free" actually means "wasted". The majority of RAM is being used as cache. If I ask for e.g. 2GB, I am gonna get them, because kernel will simply sacrifice some cache, clears some garbage from RAM and free up space so as to accommodate my demands. So, according to my opinion, on a sane, well-behaving caching operating system, the [SystemMemoryUsageLoadMetric.java|https://github.com/modcluster/mod_cluste...] will always show ~ 98% RAM usage. Due to this fact, this metric is unlikely to be of some use to anyone. What do you think? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/MODCLUSTER-288?page=com.atlassian.jira.pl... ] Michal Babacek reopened MODCLUSTER-288: --------------------------------------- @[~pferraro], I am sorry that my point is not clear, I will try to rephrase and compress the description of this JIRA: I understand the possible use cases, as you have mentioned, e.g. starting external processes or any other "extra-jvm work". The problem is, that any Linux/UNIX system that tries to utilize as much RAM as possible as e.g. a cache will always (no matter how many external non JVM processes you start) report full load with this SystemMemoryUsageLoadMetric implementation. Therefore, your statement is not right: If I have an application that does significant extra-jvm work and I want to monitor system memory, the current SystemMemoryUsageLoadMetric implementation is not gonna help me. In my humble opinion, we have to modify SystemMemoryUsageLoadMetric in such a a way, that *buffers/cache* value is actually *subtracted* from the *free* number. Thank you for spending some time commenting on this JIRA. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/MODCLUSTER-288?page=com.atlassian.jira.pl... ] Paul Ferraro commented on MODCLUSTER-288: ----------------------------------------- Ah - ok. That makes sense. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/MODCLUSTER-288?page=com.atlassian.jira.pl... ] Michal Babacek commented on MODCLUSTER-288: ------------------------------------------- @[~pferraro] I've done some significant googling & JavaDoc reading and I am convinced it is not possible to use Java API for obtaining cache/buffers number in a platform independent way. For instance, the guys from [SIGAR|http://support.hyperic.com/display/SIGAR/Home] use JNI wrappers and native system calls fro this kind of job as well... I suggest to write down in the docs that this metric does not work and suggest people to modify it with their own piece of code loaded as a native library implementing JNI native method... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/MODCLUSTER-288?page=com.atlassian.jira.pl... ] Jean-Frederic Clere commented on MODCLUSTER-288: ------------------------------------------------ +1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira