[JBoss JIRA] (ISPN-8550) Try to estimate malloc overhead and add to memory based eviction
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-8550?page=com.atlassian.jira.plugin.... ]
William Burns edited comment on ISPN-8550 at 11/27/17 3:41 PM:
---------------------------------------------------------------
I was using valgrind like the following:
{quote}
valgrind --tool=massif --time-unit=ms --detailed-freq=10000 <cmd>
{quote}
I increased detailed-freq really high to try to remove the excess.
I took the memory usage after it was primed with inserts and then subtracted that from the peak memory usage. Looking at your spreadsheet I don't see excess JVM and others subtracted from the size in your calculation. In my tests it was about 7M of memory used by that, which can throw off your other numbers, or maybe I am not reading your spreadsheet correctly.
Also I am not quite sure about the baseline value of 72M, seems quite high.
was (Author: william.burns):
I was using valgrind like the following:
{quote}
valgrind --tool=massif --time-unit=ms --detailed-freq=10000 <cmd>
{quote}
I increased detailed-freq really high to try to remove the excess.
I took the memory usage after it was primed with inserts and then subtracted that from the peak memory usage. Looking at your spreadsheet I don't see excess JVM and others subtracted from the size in your calculation. In my tests it was about 7M of memory used by that, which can throw off your other numbers, or maybe I am not reading your spreadsheet correctly.
> Try to estimate malloc overhead and add to memory based eviction
> ----------------------------------------------------------------
>
> Key: ISPN-8550
> URL: https://issues.jboss.org/browse/ISPN-8550
> Project: Infinispan
> Issue Type: Sub-task
> Reporter: William Burns
> Assignee: William Burns
> Fix For: 9.2.0.Beta2, 9.1.4.Final
>
> Attachments: allocs.ods
>
>
> We should try to also estimate malloc overhead. We could do something like Dan mentioned at https://github.com/infinispan/infinispan/pull/5590#pullrequestreview-7805...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 5 months
[JBoss JIRA] (ISPN-8568) ClassNotFoundException with Compat mode and custom Pojos
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-8568?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes commented on ISPN-8568:
-----------------------------------------
I suppose the newly introduced AggredatedClassLoader need to somehow be passed on to the GenericJbossMarshaller. I will give it a try and send a PR
> ClassNotFoundException with Compat mode and custom Pojos
> --------------------------------------------------------
>
> Key: ISPN-8568
> URL: https://issues.jboss.org/browse/ISPN-8568
> Project: Infinispan
> Issue Type: Bug
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
>
> The cache is configured with:
> {code:xml}
> <cache-container name="local">
> <modules>
> <module name="deployment.myTask.jar"/>
> </modules>
> <local-cache name="compat">
> <compatibility enabled="true"/>
> </local-cache>
> </cache-container>
> {code}
> The {{myTask.jar}} contains Entity.class that is the Pojo to be stored in the cache.
> When doing a
> {code:java}
> remoteCache.put(1, new Entity());
> {code}
> The error is
> {noformat}
> 18:13:42,142 ERROR (main) [TestSuiteProgress] Test failed: LocalServerTestCompatModeIT.shouldRunPriceTask
> org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=9 returned
> server error (status=0x85): java.lang.ClassNotFoundException: org.infinispan.server.test.Entity from [Module "org.infinispan.commons:main" from
> local module loader @4ae82894 (finder: local module finder @543788f3 (roots: /home/gfernandes/github/infinispan/server/integration/testsuite/target/server/node1/modules,/home/gfernandes/github/infinispan/server/integration/testsuite/target/server/node1/modules/system/layers/base))]
> {noformat}
> I noticed that in this case, the Marshaller used is {{GenericJBossMarshaller}} who lives in infinispan-commons, and when asked to de-serialize the pojo in byte[] form, the classloader used is ModuleClassLoader for Module "org.infinispan.commons:main" that does not have visibility of the deployed jar's classes.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 5 months
[JBoss JIRA] (ISPN-8568) ClassNotFoundException with Compat mode and custom Pojos
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-8568?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes edited comment on ISPN-8568 at 11/27/17 2:46 PM:
-------------------------------------------------------------------
I suppose the newly introduced AggredatedClassLoader needs to somehow be passed on to the GenericJbossMarshaller. I will give it a try and send a PR
was (Author: gustavonalle):
I suppose the newly introduced AggredatedClassLoader need to somehow be passed on to the GenericJbossMarshaller. I will give it a try and send a PR
> ClassNotFoundException with Compat mode and custom Pojos
> --------------------------------------------------------
>
> Key: ISPN-8568
> URL: https://issues.jboss.org/browse/ISPN-8568
> Project: Infinispan
> Issue Type: Bug
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
>
> The cache is configured with:
> {code:xml}
> <cache-container name="local">
> <modules>
> <module name="deployment.myTask.jar"/>
> </modules>
> <local-cache name="compat">
> <compatibility enabled="true"/>
> </local-cache>
> </cache-container>
> {code}
> The {{myTask.jar}} contains Entity.class that is the Pojo to be stored in the cache.
> When doing a
> {code:java}
> remoteCache.put(1, new Entity());
> {code}
> The error is
> {noformat}
> 18:13:42,142 ERROR (main) [TestSuiteProgress] Test failed: LocalServerTestCompatModeIT.shouldRunPriceTask
> org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=9 returned
> server error (status=0x85): java.lang.ClassNotFoundException: org.infinispan.server.test.Entity from [Module "org.infinispan.commons:main" from
> local module loader @4ae82894 (finder: local module finder @543788f3 (roots: /home/gfernandes/github/infinispan/server/integration/testsuite/target/server/node1/modules,/home/gfernandes/github/infinispan/server/integration/testsuite/target/server/node1/modules/system/layers/base))]
> {noformat}
> I noticed that in this case, the Marshaller used is {{GenericJBossMarshaller}} who lives in infinispan-commons, and when asked to de-serialize the pojo in byte[] form, the classloader used is ModuleClassLoader for Module "org.infinispan.commons:main" that does not have visibility of the deployed jar's classes.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 5 months
[JBoss JIRA] (ISPN-8568) ClassNotFoundException with Compat mode and custom Pojos
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-8568?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-8568:
------------------------------------
Status: Open (was: New)
> ClassNotFoundException with Compat mode and custom Pojos
> --------------------------------------------------------
>
> Key: ISPN-8568
> URL: https://issues.jboss.org/browse/ISPN-8568
> Project: Infinispan
> Issue Type: Bug
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
>
> The cache is configured with:
> {code:xml}
> <cache-container name="local">
> <modules>
> <module name="deployment.myTask.jar"/>
> </modules>
> <local-cache name="compat">
> <compatibility enabled="true"/>
> </local-cache>
> </cache-container>
> {code}
> The {{myTask.jar}} contains Entity.class that is the Pojo to be stored in the cache.
> When doing a
> {code:java}
> remoteCache.put(1, new Entity());
> {code}
> The error is
> {noformat}
> 18:13:42,142 ERROR (main) [TestSuiteProgress] Test failed: LocalServerTestCompatModeIT.shouldRunPriceTask
> org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=9 returned
> server error (status=0x85): java.lang.ClassNotFoundException: org.infinispan.server.test.Entity from [Module "org.infinispan.commons:main" from
> local module loader @4ae82894 (finder: local module finder @543788f3 (roots: /home/gfernandes/github/infinispan/server/integration/testsuite/target/server/node1/modules,/home/gfernandes/github/infinispan/server/integration/testsuite/target/server/node1/modules/system/layers/base))]
> {noformat}
> I noticed that in this case, the Marshaller used is {{GenericJBossMarshaller}} who lives in infinispan-commons, and when asked to de-serialize the pojo in byte[] form, the classloader used is ModuleClassLoader for Module "org.infinispan.commons:main" that does not have visibility of the deployed jar's classes.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 5 months
[JBoss JIRA] (ISPN-8568) ClassNotFoundException with Compat mode and custom Pojos
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-8568?page=com.atlassian.jira.plugin.... ]
Work on ISPN-8568 started by Gustavo Fernandes.
-----------------------------------------------
> ClassNotFoundException with Compat mode and custom Pojos
> --------------------------------------------------------
>
> Key: ISPN-8568
> URL: https://issues.jboss.org/browse/ISPN-8568
> Project: Infinispan
> Issue Type: Bug
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
>
> The cache is configured with:
> {code:xml}
> <cache-container name="local">
> <modules>
> <module name="deployment.myTask.jar"/>
> </modules>
> <local-cache name="compat">
> <compatibility enabled="true"/>
> </local-cache>
> </cache-container>
> {code}
> The {{myTask.jar}} contains Entity.class that is the Pojo to be stored in the cache.
> When doing a
> {code:java}
> remoteCache.put(1, new Entity());
> {code}
> The error is
> {noformat}
> 18:13:42,142 ERROR (main) [TestSuiteProgress] Test failed: LocalServerTestCompatModeIT.shouldRunPriceTask
> org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=9 returned
> server error (status=0x85): java.lang.ClassNotFoundException: org.infinispan.server.test.Entity from [Module "org.infinispan.commons:main" from
> local module loader @4ae82894 (finder: local module finder @543788f3 (roots: /home/gfernandes/github/infinispan/server/integration/testsuite/target/server/node1/modules,/home/gfernandes/github/infinispan/server/integration/testsuite/target/server/node1/modules/system/layers/base))]
> {noformat}
> I noticed that in this case, the Marshaller used is {{GenericJBossMarshaller}} who lives in infinispan-commons, and when asked to de-serialize the pojo in byte[] form, the classloader used is ModuleClassLoader for Module "org.infinispan.commons:main" that does not have visibility of the deployed jar's classes.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 5 months
[JBoss JIRA] (ISPN-8568) ClassNotFoundException with Compat mode and custom Pojos
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-8568?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes reassigned ISPN-8568:
---------------------------------------
Assignee: Gustavo Fernandes
> ClassNotFoundException with Compat mode and custom Pojos
> --------------------------------------------------------
>
> Key: ISPN-8568
> URL: https://issues.jboss.org/browse/ISPN-8568
> Project: Infinispan
> Issue Type: Bug
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
>
> The cache is configured with:
> {code:xml}
> <cache-container name="local">
> <modules>
> <module name="deployment.myTask.jar"/>
> </modules>
> <local-cache name="compat">
> <compatibility enabled="true"/>
> </local-cache>
> </cache-container>
> {code}
> The {{myTask.jar}} contains Entity.class that is the Pojo to be stored in the cache.
> When doing a
> {code:java}
> remoteCache.put(1, new Entity());
> {code}
> The error is
> {noformat}
> 18:13:42,142 ERROR (main) [TestSuiteProgress] Test failed: LocalServerTestCompatModeIT.shouldRunPriceTask
> org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=9 returned
> server error (status=0x85): java.lang.ClassNotFoundException: org.infinispan.server.test.Entity from [Module "org.infinispan.commons:main" from
> local module loader @4ae82894 (finder: local module finder @543788f3 (roots: /home/gfernandes/github/infinispan/server/integration/testsuite/target/server/node1/modules,/home/gfernandes/github/infinispan/server/integration/testsuite/target/server/node1/modules/system/layers/base))]
> {noformat}
> I noticed that in this case, the Marshaller used is {{GenericJBossMarshaller}} who lives in infinispan-commons, and when asked to de-serialize the pojo in byte[] form, the classloader used is ModuleClassLoader for Module "org.infinispan.commons:main" that does not have visibility of the deployed jar's classes.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 5 months
[JBoss JIRA] (ISPN-8550) Try to estimate malloc overhead and add to memory based eviction
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-8550?page=com.atlassian.jira.plugin.... ]
William Burns commented on ISPN-8550:
-------------------------------------
I was using valgrind like the following:
{quote}
valgrind --tool=massif --time-unit=ms --detailed-freq=10000 <cmd>
{quote}
I increased detailed-freq really high to try to remove the excess.
I took the memory usage after it was primed with inserts and then subtracted that from the peak memory usage. Looking at your spreadsheet I don't see excess JVM and others subtracted from the size in your calculation. In my tests it was about 7M of memory used by that, which can throw off your other numbers, or maybe I am not reading your spreadsheet correctly.
> Try to estimate malloc overhead and add to memory based eviction
> ----------------------------------------------------------------
>
> Key: ISPN-8550
> URL: https://issues.jboss.org/browse/ISPN-8550
> Project: Infinispan
> Issue Type: Sub-task
> Reporter: William Burns
> Assignee: William Burns
> Fix For: 9.2.0.Beta2, 9.1.4.Final
>
> Attachments: allocs.ods
>
>
> We should try to also estimate malloc overhead. We could do something like Dan mentioned at https://github.com/infinispan/infinispan/pull/5590#pullrequestreview-7805...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 5 months
[JBoss JIRA] (ISPN-8568) ClassNotFoundException with Compat mode and custom Pojos
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-8568?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes commented on ISPN-8568:
-----------------------------------------
[~anistor] Is this a bug or the newly introduced <modules> aren't supposed to work with the existing marshallers?
> ClassNotFoundException with Compat mode and custom Pojos
> --------------------------------------------------------
>
> Key: ISPN-8568
> URL: https://issues.jboss.org/browse/ISPN-8568
> Project: Infinispan
> Issue Type: Bug
> Reporter: Gustavo Fernandes
>
> The cache is configured with:
> {code:xml}
> <cache-container name="local">
> <modules>
> <module name="deployment.myTask.jar"/>
> </modules>
> <local-cache name="compat">
> <compatibility enabled="true"/>
> </local-cache>
> </cache-container>
> {code}
> The {{myTask.jar}} contains Entity.class that is the Pojo to be stored in the cache.
> When doing a
> {code:java}
> remoteCache.put(1, new Entity());
> {code}
> The error is
> {noformat}
> 18:13:42,142 ERROR (main) [TestSuiteProgress] Test failed: LocalServerTestCompatModeIT.shouldRunPriceTask
> org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=9 returned
> server error (status=0x85): java.lang.ClassNotFoundException: org.infinispan.server.test.Entity from [Module "org.infinispan.commons:main" from
> local module loader @4ae82894 (finder: local module finder @543788f3 (roots: /home/gfernandes/github/infinispan/server/integration/testsuite/target/server/node1/modules,/home/gfernandes/github/infinispan/server/integration/testsuite/target/server/node1/modules/system/layers/base))]
> {noformat}
> I noticed that in this case, the Marshaller used is {{GenericJBossMarshaller}} who lives in infinispan-commons, and when asked to de-serialize the pojo in byte[] form, the classloader used is ModuleClassLoader for Module "org.infinispan.commons:main" that does not have visibility of the deployed jar's classes.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 5 months
[JBoss JIRA] (ISPN-8568) ClassNotFoundException with Compat mode and custom Pojos
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-8568?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-8568:
------------------------------------
Description:
The cache is configured with:
{code:xml}
<cache-container name="local">
<modules>
<module name="deployment.myTask.jar"/>
</modules>
<local-cache name="compat">
<compatibility enabled="true"/>
</local-cache>
</cache-container>
{code}
The {{myTask.jar}} contains Entity.class that is the Pojo to be stored in the cache.
When doing a
{code:java}
remoteCache.put(1, new Entity());
{code}
The error is
{noformat}
18:13:42,142 ERROR (main) [TestSuiteProgress] Test failed: LocalServerTestCompatModeIT.shouldRunPriceTask
org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=9 returned
server error (status=0x85): java.lang.ClassNotFoundException: org.infinispan.server.test.Entity from [Module "org.infinispan.commons:main" from
local module loader @4ae82894 (finder: local module finder @543788f3 (roots: /home/gfernandes
/github/infinispan/server/integration/testsuite/target/server/node1/modules,/home/gfernandes/github/
infinispan/server/integration/testsuite/target/server/node1/modules/system/layers/base))]
{noformat}
I noticed that in this case, the Marshaller used is {{GenericJBossMarshaller}} who lives in infinispan-commons, and when asked to de-serialize the pojo in byte[] form, the classloader used is ModuleClassLoader for Module "org.infinispan.commons:main" that does not have visibility of the deployed jar's classes.
was:
The cache is configured with:
{code:xml}
<cache-container name="local">
<modules>
<module name="deployment.myTask.jar"/>
</modules>
<local-cache name="compat">
<compatibility enabled="true"/>
</local-cache>
</cache-container>
{code}
The {{myTask.jar}} contains Entity.class that is the Pojo to be stored in the cache.
When doing a
{code:java}
remoteCache.put(1, new Entity());
{code}
The error is
{noformat}
18:13:42,142 ERROR (main) [TestSuiteProgress] Test failed: LocalServerTestCompatModeIT.shouldRunPriceTask
org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=9 returned server error (status=0x85): java.lang.ClassNotFoundException: org.infinispan.server.test.Entity from [Module "org.infinispan.commons:main" from local module loader @4ae82894 (finder: local module finder @543788f3 (roots: /home/gfernandes/github/infinispan/server/integration/testsuite/target/server/node1/modules,/home/gfernandes/github/infinispan/server/integration/testsuite/target/server/node1/modules/system/layers/base))]
{noformat}
I noticed that in this case, the Marshaller used is {{GenericJBossMarshaller}} who lives in infinispan-commons, and when asked to de-serialize the pojo in byte[] form, the classloader used is ModuleClassLoader for Module "org.infinispan.commons:main" that does not have visibility of the deployed jar's classes.
> ClassNotFoundException with Compat mode and custom Pojos
> --------------------------------------------------------
>
> Key: ISPN-8568
> URL: https://issues.jboss.org/browse/ISPN-8568
> Project: Infinispan
> Issue Type: Bug
> Reporter: Gustavo Fernandes
>
> The cache is configured with:
> {code:xml}
> <cache-container name="local">
> <modules>
> <module name="deployment.myTask.jar"/>
> </modules>
> <local-cache name="compat">
> <compatibility enabled="true"/>
> </local-cache>
> </cache-container>
> {code}
> The {{myTask.jar}} contains Entity.class that is the Pojo to be stored in the cache.
> When doing a
> {code:java}
> remoteCache.put(1, new Entity());
> {code}
> The error is
> {noformat}
> 18:13:42,142 ERROR (main) [TestSuiteProgress] Test failed: LocalServerTestCompatModeIT.shouldRunPriceTask
> org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=9 returned
> server error (status=0x85): java.lang.ClassNotFoundException: org.infinispan.server.test.Entity from [Module "org.infinispan.commons:main" from
> local module loader @4ae82894 (finder: local module finder @543788f3 (roots: /home/gfernandes
> /github/infinispan/server/integration/testsuite/target/server/node1/modules,/home/gfernandes/github/
> infinispan/server/integration/testsuite/target/server/node1/modules/system/layers/base))]
> {noformat}
> I noticed that in this case, the Marshaller used is {{GenericJBossMarshaller}} who lives in infinispan-commons, and when asked to de-serialize the pojo in byte[] form, the classloader used is ModuleClassLoader for Module "org.infinispan.commons:main" that does not have visibility of the deployed jar's classes.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 5 months