Aurus Signature

Hi Dev Team,

I am using infinispan 8.2.4 standalone hotrod server and i am facing "java.lang.OutOfMemoryError: Java heap space" issue when putting or retrieving the data into cache.

In below two ways i have loaded the data into local cache.

Case 1 : (Data Loading is taking more time, But there is no issue with "OutOfMemoryError: Java heap space")

In this case I am storing the index into a different dedicated infinispan cache by using the below tag ,

<property name="default.directory_provider">Infinispan</property>

I have observed that it is taking much time to load the data into cache , but in this case i am not facing Java Out of memory heap space issue during data putting or retrieving.

Below is local cache configuration,

<local-cache name="MERCHANT_STORE_DETAILS" start="EAGER">

            <locking acquire-timeout="30000" concurrency-level="1000" striping="false"/>
            <transaction mode="NONE"/>
            <eviction strategy="NONE"/>
            <file-store path="/MRSD/data" purge="false" passivation="false" preload="true"/>
            <indexing index="LOCAL">
                            <property name="default.indexmanager">org.infinispan.query.indexmanager.InfinispanIndexManager</property>
                            <property name="default.directory_provider">Infinispan</property>
                            <property name="default.metadata_cachename">MRSD-INDEX_META_DATA</property>
                            <property name="default.data_cachename">MRSD-INDEX_DATA</property>
                           <property name="default.locking_cachename">MRSD-INDEX_LOCKING</property>
            </indexing>
</local-cache>
<local-cache name="MRSD-INDEX_META_DATA" start="EAGER">
             <indexing index="NONE"/>
            <file-store path="/MRSD/index/indexMetadata" shared="false" purge="false" passivation="false" preload="true" fetch-state="true" read-only="false"/>
</local-cache>
<local-cache name="MRSD-INDEX_DATA" start="EAGER">
          <indexing index="NONE"/>
        <file-store path="MRSD/index/indexData" shared="false" purge="false" passivation="false" preload="true" fetch-state="true" read-only="false"/>
</local-cache>
<local-cache name="MRSD-INDEX_LOCKING" start="EAGER">
          <indexing index="NONE"/>
        <file-store path="MRSD/index/indexLocking" shared="false" purge="false" passivation="false" preload="true" fetch-state="true" read-only="false"/>
</local-cache>

Time Taken To Load Data: 3 Min 13 Sec

Number of Records : 45024

Case 2 : (Data Loading is very quick, But issue with "OutOfMemoryError: Java heap space")

In this case I am storing the index into memory by using the below tag ,

<property name="default.directory_provider">ram</property>

I have observed that, data has loaded into cache with in some seconds for same number of records that mentioned in Case 1,

But in this case when i am trying the store the same number of records(45024) multiple times and retrieving the data in same duration , It is throwing OutOfMemoryError: Java heap space in infinispan-server-8.2.4.Final/standalone/configuration/server.log.

Below is local cache configuration,

<local-cache name="MERCHANT_STORE_DETAILS" start="EAGER">
            <locking acquire-timeout="30000" concurrency-level="1000" striping="false"/>
                    <transaction mode="NONE"/>
                   <eviction strategy="NONE"/>
                    <indexing index="LOCAL">
                            <property name="default.indexmanager">near-real-time</property>
                            <property name="default.directory_provider">ram</property>
            </indexing>
</local-cache>

Time Taken To Load Data: 32 Sec

Number of Records : 45024

Exception In Details

2017-01-09 06:38:31,777 ERROR [stderr] (HotRod-externalServerHandler-1-88) Exception in thread "HotRod-externalServerHandler-1-88" java.lang.OutOfMemoryError: Java heap space
2017-01-09 06:39:18,706 ERROR [org.hibernate.search.exception.impl.LogErrorHandler] (Lucene Merge Thread #7869 for index org.infinispan.query.remote.impl.indexing.ProtobufValueWrapper) HSEARCH000058: HSEARCH000118: Exception during index Merge operation: org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
        at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668)
        at org.hibernate.search.backend.impl.lucene.overrides.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:42)
        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648)
Caused by: java.lang.OutOfMemoryError: Java heap space

2017-01-09 06:41:23,919 ERROR [stderr] (HotRod-externalServerHandler-1-88)      at org.antlr.runtime.Lexer.emit(Lexer.java:160)
2017-01-09 06:36:54,872 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 2) WFLYCTL0013: Operation ("read-children-resources") failed - address: ([]): java.lang.OutOfMemoryError: Java heap space
        at java.util.HashMap.resize(HashMap.java:703)
        at java.util.HashMap.putVal(HashMap.java:628)
        at java.util.HashMap.put(HashMap.java:611)
        at org.jboss.as.controller.OperationContextImpl$AuthorizationResponseImpl.addResourceResult(OperationContextImpl.java:2356)
        at org.jboss.as.controller.OperationContextImpl$AuthorizationResponseImpl.access$700(OperationContextImpl.java:2328)
        at org.jboss.as.controller.OperationContextImpl.getBasicAuthorizationResponse(OperationContextImpl.java:1764)
        at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1659)
        at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1618)
        at org.jboss.as.controller.OperationContextImpl.getResourceRegistration(OperationContextImpl.java:567)
        at org.jboss.as.controller.operations.global.ReadChildrenResourcesHandler.execute(ReadChildrenResourcesHandler.java:94)
        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:890)
        at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:659)
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
        at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1344)
        at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:392)
        at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:204)
        at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:659)
        at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:649)
        at org.jboss.as.server.deployment.scanner.DefaultDeploymentOperations.getDeploymentsStatus(DefaultDeploymentOperations.java:76)
        at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1614)
        at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1563)
        at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:568)
        at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:489)
        at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$DeploymentScanRunnable.run(FileSystemDeploymentService.java:250)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        at org.jboss.threads.JBossThread.run(JBossThread.java:320)

2017-01-09 06:38:39,564 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (HotRod-externalServerHandler-1-107) ISPN000136: Error executing command PutKeyValueCommand, writing keys [[B0x4a1b313031323331..[29]]: java.lang.OutOfMemoryError: Java heap space
        at java.util.ResourceBundle.putBundleInCache(ResourceBundle.java:1694)
        at java.util.ResourceBundle.findBundle(ResourceBundle.java:1471)
        at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419)
        at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1361)
        at java.util.ResourceBundle.getBundle(ResourceBundle.java:890)
        at sun.util.resources.LocaleData$1.run(LocaleData.java:164)
        at sun.util.resources.LocaleData$1.run(LocaleData.java:160)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.util.resources.LocaleData.getBundle(LocaleData.java:160)
        at sun.util.resources.LocaleData.getNumberFormatData(LocaleData.java:156)
        at sun.util.locale.provider.LocaleResources.getNumberPatterns(LocaleResources.java:439)
        at sun.util.locale.provider.NumberFormatProviderImpl.getInstance(NumberFormatProviderImpl.java:177)
        at sun.util.locale.provider.NumberFormatProviderImpl.getNumberInstance(NumberFormatProviderImpl.java:149)
        at java.text.NumberFormat.getInstance(NumberFormat.java:875)
        at java.text.NumberFormat.getInstance(NumberFormat.java:861)
        at java.text.NumberFormat.getNumberInstance(NumberFormat.java:458)
        at org.infinispan.commons.util.Util.prettyPrintTime(Util.java:353)
        at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:238)
        at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAndRecord(AbstractLockingInterceptor.java:193)
        at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:98)
        at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:41)
        at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:65)
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:78)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
        at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:191)
        at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:177)
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:78)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
        at org.infinispan.interceptors.compat.BaseTypeConverterInterceptor.visitPutKeyValueCommand(BaseTypeConverterInterceptor.java:84)
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:78)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
        at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:114)

Could any one please help me here to fix the "OutOfMemoryError: Java heap space" issue for Case 2.

Please let me know if you need more information on this.

Thanks in advance.

--
Aurus Signature

________________________________________________________________________________________________________________________________________________________________

Thanks and Regards,

Sridhar Jena, Sr. Techincal Lead, Products

+91 - 20 - 27655062 | +91 - 9226345267 | sjena345 | www.aurusinc.com
Plot G-2, Sector 26, Pradhikaran, Pune, Maharashtra-411044

Please consider the environment before printing this e-mail

Disclaimers: This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required please request a hard-copy version.