[JBoss Cache] - JBoss Cache - Memory Leak when many Fqn/nodes added in Cache
by Sathish
Sathish [http://community.jboss.org/people/sathish_06] created the discussion
"JBoss Cache - Memory Leak when many Fqn/nodes added in Cache"
To view the discussion, visit: http://community.jboss.org/message/617503#617503
--------------------------------------------------------------
All - Iam observing OOM in JBoss Cache, with below mentioned scenario. Would like to get suggestions/help on the approach that iam doing to over come the situation is correct or not. Here is the details on the problem,
Problem:- JBoss Cache - Memory Leak when many Fqn/nodes added in Cache using TreeCache.put() API
Explanation of the problem:- When adding new data in cache with new node in the tree, its causing memory leak.
Environment/Config settings:-
JBoss Cache Version:-JBossCache 1.2.4.SP2[ $Id: Version.java,v 1.5.2.1.4.3 2006/02/15 05:08:06 bstansberry Exp $]\
CacheMode:LOCAL
EvictionPolicy: LRUPolicy
Test Cache region - /testRegionX definition:-
<region name="/testRegionX">
<attribute name="maxNodes">1000</attribute>
<attribute name="timeToLiveSeconds">30</attribute>
<!-- Maximum time an object is kept in cache regardless of idle time -->
<attribute name="maxAgeSeconds">30</attribute>
</region>
In the test code, start adding new objects in the cache(treeCache.put(identifier, key , vo)) with different FQN/tree node, as shown below.
/testRegionX/a/b/ k1, v1
/testRegionX/a1/b1/ k2, v2
/testRegionX/a12b2/ k2, v3
.....
Snippet of the test code,
TreeCache treeCache;
try {
treeCache = new TreeCache();
treeCache.setCacheMode(TreeCache.LOCAL);
System.out.println("JBoss Cache Version:-" + treeCache.getVersion());
PropertyConfigurator config = new PropertyConfigurator();
config.configure(treeCache, "TreeCache.xml");
treeCache.startService();
String identifier_prefix = "/testRegionX";
String key_prefix = "key_01_";
for(long i=0;;i++)
{
String key = key_prefix + i;
String identifier = identifier_prefix + "/" + i + "/data/";
//String identifier = identifier_prefix;
System.out.println("Identifer = " + identifier);
DummyVO vo = new DummyVO();
System.out.println("Adding[Id=new node, Key=unique] - Key - " + key);
treeCache.put(identifier, key , vo);
//access the data from cache
treeCache.get(identifier, key);
//dont call remove
//treeCache.remove(identifier, key);
//System.out.println("cache.remove called");
}
//treeCache.stopService();
} catch (Exception e) {
e.printStackTrace();
}
Refer attached zip file contains complete Tree cache configuration(TreeCache.xml) and test code. Just point the jboss library location in the ant.properits file to run the test code.
Note that i dont call the tree.remove() to remove the cache data, instead let the Cache manager evict the object once it gets expired(configured expiry time to 30 seconds.).
Ran test with default JVM heap size, and reunning out of memory in the 50K iteration. Took JMap histo and noticed that below 2 objects are not releasing, Attached histo in text file.
C:\Documents and Settings\skandasa>jmap -histo:live 4508
num #instances #bytes class name
----------------------------------------------
1: 107609 16019856 [LEDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap$Entry;
3: 107609 6026104 EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap
When I start adding new objects in the cache(treeCache.put(identifier, key , vo)) in the same tree node ('/testRegionX' ie., not creating new nodes in the tree, as shown below) then not observing this OOM problem.
/testRegionX/ k1, v1
/testRegionX/ k2, v2
/testRegionX/ k2, v3
.....
In other words, if i simply store data in cache as a Hashmap, without creating many tree nodes, then not observing OOM.
Any thoughts on this behaviour of the JBoss cache ? How can i get control over the OOM problem when adding new nodes in the tree cache ?
Please advice!
Thanks,
Sathish
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/617503#617503]
Start a new discussion in JBoss Cache at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&cont...]
12 years, 5 months
[jBPM] - Re: JBPM 3.2.5, MSSQL 2005 - blocking in JBPM_JOB table
by Alejandro Guizar
Alejandro Guizar [http://community.jboss.org/people/alex.guizar%40jboss.com] created the discussion
"Re: JBPM 3.2.5,MSSQL 2005 - blocking in JBPM_JOB table"
To view the discussion, visit: http://community.jboss.org/message/563437#563437
--------------------------------------------------------------
> Looking at how the jbpm_job table is utilized, yes, I think these specific indexes can be removed. Just to clarify. From what I have seen in *our use-case*, under normal circumstances the jbpm_job table has very few entries at any point in time. Seems like entries are added and removed constantly. That being the case, I'm not sure I can see a benefit of spending the extra time/resources maintaining indexes which are never utilized.
I'm going to explore the possibility of removing the job indexes. Can you please https://jira.jboss.org/browse/JBPM create a JIRA issue for this?
> By the way, this is as good a time to explain the use-case that is failing for us. We have a workflow with three steps synchronous steps. The first step takes a couple to a few hundred milliseconds to complete, the second step can take anywhere from 5 to 30 seconds to complete and then the last step usually takes a half a second. When I mentioned jobs earlier, I was referring to workflows contained the three steps just mentioned.
If possible, could you attach your workflow and test harness to the JIRA issue? Please remove any sensitive information you would not like to see published.
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/563437#563437]
Start a new discussion in jBPM at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&cont...]
12 years, 5 months
[JBoss Remoting] - Out of Memory Error with ConcurrentReaderHashMap on Jboss 5.1.0
by Anil Mathew
Anil Mathew [http://community.jboss.org/people/amathewjboss1] created the discussion
"Out of Memory Error with ConcurrentReaderHashMap on Jboss 5.1.0"
To view the discussion, visit: http://community.jboss.org/message/626575#626575
--------------------------------------------------------------
Hi,
We are running on JBoss 5.1 / JDK 1.6 on Linux servers. During very high load we are seeing Out of Memory Error and it seems like related to the JBoss remoting.
We took a heap dump and the Memory Analyzer shows the below instance is taking 25% of the memory:
One instance of *"org.jboss.mx.server.MBeanServerImpl"* loaded by *"org.jboss.classloader.spi.base.BaseClassLoader @ 0x415b2fc8"* occupies *65,055,032 (24.64%)* bytes. The memory is accumulated in one instance of *"EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap$Entry[]"* loaded by *"org.jboss.bootstrap.NoAnnotationURLClassLoader @ 0x42f8ba78"*.
The details which is pointing ot the JBoss remoting classes:
Class Name Shallow Heap Retained Heap
1. EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap$Entry[2048] Shallow Heap => 8,208 Retained Heap => 45,693,888
EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap Shallow Heap => 56 Retained Heap => 45,761,640
EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap$Entry Shallow Heap => 24 Retained Heap => 45,761,688
EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap$Entry[64] Shallow Heap => 272 Retained Heap => 64,584,776
EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap Shallow Heap => 56 Retained Heap => 64,621,408
org.jboss.mx.server.registry.BasicMBeanRegistry Shallow Heap => 48 Retained Heap => 64,621,680
org.jboss.mx.server.MBeanServerImpl Shallow Heap => 24 Retained Heap => 65,055,032
com.arjuna.ats.internal.jbossatx.agent.LocalJBossAgentImpl
java.lang.Thread @ 0x42135bb8 Signal Dispatcher Native Stack, Thread
org.jboss.invocation.pooled.server.PooledInvokerHA
org.jboss.invocation.pooled.server.PooledInvoker
org.jboss.remoting.transport.bisocket.BisocketServerInvoker
org.jboss.remoting.transport.socket.SocketServerInvoker
Any thoughts on this please?
Thanks
Anil Mathew
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/626575#626575]
Start a new discussion in JBoss Remoting at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&cont...]
12 years, 5 months
[JBoss Remoting] - Messaging server lock up issues
by vidhya Baskaran
vidhya Baskaran [http://community.jboss.org/people/vbs_jboss_accnt] created the discussion
"Messaging server lock up issues"
To view the discussion, visit: http://community.jboss.org/message/624162#624162
--------------------------------------------------------------
We are having issues with a JBOSS Application server running with the following versions
jboss messaging 1.4.6
jboss remoting 2.5.2
primarily being used for messaging purposes.
The server often locks up and the only way to get the clients to connect again is to restart the server.
Looking at the stack trace ,multiple workerthreads are waiting on this following lock
- waiting on <0x1990f6eb> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(WriterPreferenceReadWriteLock.java:163)
org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2210)
org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:515)
org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:777)
org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:399)
org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
sun.reflect.GeneratedMethodAccessor233.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:157)
org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:897)
org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:768)
- locked <0x69b3f1c0> (a org.jboss.remoting.transport.socket.ServerThread)
org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:721)
org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:548)
org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)
Attached is the settings from the remoting-bisocket-service.xml file
<!--
<attribute name="writeTimeout" isParam="true">30000</attribute>
-->
<!-- End immutable parameters -->
<attribute name="stopLeaseOnFailure" isParam="true">true</attribute>
<!-- Periodicity of client pings. Server window by default is twice this figure -->
<attribute name="clientLeasePeriod" isParam="true">10000</attribute>
<attribute name="validatorPingPeriod" isParam="true">10000</attribute>
<attribute name="validatorPingTimeout" isParam="true">5000</attribute>
<attribute name="failureDisconnectTimeout" isParam="true">0</attribute>
<attribute name="callbackErrorsAllowed">1</attribute>
<attribute name="registerCallbackListener">false</attribute>
<attribute name="useClientConnectionIdentity" isParam="true">true</attribute>
<attribute name="timeout" isParam="true">0</attribute>
<!-- Max Number of connections in client pool. This should be significantly higher than
the max number of sessions/consumers you expect -->
<attribute name="JBM_clientMaxPoolSize" isParam="true">1000</attribute>
<!-- The maximum time to wait before timing out on trying to write a message to socket for delivery -->
<attribute name="callbackTimeout">10000</attribute>
I am trying to find out if its a configuration issue or a code issue on our side ?
Is it possible that one particular client is having problems and that in turn causes the server to lock up ?
Please help.
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/624162#624162]
Start a new discussion in JBoss Remoting at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&cont...]
12 years, 5 months
