[jboss-user] [JBoss Messaging] - Xa - Memory Leak ?
jeanbobby
do-not-reply at jboss.com
Tue Feb 20 11:45:37 EST 2007
We are using 1.0.1SP4 on JBoss405.
The queues are persisted on oracle 10g.
JBM has its own non-xa oracle datasource
The multithreaded consumers rely on the XA Connection Factory and are wrapped in XA transactions.
One set of connection/session/consumer is created for each sequence of polling, no caching whatsoever, and we proceed with the usual pattern :
xaTx.begin
con = jmxXACf.createCon
ses = con.createSession
con.start
consumer = ses.createConsumer
consumer.receive()
consumer.close
session.close
con.close
xaTx.commit
they're all cleaned up after the receive call, and the XA transaction is always commited/rollbacked.
The jboss instance hosting both JBM and the consumers (bootstraped from a War using JBM loader) runs slowly but steadily to memory and cpu exhaustion, even when it's activity is limited to the constant jms 'polling' (8 consumer threads).
As the heap graphs were showing textbook linear memory consumption I ran jmap/jhat after a 2 hours run (with only the 30th first minutes of dense activity). It showed that the number of ServerConsumerEndpoint instances had made the top 5 with approx 200000 :
num #instances #bytes class name
--------------------------------------
1: 217640 42500368 [Ljava.util.HashMap$Entry;
2: 223956 25354776 [Ljava.lang.Object;
3: 306257 23970568 [C
4: 468932 22508736 java.util.HashMap$Entry
5: 199658 20764432 org.jboss.jms.server.endpoint.ServerConsumerEndpoint
Then I get all kind of stacks bound to the memory shortage like :
java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:50:39,420 ERROR [org.jboss.resource.adapter.jms.JmsSessionFactoryImpl] - could not create session
| org.jboss.resource.JBossResourceException: Unchecked throwable in ManagedConnection.getConnection() cl=org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener at 1a39aa42[state=NORMAL mc=org.jboss.resource.adapter.jms.JmsManagedConnection at 3f9c2783 handles=0 lastUse=1171014618826 permit=false trackByTx=false mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubjectAndCri at 2e2722df context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool at b627625 xaResource=MessagingXAResource[-2147483607] txSync=null]; - nested throwable: (java.lang.reflect.UndeclaredThrowableException)
| at org.jboss.resource.JBossResourceException.rethrowAsResourceException(JBossResourceException.java:61)
| at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:410)
| at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
| at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.allocateConnection(JmsSessionFactoryImpl.java:389)
| at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.createSession(JmsSessionFactoryImpl.java:369)
| at org.springframework.jms.listener.AbstractMessageListenerContainer.createSession(AbstractMessageListenerContainer.java:1002)
| at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.doReceiveAndExecute(DestinationBalancingMessageListenerContainer.java:360)
| at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.receiveAndExecute(DestinationBalancingMessageListenerContainer.java:326)
| at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DestinationBalancingMessageListenerContainer.java:716)
| at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.run(DestinationBalancingMessageListenerContainer.java:680)
| at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203)
| at java.lang.Thread.run(Thread.java:595)
| Caused by: java.lang.reflect.UndeclaredThrowableException
| at org.jboss.resource.JBossResourceException.process(JBossResourceException.java:204)
| at org.jboss.resource.JBossResourceException.<init>(JBossResourceException.java:111)
| ... 12 more
| Caused by: java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:51:13,748 ERROR [STDERR] - Exception in thread "Timer-1"
| 2007-02-09 10:51:13,748 ERROR [STDERR] - Exception in thread "TimeoutFactory-0"
| 2007-02-09 10:51:13,748 ERROR [STDERR] - java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:51:13,748 ERROR [STDERR] - java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:51:24,326 ERROR [org.jboss.jms.util.ExceptionUtil] - ConnectionEndpoint[-2147483643] start [a1885-6t7h9a-exxk6yht-1-exyfwp7q-3s]
| java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:51:38,514 ERROR [org.jboss.jms.util.ExceptionUtil] - SessionEndpoint[-2147483608] close [a1885-6t7h9a-exxk6yht-1-exyfx05u-3t]
| java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:51:41,951 ERROR [org.jboss.jms.client.container.ExceptionInterceptor] - Caught Error:
| java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:51:55,701 ERROR [org.jboss.jms.util.ExceptionUtil] - SessionEndpoint[-2147483602] createConsumerDelegate [a1885-6t7h9a-exxk6yht-1-exyfxarc-3u]
| java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:51:55,717 ERROR [org.jboss.jms.client.container.ExceptionInterceptor] - Caught Error:
| java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:51:59,170 ERROR [org.jboss.jms.client.container.ExceptionInterceptor] - Caught Error:
| java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:51:59,170 ERROR [com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer] - Setup of JMS message listener invoker failed - trying to recover
| java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:52:02,623 INFO [com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer] - Successfully refreshed JMS Connection
| 2007-02-09 10:52:06,061 ERROR [com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer] - Setup of JMS message listener invoker failed - trying to recover
| java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:52:19,858 INFO [com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer] - Successfully refreshed JMS Connection
| 2007-02-09 10:52:23,326 ERROR [cms.transcoding.technical] - Could not create a session: org.jboss.resource.JBossResourceException: Unchecked throwable in ManagedConnection.getConnection() cl=org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener at 1a39aa42[state=NORMAL mc=org.jboss.resource.adapter.jms.JmsManagedConnection at 3f9c2783 handles=0 lastUse=1171014618826 permit=false trackByTx=false mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubjectAndCri at 2e2722df context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool at b627625 xaResource=MessagingXAResource[-2147483607] txSync=null]; - nested throwable: (java.lang.reflect.UndeclaredThrowableException)
| javax.jms.JMSException: Could not create a session: org.jboss.resource.JBossResourceException: Unchecked throwable in ManagedConnection.getConnection() cl=org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener at 1a39aa42[state=NORMAL mc=org.jboss.resource.adapter.jms.JmsManagedConnection at 3f9c2783 handles=0 lastUse=1171014618826 permit=false trackByTx=false mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubjectAndCri at 2e2722df context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool at b627625 xaResource=MessagingXAResource[-2147483607] txSync=null]; - nested throwable: (java.lang.reflect.UndeclaredThrowableException)
| at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.allocateConnection(JmsSessionFactoryImpl.java:403)
| at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.createSession(JmsSessionFactoryImpl.java:369)
| at org.springframework.jms.listener.AbstractMessageListenerContainer.createSession(AbstractMessageListenerContainer.java:1002)
| at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.doReceiveAndExecute(DestinationBalancingMessageListenerContainer.java:360)
| at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.receiveAndExecute(DestinationBalancingMessageListenerContainer.java:326)
| at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DestinationBalancingMessageListenerContainer.java:716)
| at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.run(DestinationBalancingMessageListenerContainer.java:680)
| at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203)
| at java.lang.Thread.run(Thread.java:595)
| 2007-02-09 10:52:33,920 ERROR [org.jboss.jms.client.container.ExceptionInterceptor] - Caught Error:
| java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:52:48,030 ERROR [STDERR] - Exception in thread "JCA PoolFiller"
| 2007-02-09 10:52:48,030 ERROR [STDERR] - java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:53:06,014 ERROR [cms.transcoding.technical] - A failure has occurred during processing of the request. Please consult the server logs for more details. ConnectionEndpoint[-2147483643] start [a1885-6t7h9a-exxk6yht-1-exyfwp7q-3s]
| org.jboss.jms.util.MessagingJMSException: A failure has occurred during processing of the request. Please consult the server logs for more details. ConnectionEndpoint[-2147483643] start [a1885-6t7h9a-exxk6yht-1-exyfwp7q-3s]
| at org.jboss.jms.util.ExceptionUtil.handleJMSInvocation(ExceptionUtil.java:72)
| at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.start(ServerConnectionEndpoint.java:254)
| at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.org$jboss$jms$server$endpoint$advised$ConnectionAdvised$start$aop(ConnectionAdvised.java:94)
| at sun.reflect.GeneratedMethodAccessor134.invoke(Unknown Source)
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
| at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
| at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
| at org.jboss.aop.Advisor.dynamicInvoke(Advisor.java:723)
| at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:101)
| at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127)
| at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:1014)
| at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:98)
| at org.jboss.remoting.Client.invoke(Client.java:607)
| at org.jboss.remoting.Client.invoke(Client.java:599)
| at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111)
| at org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeNext(ClientConnectionDelegate$start_N8025343665958530775.java)
| at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
| at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
| at org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeNext(ClientConnectionDelegate$start_N8025343665958530775.java)
| at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
| at org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeNext(ClientConnectionDelegate$start_N8025343665958530775.java)
| at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
| at org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeNext(ClientConnectionDelegate$start_N8025343665958530775.java)
| at org.jboss.jms.client.delegate.ClientConnectionDelegate.start(ClientConnectionDelegate.java)
| at org.jboss.jms.client.JBossConnection.start(JBossConnection.java:121)
| at org.jboss.resource.adapter.jms.JmsManagedConnection.start(JmsManagedConnection.java:599)
| at org.jboss.resource.adapter.jms.JmsSession.start(JmsSession.java:530)
| at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.start(JmsSessionFactoryImpl.java:239)
| at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.doReceiveAndExecute(DestinationBalancingMessageListenerContainer.java:361)
| at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.receiveAndExecute(DestinationBalancingMessageListenerContainer.java:326)
| at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DestinationBalancingMessageListenerContainer.java:716)
| at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.run(DestinationBalancingMessageListenerContainer.java:680)
| at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203)
| at java.lang.Thread.run(Thread.java:595)
| 2007-02-09 10:54:39,248 ERROR [cms.transcoding.technical] - A failure has occurred during processing of the request. Please consult the server logs for more details. SessionEndpoint[-2147483602] createConsumerDelegate [a1885-6t7h9a-exxk6yht-1-exyfxarc-3u]
| org.jboss.jms.util.MessagingJMSException: A failure has occurred during processing of the request. Please consult the server logs for more details. SessionEndpoint[-2147483602] createConsumerDelegate [a1885-6t7h9a-exxk6yht-1-exyfxarc-3u]
| at org.jboss.jms.util.ExceptionUtil.handleJMSInvocation(ExceptionUtil.java:72)
| at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegate(ServerSessionEndpoint.java:287)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$createConsumerDelegate$aop(SessionAdvised.java:90)
| at sun.reflect.GeneratedMethodAccessor137.invoke(Unknown Source)
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
| at org.jboss.jms.server.container.SecurityAspect.handleCreateConsumerDelegate(SecurityAspect.java:116)
| at sun.reflect.GeneratedMethodAccessor136.invoke(Unknown Source)
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:130)
| at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
| at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
| at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
| at org.jboss.aop.Advisor.dynamicInvoke(Advisor.java:723)
| at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:101)
| at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127)
| at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:1014)
| at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:98)
| at org.jboss.remoting.Client.invoke(Client.java:607)
| at org.jboss.remoting.Client.invoke(Client.java:599)
| at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111)
| at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java)
| at org.jboss.jms.client.container.StateCreationAspect.handleCreateConsumerDelegate(StateCreationAspect.java:122)
| at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect20.invoke(StateCreationAspect20.java)
| at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java)
| at org.jboss.jms.client.container.ConsumerAspect.handleCreateConsumerDelegate(ConsumerAspect.java:69)
| at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect19.invoke(ConsumerAspect19.java)
| at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java)
| at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
| at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
| at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java)
| at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
| at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java)
| at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
| at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java)
| at org.jboss.jms.client.delegate.ClientSessionDelegate.createConsumerDelegate(ClientSessionDelegate.java)
| at org.jboss.jms.client.JBossSession.createConsumer(JBossSession.java:254)
| at org.jboss.jms.client.JBossSession.createConsumer(JBossSession.java:231)
| at org.jboss.resource.adapter.jms.JmsSession.createConsumer(JmsSession.java:473)
| at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.createConsumer(DestinationBalancingMessageListenerContainer.java:648)
| at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.createListenerConsumer(DestinationBalancingMessageListenerContainer.java:302)
| at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.doReceiveAndExecute(DestinationBalancingMessageListenerContainer.java:363)
| at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.receiveAndExecute(DestinationBalancingMessageListenerContainer.java:326)
| at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DestinationBalancingMessageListenerContainer.java:716)
| at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.run(DestinationBalancingMessageListenerContainer.java:680)
| at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203)
| at java.lang.Thread.run(Thread.java:595)
I saw that some XA related memory leaks issues had been fixed into SP4, so I wondered if these elements might ring a bell before I dig further down...
This scenario is very easy to reproduce as-is on production,preprod or dev environments, I didnt have the time yet to create a barebone test, Id rather not write several and Im not sure how much to cut (xa, oracle, spring ?)
This not a critical issue as the server handles perfectly the load peaks during the first hour(s) of its (short) life, and we can restart it easily
but it's obviously a far-from-ideal situation.
Thanks for your help,
Best Regards
Olivier Cuzacq
Digiplug.com
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4019443#4019443
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4019443
More information about the jboss-user
mailing list