[jboss-user] [JBoss Messaging] - Xa - Memory Leak ?

jeanbobby do-not-reply at jboss.com
Tue Feb 20 11:45:37 EST 2007


We are using 1.0.1SP4 on JBoss405.
The queues are persisted on oracle 10g.
JBM has its own non-xa oracle datasource

The multithreaded consumers rely on the XA Connection Factory and are wrapped in XA transactions.
One set of connection/session/consumer is created for each sequence of polling, no caching whatsoever, and we proceed with the usual pattern : 

xaTx.begin

con = jmxXACf.createCon
ses = con.createSession
con.start
consumer = ses.createConsumer
consumer.receive()
consumer.close
session.close
con.close

xaTx.commit

they're all cleaned up after the receive call, and the XA transaction is always commited/rollbacked.

The jboss instance hosting both JBM and the consumers (bootstraped from a War using JBM loader) runs slowly but steadily to memory and cpu exhaustion, even when it's activity is limited to the constant jms 'polling' (8 consumer threads).

As the heap graphs were showing textbook linear memory consumption I ran jmap/jhat after a 2 hours run (with only the 30th first minutes of dense activity). It showed that the number of ServerConsumerEndpoint instances had made the top 5 with approx 200000 :

num   #instances    #bytes  class name
--------------------------------------
  1:    217640    42500368  [Ljava.util.HashMap$Entry;
  2:    223956    25354776  [Ljava.lang.Object;
  3:    306257    23970568  [C
  4:    468932    22508736  java.util.HashMap$Entry
  5:    199658    20764432  org.jboss.jms.server.endpoint.ServerConsumerEndpoint


Then I get all kind of stacks bound to the memory shortage like :

java.lang.OutOfMemoryError: Java heap space
  | 2007-02-09 10:50:39,420  ERROR [org.jboss.resource.adapter.jms.JmsSessionFactoryImpl] - could not create session
  | org.jboss.resource.JBossResourceException: Unchecked throwable in ManagedConnection.getConnection() cl=org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener at 1a39aa42[state=NORMAL mc=org.jboss.resource.adapter.jms.JmsManagedConnection at 3f9c2783 handles=0 lastUse=1171014618826 permit=false trackByTx=false mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubjectAndCri at 2e2722df context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool at b627625 xaResource=MessagingXAResource[-2147483607] txSync=null]; - nested throwable: (java.lang.reflect.UndeclaredThrowableException)
  | 	at org.jboss.resource.JBossResourceException.rethrowAsResourceException(JBossResourceException.java:61)
  | 	at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:410)
  | 	at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
  | 	at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.allocateConnection(JmsSessionFactoryImpl.java:389)
  | 	at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.createSession(JmsSessionFactoryImpl.java:369)
  | 	at org.springframework.jms.listener.AbstractMessageListenerContainer.createSession(AbstractMessageListenerContainer.java:1002)
  | 	at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.doReceiveAndExecute(DestinationBalancingMessageListenerContainer.java:360)
  | 	at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.receiveAndExecute(DestinationBalancingMessageListenerContainer.java:326)
  | 	at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DestinationBalancingMessageListenerContainer.java:716)
  | 	at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.run(DestinationBalancingMessageListenerContainer.java:680)
  | 	at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203)
  | 	at java.lang.Thread.run(Thread.java:595)
  | Caused by: java.lang.reflect.UndeclaredThrowableException
  | 	at org.jboss.resource.JBossResourceException.process(JBossResourceException.java:204)
  | 	at org.jboss.resource.JBossResourceException.<init>(JBossResourceException.java:111)
  | 	... 12 more
  | Caused by: java.lang.OutOfMemoryError: Java heap space
  | 2007-02-09 10:51:13,748  ERROR [STDERR] - Exception in thread "Timer-1" 
  | 2007-02-09 10:51:13,748  ERROR [STDERR] - Exception in thread "TimeoutFactory-0" 
  | 2007-02-09 10:51:13,748  ERROR [STDERR] - java.lang.OutOfMemoryError: Java heap space
  | 2007-02-09 10:51:13,748  ERROR [STDERR] - java.lang.OutOfMemoryError: Java heap space
  | 2007-02-09 10:51:24,326  ERROR [org.jboss.jms.util.ExceptionUtil] - ConnectionEndpoint[-2147483643] start [a1885-6t7h9a-exxk6yht-1-exyfwp7q-3s]
  | java.lang.OutOfMemoryError: Java heap space
  | 2007-02-09 10:51:38,514  ERROR [org.jboss.jms.util.ExceptionUtil] - SessionEndpoint[-2147483608] close [a1885-6t7h9a-exxk6yht-1-exyfx05u-3t]
  | java.lang.OutOfMemoryError: Java heap space
  | 2007-02-09 10:51:41,951  ERROR [org.jboss.jms.client.container.ExceptionInterceptor] - Caught Error: 
  | java.lang.OutOfMemoryError: Java heap space
  | 2007-02-09 10:51:55,701  ERROR [org.jboss.jms.util.ExceptionUtil] - SessionEndpoint[-2147483602] createConsumerDelegate [a1885-6t7h9a-exxk6yht-1-exyfxarc-3u]
  | java.lang.OutOfMemoryError: Java heap space
  | 2007-02-09 10:51:55,717  ERROR [org.jboss.jms.client.container.ExceptionInterceptor] - Caught Error: 
  | java.lang.OutOfMemoryError: Java heap space
  | 2007-02-09 10:51:59,170  ERROR [org.jboss.jms.client.container.ExceptionInterceptor] - Caught Error: 
  | java.lang.OutOfMemoryError: Java heap space
  | 2007-02-09 10:51:59,170  ERROR [com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer] - Setup of JMS message listener invoker failed - trying to recover
  | java.lang.OutOfMemoryError: Java heap space
  | 2007-02-09 10:52:02,623  INFO  [com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer] - Successfully refreshed JMS Connection
  | 2007-02-09 10:52:06,061  ERROR [com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer] - Setup of JMS message listener invoker failed - trying to recover
  | java.lang.OutOfMemoryError: Java heap space
  | 2007-02-09 10:52:19,858  INFO  [com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer] - Successfully refreshed JMS Connection
  | 2007-02-09 10:52:23,326  ERROR [cms.transcoding.technical] - Could not create a session: org.jboss.resource.JBossResourceException: Unchecked throwable in ManagedConnection.getConnection() cl=org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener at 1a39aa42[state=NORMAL mc=org.jboss.resource.adapter.jms.JmsManagedConnection at 3f9c2783 handles=0 lastUse=1171014618826 permit=false trackByTx=false mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubjectAndCri at 2e2722df context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool at b627625 xaResource=MessagingXAResource[-2147483607] txSync=null]; - nested throwable: (java.lang.reflect.UndeclaredThrowableException)
  | javax.jms.JMSException: Could not create a session: org.jboss.resource.JBossResourceException: Unchecked throwable in ManagedConnection.getConnection() cl=org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener at 1a39aa42[state=NORMAL mc=org.jboss.resource.adapter.jms.JmsManagedConnection at 3f9c2783 handles=0 lastUse=1171014618826 permit=false trackByTx=false mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubjectAndCri at 2e2722df context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool at b627625 xaResource=MessagingXAResource[-2147483607] txSync=null]; - nested throwable: (java.lang.reflect.UndeclaredThrowableException)
  | 	at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.allocateConnection(JmsSessionFactoryImpl.java:403)
  | 	at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.createSession(JmsSessionFactoryImpl.java:369)
  | 	at org.springframework.jms.listener.AbstractMessageListenerContainer.createSession(AbstractMessageListenerContainer.java:1002)
  | 	at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.doReceiveAndExecute(DestinationBalancingMessageListenerContainer.java:360)
  | 	at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.receiveAndExecute(DestinationBalancingMessageListenerContainer.java:326)
  | 	at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DestinationBalancingMessageListenerContainer.java:716)
  | 	at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.run(DestinationBalancingMessageListenerContainer.java:680)
  | 	at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203)
  | 	at java.lang.Thread.run(Thread.java:595)
  | 2007-02-09 10:52:33,920  ERROR [org.jboss.jms.client.container.ExceptionInterceptor] - Caught Error: 
  | java.lang.OutOfMemoryError: Java heap space
  | 2007-02-09 10:52:48,030  ERROR [STDERR] - Exception in thread "JCA PoolFiller" 
  | 2007-02-09 10:52:48,030  ERROR [STDERR] - java.lang.OutOfMemoryError: Java heap space
  | 2007-02-09 10:53:06,014  ERROR [cms.transcoding.technical] - A failure has occurred during processing of the request. Please consult the server logs for more details. ConnectionEndpoint[-2147483643] start [a1885-6t7h9a-exxk6yht-1-exyfwp7q-3s]
  | org.jboss.jms.util.MessagingJMSException: A failure has occurred during processing of the request. Please consult the server logs for more details. ConnectionEndpoint[-2147483643] start [a1885-6t7h9a-exxk6yht-1-exyfwp7q-3s]
  | 	at org.jboss.jms.util.ExceptionUtil.handleJMSInvocation(ExceptionUtil.java:72)
  | 	at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.start(ServerConnectionEndpoint.java:254)
  | 	at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.org$jboss$jms$server$endpoint$advised$ConnectionAdvised$start$aop(ConnectionAdvised.java:94)
  | 	at sun.reflect.GeneratedMethodAccessor134.invoke(Unknown Source)
  | 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  | 	at java.lang.reflect.Method.invoke(Method.java:585)
  | 	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
  | 	at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
  | 	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
  | 	at org.jboss.aop.Advisor.dynamicInvoke(Advisor.java:723)
  | 	at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:101)
  | 	at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127)
  | 	at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:1014)
  | 	at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:98)
  | 	at org.jboss.remoting.Client.invoke(Client.java:607)
  | 	at org.jboss.remoting.Client.invoke(Client.java:599)
  | 	at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111)
  | 	at org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeNext(ClientConnectionDelegate$start_N8025343665958530775.java)
  | 	at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
  | 	at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
  | 	at org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeNext(ClientConnectionDelegate$start_N8025343665958530775.java)
  | 	at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
  | 	at org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeNext(ClientConnectionDelegate$start_N8025343665958530775.java)
  | 	at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
  | 	at org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeNext(ClientConnectionDelegate$start_N8025343665958530775.java)
  | 	at org.jboss.jms.client.delegate.ClientConnectionDelegate.start(ClientConnectionDelegate.java)
  | 	at org.jboss.jms.client.JBossConnection.start(JBossConnection.java:121)
  | 	at org.jboss.resource.adapter.jms.JmsManagedConnection.start(JmsManagedConnection.java:599)
  | 	at org.jboss.resource.adapter.jms.JmsSession.start(JmsSession.java:530)
  | 	at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.start(JmsSessionFactoryImpl.java:239)
  | 	at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.doReceiveAndExecute(DestinationBalancingMessageListenerContainer.java:361)
  | 	at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.receiveAndExecute(DestinationBalancingMessageListenerContainer.java:326)
  | 	at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DestinationBalancingMessageListenerContainer.java:716)
  | 	at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.run(DestinationBalancingMessageListenerContainer.java:680)
  | 	at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203)
  | 	at java.lang.Thread.run(Thread.java:595)
  | 2007-02-09 10:54:39,248  ERROR [cms.transcoding.technical] - A failure has occurred during processing of the request. Please consult the server logs for more details. SessionEndpoint[-2147483602] createConsumerDelegate [a1885-6t7h9a-exxk6yht-1-exyfxarc-3u]
  | org.jboss.jms.util.MessagingJMSException: A failure has occurred during processing of the request. Please consult the server logs for more details. SessionEndpoint[-2147483602] createConsumerDelegate [a1885-6t7h9a-exxk6yht-1-exyfxarc-3u]
  | 	at org.jboss.jms.util.ExceptionUtil.handleJMSInvocation(ExceptionUtil.java:72)
  | 	at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegate(ServerSessionEndpoint.java:287)
  | 	at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$createConsumerDelegate$aop(SessionAdvised.java:90)
  | 	at sun.reflect.GeneratedMethodAccessor137.invoke(Unknown Source)
  | 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  | 	at java.lang.reflect.Method.invoke(Method.java:585)
  | 	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
  | 	at org.jboss.jms.server.container.SecurityAspect.handleCreateConsumerDelegate(SecurityAspect.java:116)
  | 	at sun.reflect.GeneratedMethodAccessor136.invoke(Unknown Source)
  | 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  | 	at java.lang.reflect.Method.invoke(Method.java:585)
  | 	at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:130)
  | 	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
  | 	at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
  | 	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
  | 	at org.jboss.aop.Advisor.dynamicInvoke(Advisor.java:723)
  | 	at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:101)
  | 	at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127)
  | 	at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:1014)
  | 	at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:98)
  | 	at org.jboss.remoting.Client.invoke(Client.java:607)
  | 	at org.jboss.remoting.Client.invoke(Client.java:599)
  | 	at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111)
  | 	at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java)
  | 	at org.jboss.jms.client.container.StateCreationAspect.handleCreateConsumerDelegate(StateCreationAspect.java:122)
  | 	at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect20.invoke(StateCreationAspect20.java)
  | 	at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java)
  | 	at org.jboss.jms.client.container.ConsumerAspect.handleCreateConsumerDelegate(ConsumerAspect.java:69)
  | 	at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect19.invoke(ConsumerAspect19.java)
  | 	at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java)
  | 	at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
  | 	at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
  | 	at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java)
  | 	at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
  | 	at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java)
  | 	at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
  | 	at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java)
  | 	at org.jboss.jms.client.delegate.ClientSessionDelegate.createConsumerDelegate(ClientSessionDelegate.java)
  | 	at org.jboss.jms.client.JBossSession.createConsumer(JBossSession.java:254)
  | 	at org.jboss.jms.client.JBossSession.createConsumer(JBossSession.java:231)
  | 	at org.jboss.resource.adapter.jms.JmsSession.createConsumer(JmsSession.java:473)
  | 	at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.createConsumer(DestinationBalancingMessageListenerContainer.java:648)
  | 	at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.createListenerConsumer(DestinationBalancingMessageListenerContainer.java:302)
  | 	at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.doReceiveAndExecute(DestinationBalancingMessageListenerContainer.java:363)
  | 	at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.receiveAndExecute(DestinationBalancingMessageListenerContainer.java:326)
  | 	at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DestinationBalancingMessageListenerContainer.java:716)
  | 	at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.run(DestinationBalancingMessageListenerContainer.java:680)
  | 	at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203)
  | 	at java.lang.Thread.run(Thread.java:595)



I saw that some XA related memory leaks issues had been fixed into SP4, so I wondered if these elements might ring a bell before I dig further down...

This scenario is very easy to reproduce as-is on production,preprod or dev environments, I didnt have the time yet to create a barebone test, Id rather not write several and Im not sure how much to cut (xa, oracle, spring ?)

This not a critical issue as the server handles perfectly the load peaks during the first hour(s) of its (short) life, and we can restart it easily
but it's obviously a far-from-ideal situation.

Thanks for your help,
Best Regards

Olivier Cuzacq
Digiplug.com

View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4019443#4019443

Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4019443



More information about the jboss-user mailing list