10:45 a.m.
We are using 1.0.1SP4 on JBoss405.
The queues are persisted on oracle 10g.
JBM has its own non-xa oracle datasource
The multithreaded consumers rely on the XA Connection Factory and are wrapped in XA
transactions.
One set of connection/session/consumer is created for each sequence of polling, no caching
whatsoever, and we proceed with the usual pattern :
xaTx.begin
con = jmxXACf.createCon
ses = con.createSession
con.start
consumer = ses.createConsumer
consumer.receive()
consumer.close
session.close
con.close
xaTx.commit
they're all cleaned up after the receive call, and the XA transaction is always
commited/rollbacked.
The jboss instance hosting both JBM and the consumers (bootstraped from a War using JBM
loader) runs slowly but steadily to memory and cpu exhaustion, even when it's activity
is limited to the constant jms 'polling' (8 consumer threads).
As the heap graphs were showing textbook linear memory consumption I ran jmap/jhat after a
2 hours run (with only the 30th first minutes of dense activity). It showed that the
number of ServerConsumerEndpoint instances had made the top 5 with approx 200000 :
num #instances #bytes class name
--------------------------------------
1: 217640 42500368 [Ljava.util.HashMap$Entry;
2: 223956 25354776 [Ljava.lang.Object;
3: 306257 23970568 [C
4: 468932 22508736 java.util.HashMap$Entry
5: 199658 20764432 org.jboss.jms.server.endpoint.ServerConsumerEndpoint
Then I get all kind of stacks bound to the memory shortage like :
java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:50:39,420 ERROR [org.jboss.resource.adapter.jms.JmsSessionFactoryImpl]
- could not create session
| org.jboss.resource.JBossResourceException: Unchecked throwable in
ManagedConnection.getConnection()
cl=org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener@1a39aa42[state=NORMAL
mc=org.jboss.resource.adapter.jms.JmsManagedConnection@3f9c2783 handles=0
lastUse=1171014618826 permit=false trackByTx=false
mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubjectAndCri@2e2722df
context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool@b627625
xaResource=MessagingXAResource[-2147483607] txSync=null]; - nested throwable:
(java.lang.reflect.UndeclaredThrowableException)
| at
org.jboss.resource.JBossResourceException.rethrowAsResourceException(JBossResourceException.java:61)
| at
org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:410)
| at
org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
| at
org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.allocateConnection(JmsSessionFactoryImpl.java:389)
| at
org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.createSession(JmsSessionFactoryImpl.java:369)
| at
org.springframework.jms.listener.AbstractMessageListenerContainer.createSession(AbstractMessageListenerContainer.java:1002)
| at
com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.doReceiveAndExecute(DestinationBalancingMessageListenerContainer.java:360)
| at
com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.receiveAndExecute(DestinationBalancingMessageListenerContainer.java:326)
| at
com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DestinationBalancingMessageListenerContainer.java:716)
| at
com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.run(DestinationBalancingMessageListenerContainer.java:680)
| at
org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203)
| at java.lang.Thread.run(Thread.java:595)
| Caused by: java.lang.reflect.UndeclaredThrowableException
| at
org.jboss.resource.JBossResourceException.process(JBossResourceException.java:204)
| at
org.jboss.resource.JBossResourceException.<init>(JBossResourceException.java:111)
| ... 12 more
| Caused by: java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:51:13,748 ERROR [STDERR] - Exception in thread "Timer-1"
| 2007-02-09 10:51:13,748 ERROR [STDERR] - Exception in thread
"TimeoutFactory-0"
| 2007-02-09 10:51:13,748 ERROR [STDERR] - java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:51:13,748 ERROR [STDERR] - java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:51:24,326 ERROR [org.jboss.jms.util.ExceptionUtil] -
ConnectionEndpoint[-2147483643] start [a1885-6t7h9a-exxk6yht-1-exyfwp7q-3s]
| java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:51:38,514 ERROR [org.jboss.jms.util.ExceptionUtil] -
SessionEndpoint[-2147483608] close [a1885-6t7h9a-exxk6yht-1-exyfx05u-3t]
| java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:51:41,951 ERROR [org.jboss.jms.client.container.ExceptionInterceptor] -
Caught Error:
| java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:51:55,701 ERROR [org.jboss.jms.util.ExceptionUtil] -
SessionEndpoint[-2147483602] createConsumerDelegate [a1885-6t7h9a-exxk6yht-1-exyfxarc-3u]
| java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:51:55,717 ERROR [org.jboss.jms.client.container.ExceptionInterceptor] -
Caught Error:
| java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:51:59,170 ERROR [org.jboss.jms.client.container.ExceptionInterceptor] -
Caught Error:
| java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:51:59,170 ERROR
[com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer]
- Setup of JMS message listener invoker failed - trying to recover
| java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:52:02,623 INFO
[com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer]
- Successfully refreshed JMS Connection
| 2007-02-09 10:52:06,061 ERROR
[com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer]
- Setup of JMS message listener invoker failed - trying to recover
| java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:52:19,858 INFO
[com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer]
- Successfully refreshed JMS Connection
| 2007-02-09 10:52:23,326 ERROR [cms.transcoding.technical] - Could not create a
session: org.jboss.resource.JBossResourceException: Unchecked throwable in
ManagedConnection.getConnection()
cl=org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener@1a39aa42[state=NORMAL
mc=org.jboss.resource.adapter.jms.JmsManagedConnection@3f9c2783 handles=0
lastUse=1171014618826 permit=false trackByTx=false
mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubjectAndCri@2e2722df
context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool@b627625
xaResource=MessagingXAResource[-2147483607] txSync=null]; - nested throwable:
(java.lang.reflect.UndeclaredThrowableException)
| javax.jms.JMSException: Could not create a session:
org.jboss.resource.JBossResourceException: Unchecked throwable in
ManagedConnection.getConnection()
cl=org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener@1a39aa42[state=NORMAL
mc=org.jboss.resource.adapter.jms.JmsManagedConnection@3f9c2783 handles=0
lastUse=1171014618826 permit=false trackByTx=false
mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubjectAndCri@2e2722df
context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool@b627625
xaResource=MessagingXAResource[-2147483607] txSync=null]; - nested throwable:
(java.lang.reflect.UndeclaredThrowableException)
| at
org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.allocateConnection(JmsSessionFactoryImpl.java:403)
| at
org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.createSession(JmsSessionFactoryImpl.java:369)
| at
org.springframework.jms.listener.AbstractMessageListenerContainer.createSession(AbstractMessageListenerContainer.java:1002)
| at
com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.doReceiveAndExecute(DestinationBalancingMessageListenerContainer.java:360)
| at
com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.receiveAndExecute(DestinationBalancingMessageListenerContainer.java:326)
| at
com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DestinationBalancingMessageListenerContainer.java:716)
| at
com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.run(DestinationBalancingMessageListenerContainer.java:680)
| at
org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203)
| at java.lang.Thread.run(Thread.java:595)
| 2007-02-09 10:52:33,920 ERROR [org.jboss.jms.client.container.ExceptionInterceptor] -
Caught Error:
| java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:52:48,030 ERROR [STDERR] - Exception in thread "JCA
PoolFiller"
| 2007-02-09 10:52:48,030 ERROR [STDERR] - java.lang.OutOfMemoryError: Java heap space
| 2007-02-09 10:53:06,014 ERROR [cms.transcoding.technical] - A failure has occurred
during processing of the request. Please consult the server logs for more details.
ConnectionEndpoint[-2147483643] start [a1885-6t7h9a-exxk6yht-1-exyfwp7q-3s]
| org.jboss.jms.util.MessagingJMSException: A failure has occurred during processing of
the request. Please consult the server logs for more details.
ConnectionEndpoint[-2147483643] start [a1885-6t7h9a-exxk6yht-1-exyfwp7q-3s]
| at org.jboss.jms.util.ExceptionUtil.handleJMSInvocation(ExceptionUtil.java:72)
| at
org.jboss.jms.server.endpoint.ServerConnectionEndpoint.start(ServerConnectionEndpoint.java:254)
| at
org.jboss.jms.server.endpoint.advised.ConnectionAdvised.org$jboss$jms$server$endpoint$advised$ConnectionAdvised$start$aop(ConnectionAdvised.java:94)
| at sun.reflect.GeneratedMethodAccessor134.invoke(Unknown Source)
| at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
| at
org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
| at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
| at org.jboss.aop.Advisor.dynamicInvoke(Advisor.java:723)
| at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:101)
| at
org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127)
| at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:1014)
| at
org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:98)
| at org.jboss.remoting.Client.invoke(Client.java:607)
| at org.jboss.remoting.Client.invoke(Client.java:599)
| at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111)
| at
org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeNext(ClientConnectionDelegate$start_N8025343665958530775.java)
| at
org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
| at
org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
| at
org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeNext(ClientConnectionDelegate$start_N8025343665958530775.java)
| at
org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
| at
org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeNext(ClientConnectionDelegate$start_N8025343665958530775.java)
| at
org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
| at
org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeNext(ClientConnectionDelegate$start_N8025343665958530775.java)
| at
org.jboss.jms.client.delegate.ClientConnectionDelegate.start(ClientConnectionDelegate.java)
| at org.jboss.jms.client.JBossConnection.start(JBossConnection.java:121)
| at
org.jboss.resource.adapter.jms.JmsManagedConnection.start(JmsManagedConnection.java:599)
| at org.jboss.resource.adapter.jms.JmsSession.start(JmsSession.java:530)
| at
org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.start(JmsSessionFactoryImpl.java:239)
| at
com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.doReceiveAndExecute(DestinationBalancingMessageListenerContainer.java:361)
| at
com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.receiveAndExecute(DestinationBalancingMessageListenerContainer.java:326)
| at
com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DestinationBalancingMessageListenerContainer.java:716)
| at
com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.run(DestinationBalancingMessageListenerContainer.java:680)
| at
org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203)
| at java.lang.Thread.run(Thread.java:595)
| 2007-02-09 10:54:39,248 ERROR [cms.transcoding.technical] - A failure has occurred
during processing of the request. Please consult the server logs for more details.
SessionEndpoint[-2147483602] createConsumerDelegate [a1885-6t7h9a-exxk6yht-1-exyfxarc-3u]
| org.jboss.jms.util.MessagingJMSException: A failure has occurred during processing of
the request. Please consult the server logs for more details. SessionEndpoint[-2147483602]
createConsumerDelegate [a1885-6t7h9a-exxk6yht-1-exyfxarc-3u]
| at org.jboss.jms.util.ExceptionUtil.handleJMSInvocation(ExceptionUtil.java:72)
| at
org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegate(ServerSessionEndpoint.java:287)
| at
org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$createConsumerDelegate$aop(SessionAdvised.java:90)
| at sun.reflect.GeneratedMethodAccessor137.invoke(Unknown Source)
| at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
| at
org.jboss.jms.server.container.SecurityAspect.handleCreateConsumerDelegate(SecurityAspect.java:116)
| at sun.reflect.GeneratedMethodAccessor136.invoke(Unknown Source)
| at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:130)
| at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
| at
org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
| at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
| at org.jboss.aop.Advisor.dynamicInvoke(Advisor.java:723)
| at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:101)
| at
org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127)
| at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:1014)
| at
org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:98)
| at org.jboss.remoting.Client.invoke(Client.java:607)
| at org.jboss.remoting.Client.invoke(Client.java:599)
| at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111)
| at
org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java)
| at
org.jboss.jms.client.container.StateCreationAspect.handleCreateConsumerDelegate(StateCreationAspect.java:122)
| at
org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect20.invoke(StateCreationAspect20.java)
| at
org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java)
| at
org.jboss.jms.client.container.ConsumerAspect.handleCreateConsumerDelegate(ConsumerAspect.java:69)
| at
org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect19.invoke(ConsumerAspect19.java)
| at
org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java)
| at
org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
| at
org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
| at
org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java)
| at
org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
| at
org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java)
| at
org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
| at
org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java)
| at
org.jboss.jms.client.delegate.ClientSessionDelegate.createConsumerDelegate(ClientSessionDelegate.java)
| at org.jboss.jms.client.JBossSession.createConsumer(JBossSession.java:254)
| at org.jboss.jms.client.JBossSession.createConsumer(JBossSession.java:231)
| at org.jboss.resource.adapter.jms.JmsSession.createConsumer(JmsSession.java:473)
| at
com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.createConsumer(DestinationBalancingMessageListenerContainer.java:648)
| at
com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.createListenerConsumer(DestinationBalancingMessageListenerContainer.java:302)
| at
com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.doReceiveAndExecute(DestinationBalancingMessageListenerContainer.java:363)
| at
com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.receiveAndExecute(DestinationBalancingMessageListenerContainer.java:326)
| at
com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DestinationBalancingMessageListenerContainer.java:716)
| at
com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.run(DestinationBalancingMessageListenerContainer.java:680)
| at
org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203)
| at java.lang.Thread.run(Thread.java:595)
I saw that some XA related memory leaks issues had been fixed into SP4, so I wondered if
these elements might ring a bell before I dig further down...
This scenario is very easy to reproduce as-is on production,preprod or dev environments, I
didnt have the time yet to create a barebone test, Id rather not write several and Im not
sure how much to cut (xa, oracle, spring ?)
This not a critical issue as the server handles perfectly the load peaks during the first
hour(s) of its (short) life, and we can restart it easily
but it's obviously a far-from-ideal situation.
Thanks for your help,
Best Regards
Olivier Cuzacq
Digiplug.com
View the original post :
http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4019443#...
Reply to the post :
http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&a...