[jboss-user] [JBoss Messaging] - Issues while posting large number for messages to the Queue
Rajasekhar Devi Reddy
do-not-reply at jboss.com
Thu Sep 16 16:29:31 EDT 2010
Rajasekhar Devi Reddy [http://community.jboss.org/people/devireddy] created the discussion
"Issues while posting large number for messages to the Queue"
To view the discussion, visit: http://community.jboss.org/message/562263#562263
--------------------------------------------------------------
+**I am sorry if this had to be posted to Remoting forum. I was not sure so I posted it here. I have also read other posts that talk abut similar issues but all of the posts there tell me that these issues should be resolved in the JBM & Remoting versions I am using (or in some earlier versions). But I still see them.+
Environment/setup
* Solaris SunOS 5.10
* java 1.6.0_20 *64* bit
* JBoss AS 4.2.3
* JBoss Messaging 1.4.5 GA
* Remoting 2.2.3 sp2
* Cluster of two nodes
* Two queues (clustered)
* Two MDBs one for each queue, MDB pool size is set to 30 for each.
* Databsae Oracle 10g
* Number of DefaultDS connections (form JMS persistence) is set to a maximum of 100.
We have an application that uses JMS queues and MDB to achieve asynchronous processing. We have two queues and two MDBs that receive and processes messages from these queues. Work flow starts by posting messages on to queue 1, MDB 1 picks up, processes the messages and write a message on to queue2. MDB 2 picks up the message and continues with its processing.
We are seeing intermittent issues when processing large load of messages for example one scenario is:
When I post a large number (20,000) of messages to queue 1 I see errors talking about a failure while posting the message to queue 2. After this 20,000 message run is complete (with some failures) if I post another 1000 messages non of them appear in the queue (I was checking the JBM tables to verify this). At this point logs on both the nodes are filled with debug and warn messages shown below, and messages wont get posted/processed.
If I kill node 2 (shutdown wont work) then all the 1000 messages get posted to Queue and they all get processed on node1.
We had similar problems before and we changed the remoting component to version 2.2.3 SP2. But we still have these problems. Any advice on what should we look in to will be greatly helpful.
Thanks for your time.
Debug/Warn message on posting the second load of 1000 messages
2010-09-16 15:21:55,983 DEBUG org.jboss.remoting.transport.bisocket.BisocketServerInvoker org.jboss.remoting.transport.bisocket.BisocketServerInvoker received new control socket for unrecognized listenerId: c3l2mo-kgijx1-ge5qqvbw-1-ge5t0s2s-uq22010-09-16 15:21:56,000 DEBUG org.jboss.remoting.transport.bisocket.BisocketServerInvoker org.jboss.remoting.transport.bisocket.BisocketServerInvoker received new control socket for unrecognized listenerId: c3l2mo-kgijx1-ge5qqvbw-1-ge5s845p-7sx
2010-09-16 15:22:20,605 WARN org.jboss.remoting.transport.bisocket.BisocketServerInvoker org.jboss.remoting.transport.bisocket.BisocketServerInvoker org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask at 4f2ea142: detected failure on control connection Thread[control: Socket[addr=/12.129.94.24,port=43136,localport=55901],5,jboss] (c3l2mp-ougfh2-ge5qshw2-1-ge5saop5-ct1: requesting new control connection
ERROR form the log while processing the initial large load (20,000)
org.jboss.jms.exception.MessagingJMSException: Failed to invoke at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:271) at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:205) at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160) at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:499) at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java) at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:661) at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java) at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java) at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105) at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java) at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105) at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java) at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java) at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:276) at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect39.invoke(ProducerAspect39.java) at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java) at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105) at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java) at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java) at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:165) at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:208) at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:146) at com.liaison.merckContivoPilot.MDBClients.DeltaThreadMDBClient.writeToQueue(DeltaThreadMDBClient.java:120) at com.liaison.merckContivoPilot.bo.SDTMTransformer.performTransformation(SDTMTransformer.java:317) at com.liaison.merckContivoPilot.ejb.TransformThreadMDB.onMessage(TransformThreadMDB.java:80) at sun.reflect.GeneratedMethodAccessor128.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112) at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166) at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:195)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
at $Proxy84.onMessage(Unknown Source)
at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
at org.jboss.jms.client.container.ClientConsumer.callOnMessageStatic(ClientConsumer.java:160)
at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:831)
at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalStateException: Cannot find object in dispatcher with id 8i8-3sy8s5eg-1-wbvqq5eg-1xjigk-om2l3c
at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:101)
at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:157)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:862)
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:634)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:434)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:176)
at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:182)
at org.jboss.remoting.Client.invoke(Client.java:1935)
at org.jboss.remoting.Client.invoke(Client.java:788)
at org.jboss.remoting.Client.invoke(Client.java:776)
at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/562263#562263]
Start a new discussion in JBoss Messaging at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2042]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/jboss-user/attachments/20100916/e511209a/attachment-0001.html
More information about the jboss-user
mailing list