<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<body link="#355491" alink="#4262a1" vlink="#355491" style="background: #e2e2e2; margin: 0; padding: 20px;">
<div>
<table cellpadding="0" bgcolor="#FFFFFF" border="0" cellspacing="0" style="border: 1px solid #dadada; margin-bottom: 30px; width: 100%; -moz-border-radius: 6px; -webkit-border-radius: 6px;">
<tbody>
<tr>
<td>
<table border="0" cellpadding="0" cellspacing="0" bgcolor="#FFFFFF" style="border: solid 2px #ccc; background: #dadada; width: 100%; -moz-border-radius: 6px; -webkit-border-radius: 6px;">
<tbody>
<tr>
<td bgcolor="#000000" valign="middle" height="58px" style="border-bottom: 1px solid #ccc; padding: 20px; -moz-border-radius-topleft: 3px; -moz-border-radius-topright: 3px; -webkit-border-top-right-radius: 5px; -webkit-border-top-left-radius: 5px;">
<h1 style="color: #333333; font: bold 22px Arial, Helvetica, sans-serif; margin: 0; display: block !important;">
<!-- To have a header image/logo replace the name below with your img tag -->
<!-- Email clients will render the images when the message is read so any image -->
<!-- must be made available on a public server, so that all recipients can load the image. -->
<a href="http://community.jboss.org/index.jspa" style="text-decoration: none; color: #E1E1E1">JBoss Community</a></h1>
</td>
</tr>
<tr>
<td bgcolor="#FFFFFF" style="font: normal 12px Arial, Helvetica, sans-serif; color:#333333; padding: 20px; -moz-border-radius-bottomleft: 4px; -moz-border-radius-bottomright: 4px; -webkit-border-bottom-right-radius: 5px; -webkit-border-bottom-left-radius: 5px;"><h3 style="margin: 10px 0 5px; font-size: 17px; font-weight: normal;">
Issues while posting large number for messages to the Queue
</h3>
<span style="margin-bottom: 10px;">
created by <a href="http://community.jboss.org/people/devireddy">Rajasekhar Devi Reddy</a> in <i>JBoss Messaging</i> - <a href="http://community.jboss.org/message/562263#562263">View the full discussion</a>
</span>
<hr style="margin: 20px 0; border: none; background-color: #dadada; height: 1px;">
<div class="jive-rendered-content"><p><em>**I am sorry if this had to be posted to Remoting forum. I was not sure so I posted it here. I have also read other posts that talk abut similar issues but all of the posts there tell me that these issues should be resolved in the JBM & Remoting versions I am using (or in some earlier versions). But I still see them.</em></p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>Environment/setup</p><ul><li>Solaris SunOS 5.10 </li><li>java 1.6.0_20 <strong>64</strong> bit</li><li>JBoss AS 4.2.3</li><li>JBoss Messaging 1.4.5 GA</li><li>Remoting 2.2.3 sp2</li><li>Cluster of two nodes</li><li>Two queues (clustered)</li><li>Two MDBs one for each queue, MDB pool size is set to 30 for each.</li><li>Databsae Oracle 10g</li><li>Number of DefaultDS connections (form JMS persistence)  is set to a maximum of 100.</li></ul><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>We have an application that uses JMS queues and MDB to achieve asynchronous processing. We have two queues and two MDBs that receive and processes messages from these queues. Work flow starts by posting messages on to queue 1, MDB 1 picks up, processes the messages and write a message on to queue2. MDB 2 picks up the message and continues with its processing.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>We are seeing intermittent issues when processing large load of messages for example one scenario is:</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>When I post a large number (20,000) of messages to queue 1 I see errors talking about a failure while posting the message to queue 2. After this 20,000 message run is complete (with some failures) if I post another 1000 messages non of them appear in the queue (I was checking the JBM tables to verify this). At this point logs on both the nodes are filled with debug and warn messages shown below, and messages wont get posted/processed.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>If I kill node 2 (shutdown wont work) then all the 1000 messages get posted to Queue and they all get processed on node1.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>We had similar problems before and we changed the remoting component to version 2.2.3 SP2. But we still have these problems. Any advice on what should we look in to will be greatly helpful.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>Thanks for your time.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>Debug/Warn message on posting the second load of 1000 messages</p><p>2010-09-16 15:21:55,983 DEBUG <a class="jive-link-external-small" href="org.jboss.remoting.transport.bisocket.BisocketServerInvoker">org.jboss.remoting.transport.bisocket.BisocketServerInvoker</a> received new control socket for unrecognized listenerId: c3l2mo-kgijx1-ge5qqvbw-1-ge5t0s2s-uq22010-09-16 15:21:56,000 DEBUG <a class="jive-link-external-small" href="org.jboss.remoting.transport.bisocket.BisocketServerInvoker">org.jboss.remoting.transport.bisocket.BisocketServerInvoker</a> received new control socket for unrecognized listenerId: c3l2mo-kgijx1-ge5qqvbw-1-ge5s845p-7sx</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>2010-09-16 15:22:20,605 WARN  <a class="jive-link-external-small" href="org.jboss.remoting.transport.bisocket.BisocketServerInvoker">org.jboss.remoting.transport.bisocket.BisocketServerInvoker</a> org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@4f2ea142: detected failure on control connection Thread[control: Socket[addr=/12.129.94.24,port=43136,localport=55901],5,jboss] (c3l2mp-ougfh2-ge5qshw2-1-ge5saop5-ct1: requesting new control connection</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>ERROR form the log while processing the initial large load (20,000)</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>org.jboss.jms.exception.MessagingJMSException: Failed to invoke at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:271) at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:205) at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160) at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:499) at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java) at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:661) at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java) at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java) at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105) at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java) at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105) at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java) at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java) at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:276) at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect39.invoke(ProducerAspect39.java) at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java) at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105) at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java) at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java) at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:165) at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:208) at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:146) at com.liaison.merckContivoPilot.MDBClients.DeltaThreadMDBClient.writeToQueue(DeltaThreadMDBClient.java:120) at com.liaison.merckContivoPilot.bo.SDTMTransformer.performTransformation(SDTMTransformer.java:317) at com.liaison.merckContivoPilot.ejb.TransformThreadMDB.onMessage(TransformThreadMDB.java:80) at sun.reflect.GeneratedMethodAccessor128.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112) at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166) at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)<br/> at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)<br/> at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)<br/> at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)<br/> at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:195)<br/> at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)<br/> at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)<br/> at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)<br/> at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)<br/> at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)<br/> at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)<br/> at $Proxy84.onMessage(Unknown Source)<br/> at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)<br/> at org.jboss.jms.client.container.ClientConsumer.callOnMessageStatic(ClientConsumer.java:160)<br/> at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:831)<br/> at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)<br/> at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)<br/> at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)<br/> at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)<br/> at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)<br/> at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)<br/> at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)<br/> at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)<br/> at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)<br/> at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)<br/> at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761)<br/> at java.lang.Thread.run(Thread.java:619)<br/>Caused by: java.lang.IllegalStateException: Cannot find object in dispatcher with id 8i8-3sy8s5eg-1-wbvqq5eg-1xjigk-om2l3c<br/> at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:101)<br/> at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:157)<br/> at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:862)<br/> at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:634)<br/> at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:434)<br/> at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:176)<br/> at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:182)<br/> at org.jboss.remoting.Client.invoke(Client.java:1935)<br/> at org.jboss.remoting.Client.invoke(Client.java:788)<br/> at org.jboss.remoting.Client.invoke(Client.java:776)<br/> at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)</p></div>
<div style="background-color: #f4f4f4; padding: 10px; margin-top: 20px;">
<p style="margin: 0;">Reply to this message by <a href="http://community.jboss.org/message/562263#562263">going to Community</a></p>
<p style="margin: 0;">Start a new discussion in JBoss Messaging at <a href="http://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2042">Community</a></p>
</div></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
</body>
</html>