<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<body link="#355491" alink="#4262a1" vlink="#355491" style="background: #e2e2e2; margin: 0; padding: 20px;">
<div>
<table cellpadding="0" bgcolor="#FFFFFF" border="0" cellspacing="0" style="border: 1px solid #dadada; margin-bottom: 30px; width: 100%; -moz-border-radius: 6px; -webkit-border-radius: 6px;">
<tbody>
<tr>
<td>
<table border="0" cellpadding="0" cellspacing="0" bgcolor="#FFFFFF" style="border: solid 2px #ccc; background: #dadada; width: 100%; -moz-border-radius: 6px; -webkit-border-radius: 6px;">
<tbody>
<tr>
<td bgcolor="#000000" valign="middle" height="58px" style="border-bottom: 1px solid #ccc; padding: 20px; -moz-border-radius-topleft: 3px; -moz-border-radius-topright: 3px; -webkit-border-top-right-radius: 5px; -webkit-border-top-left-radius: 5px;">
<h1 style="color: #333333; font: bold 22px Arial, Helvetica, sans-serif; margin: 0; display: block !important;">
<!-- To have a header image/logo replace the name below with your img tag -->
<!-- Email clients will render the images when the message is read so any image -->
<!-- must be made available on a public server, so that all recipients can load the image. -->
<a href="http://community.jboss.org/index.jspa" style="text-decoration: none; color: #E1E1E1">JBoss Community</a></h1>
</td>
</tr>
<tr>
<td bgcolor="#FFFFFF" style="font: normal 12px Arial, Helvetica, sans-serif; color:#333333; padding: 20px; -moz-border-radius-bottomleft: 4px; -moz-border-radius-bottomright: 4px; -webkit-border-bottom-right-radius: 5px; -webkit-border-bottom-left-radius: 5px;"><h3 style="margin: 10px 0 5px; font-size: 17px; font-weight: normal;">
Stalled Queue Consumer on Windows(NIO) with Large Message and ConsumerWindow
</h3>
<span style="margin-bottom: 10px;">
created by <a href="http://community.jboss.org/people/amalter">Adam Malter</a> in <i>JBoss Messaging</i> - <a href="http://community.jboss.org/message/624167#624167">View the full discussion</a>
</span>
<hr style="margin: 20px 0; border: none; background-color: #dadada; height: 1px;">
<div class="jive-rendered-content"><p>Hi All,</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>We use HornetQ as our development JMS provider, and I'm trying to get everyone comfortable enough to upgrade it to production. Having some troubles however when we really start to bang on it.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>This is the case:</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>HornetQ running in our old customized JBoss instance (I won't even mention the version number, needless to say we've re-written enough stuff that I worry my case is not generic/easily reproducible)</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>Anyway - this is what I'm seeing..</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>Server config is:</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><pre class="jive-pre"><code class="jive-code">
         Configuration config = new ConfigurationImpl();
         config.setPersistenceEnabled(true);
         config.setJournalDirectory(FileUtil.buildPath(rootData, "journal").getPath());
         config.setBindingsDirectory(FileUtil.buildPath(rootData, "bindings").getPath());
         config.setPagingDirectory(FileUtil.buildPath(rootData, "paging").getPath());
         config.setLargeMessagesDirectory(FileUtil.buildPath(rootData, "lobs").getPath());
         config.setPersistDeliveryCountBeforeDelivery(true); //required for redelivery count to work on rollbacks
         if (SystemUtils.IS_OS_WINDOWS) {
            config.setJournalType(JournalType.NIO);
         }
         //HornetQ requires cluster user and pw or spits out warnings
         config.setClustered(false);
         config.setClusterUser("tcuser");
         config.setClusterPassword("tradecard");
         config.setSecurityEnabled(false);
         //Simple Q Service for now - only in VM transport
         config.getAcceptorConfigurations().add(new TransportConfiguration(InVMAcceptorFactory.class.getName()));
         config.setCreateJournalDir(true);
         config.setCreateBindingsDir(true);
</code></pre><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>Contents of hornetq-version.properties from our jar file:</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><pre class="jive-pre"><code class="jive-code">
hornetq.version.versionName=HQ_2_2_5_FINAL_AS7
hornetq.version.majorVersion=2
hornetq.version.minorVersion=2
hornetq.version.microVersion=5
hornetq.version.incrementingVersion=121
hornetq.version.versionSuffix=Final
hornetq.version.versionTag=Final
hornetq.netty.version=3.2.3.Final-r${buildNumber}
hornetq.version.compatibleVersionList=121
</code></pre><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>We're running in XA mode with durable queues and no selectors or funny priorities. Additionally, the issue only occurs when I leave the consumer window size at default. If I change setConsumerWindowSize(0), my issue disappears.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>So inside a single transaction, I send about 10 ~512kb to the same destination. Destination receives them okay, starts to process them, and then triggers an application level error (In this case I'm purposefully triggering it to test our resubmit handling)</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>Our application logic is that we try to process the message a pre-set number of times (lets say 3) and before moving it to dead letter. This usually happens on the very first rollback, but not always - essentially the consumer/mdb picks up a message, rolls it back, starts new tx, tries to pick up another one and then hangs with the following stack:</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><pre class="jive-pre"><code class="jive-code">
"AuditMDB[000]" - Thread t@113
   java.lang.Thread.State: TIMED_WAITING
          at java.lang.Object.wait(Native Method)
          - waiting on <1a647ea> (a org.hornetq.core.client.impl.LargeMessageControllerImpl)
          at org.hornetq.core.client.impl.LargeMessageControllerImpl.waitCompletion(LargeMessageControllerImpl.java:304)
          at org.hornetq.core.client.impl.LargeMessageControllerImpl.saveBuffer(LargeMessageControllerImpl.java:283)
          at org.hornetq.core.client.impl.ClientLargeMessageImpl.checkBuffer(ClientLargeMessageImpl.java:201)
          at org.hornetq.core.client.impl.ClientLargeMessageImpl.getBodyBuffer(ClientLargeMessageImpl.java:101)
          at org.hornetq.jms.client.HornetQBytesMessage.getBuffer(HornetQBytesMessage.java:451)
          at org.hornetq.jms.client.HornetQBytesMessage.readBytes(HornetQBytesMessage.java:248)
          at com.tradecard.server.util.BytesMessageInputStream.read(BytesMessageInputStream.java:68)
          at com.tradecard.framework.util.ChunkedByteBuffer.append(ChunkedByteBuffer.java:344)
          at com.tradecard.framework.util.ChunkedByteBuffer.append(ChunkedByteBuffer.java:332)
          at com.tradecard.server.mdb.HornetQContainerInvoker$HornetQServerSession.run(HornetQContainerInvoker.java:287)
          at java.lang.Thread.run(Thread.java:662)
   Locked ownable synchronizers:
          - None
</code></pre><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>The somewhat relevant portions of our Hornet Invoker are:</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><pre class="jive-pre"><code class="jive-code">
                  xaResource = ((XAQueueSession)session).getXAResource();
                  txManager.begin();
                  tx = txManager.getTransaction();
                  tx.enlistResource(xaResource);
                  pooledSession.syncRegister();
                  tx.registerSynchronization(pooledSession);
                  //cache the session/senders for QueueSender to use
                  TransactionCache.put(TransactionCache.MQXA_SESSION, pooledSession);
                  txid = TransactionToolbox.getGlobalTransactionId();
                  Message requestMessage = null;
                  requestMessage = receiver.receive(2000);
                  if (requestMessage != null &amp;&amp; !quit) {
                     Message message = null;
                     try {
                        BytesMessage bytesMessage = (BytesMessage)requestMessage;
                        BytesMessageInputStream stream = new BytesMessageInputStream(bytesMessage);
                        ChunkedByteBuffer buffer = new ChunkedByteBuffer(10000, 100);
                        buffer.append(stream);   // <--- hanging here
....
</code></pre><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>There doesn't seem to be anyone owning the lock. Eventually I get timeout warnings from my TxManager and have to bounce the server. As soon as I bounce, it picks up another message and processes, quickly getting the error again.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>Additionally, I can see a number of files pending in my lobs directory.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>All of our custom infrastructure makes it difficult to extract the case into a postable test case. However, if anyone thinks it worthwhile and this doesn't seem like anything straight forward, I'll make the attempt.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>The idea for setting the consumer window size to zero came from the following closed bug - <a class="jive-link-external-small" href="https://issues.jboss.org/browse/HORNETQ-111">https://issues.jboss.org/browse/HORNETQ-111</a></p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>Thanks for your time and any help!</p></div>
<div style="background-color: #f4f4f4; padding: 10px; margin-top: 20px;">
<p style="margin: 0;">Reply to this message by <a href="http://community.jboss.org/message/624167#624167">going to Community</a></p>
<p style="margin: 0;">Start a new discussion in JBoss Messaging at <a href="http://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2042">Community</a></p>
</div></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
</body>
</html>