<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0">
<base href="https://issues.jboss.org">
<title>Message Title</title>
</head>
<body class="jira" style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 1.429">
<table id="background-table" cellpadding="0" cellspacing="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; background-color: #f5f5f5; border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt" bgcolor="#f5f5f5">
<!-- header here -->
<tbody>
<tr>
<td id="header-pattern-container" style="padding: 0px; border-collapse: collapse; padding: 10px 20px">
<table id="header-pattern" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td id="header-avatar-image-container" valign="top" style="padding: 0px; border-collapse: collapse; vertical-align: top; width: 32px; padding-right: 8px" width="32"> <img id="header-avatar-image" class="image_fix" src="https://static.jboss.org/developer/gravatar/cc058d8d4bde1b49e2654d79981be257?d=mm&s=48" height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top"> </td>
<td id="header-text-container" valign="middle" style="padding: 0px; border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px"> <a class="user-hover" rel="mwessendorf" id="email_mwessendorf" href="https://issues.jboss.org/secure/ViewProfile.jspa?name=mwessendorf" style="color:#3b73af;; color: #3b73af; text-decoration: none">Matthias Wessendorf</a> <strong>updated</strong> an issue </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="email-content-container" style="padding: 0px; border-collapse: collapse; padding: 0 20px">
<table id="email-content-table" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; border-spacing: 0; border-collapse: separate">
<tbody>
<tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<td class="email-content-rounded-top mobile-expand" style="padding: 0px; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 15px; background-color: #ffffff; border-left: 1px solid #cccccc; border-top: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom: 0; border-top-right-radius: 5px; border-top-left-radius: 5px; height: 10px; line-height: 10px; padding: 0 15px 0 16px; mso-line-height-rule: exactly" height="10" bgcolor="#ffffff"> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0px; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff">
<table class="page-title-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td class="page-title-pattern-first-line " style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; padding-top: 10px"> <a href="https://issues.jboss.org/browse/AGPUSH" style="color: #3b73af; text-decoration: none">AeroGear Push</a> / <a href="https://issues.jboss.org/browse/AGPUSH-2067" style="color: #3b73af; text-decoration: none"><img src="cid:jira-generated-image-avatar-1aeb4372-8412-4e7c-8ce5-40d20ca013c7" height="16" width="16" border="0" align="absmiddle" alt="Feature Request" style="vertical-align: text-bottom"></a> <a href="https://issues.jboss.org/browse/AGPUSH-2067" style="color: #3b73af; text-decoration: none">AGPUSH-2067</a> </td>
</tr>
<tr>
<td style="vertical-align: top;; padding: 0px; border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: 30px; mso-line-height-rule: exactly" class="page-title-pattern-header-container"> <span class="page-title-pattern-header" style="font-family: Arial, sans-serif; padding: 0; font-size: 20px; line-height: 30px; mso-text-raise: 2px; mso-line-height-rule: exactly; vertical-align: middle"> <a href="https://issues.jboss.org/browse/AGPUSH-2067" style="color: #3b73af; text-decoration: none">HornetQ IllegalStateException: Invalid messageNumber </a> </span> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand wrapper-special-margin" style="padding: 0px; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff; padding-top: 10px; padding-bottom: 5px" bgcolor="#ffffff">
<table class="keyvalue-table" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">Change By:</th>
<td style="padding: 0px; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top"> <a class="user-hover" rel="mwessendorf" id="email_mwessendorf" href="https://issues.jboss.org/secure/ViewProfile.jspa?name=mwessendorf" style="color:#3b73af;; color: #3b73af; text-decoration: none">Matthias Wessendorf</a> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand issue-description-container" style="padding: 0px; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff; padding-top: 5px; padding-bottom: 10px" bgcolor="#ffffff">
<table class="text-paragraph-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 2px">
<tbody>
<tr>
<td class="text-paragraph-pattern-container mobile-resize-text " style="padding: 0px; border-collapse: collapse; padding: 0 0 10px 0"> <span class="diffcontext">Under high load, we notice OOME from HornetQ and EJB layer. We need to investigate more, and do log file analytics to see what's really causing the problems.<br><br>With our tests, it's fairly easy to reproduce the</span> <span class="diffremovedchars" style="background-color: #ffe7e7; text-decoration:line-through;"> <br>I see IllegalStateExceptions</span> <span class="diffaddedchars" style="background-color:#ddfade;"> "IllegalStateException</span> <span class="diffcontext">:</span> <span class="diffaddedchars" style="background-color:#ddfade;"> Invalid messageNumber" issue, like:</span> <span class="diffcontext"><br>{code}<br>09:58:19,192 WARN [org.hornetq.core.server] (Thread-23 (HornetQ-server-HornetQServerImpl::serverUUID=938696ea-0959-11e7-a89e-7f7ac77e215d-378692510)) Invalid messageNumber passed = PagePositionImpl [pageNr=146, messageNr=332, recordID=-1] on null: java.lang.IllegalStateException: Invalid messageNumber passed = PagePositionImpl [pageNr=146, messageNr=332, recordID=-1] on null<br> at org.hornetq.core.paging.cursor.impl.PageCursorProviderImpl.getMessage(PageCursorProviderImpl.java:123) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]<br> at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl.queryMessage(PageSubscriptionImpl.java:609) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]<br> at org.hornetq.core.paging.cursor.PagedReferenceImpl.getPagedMessage(PagedReferenceImpl.java:67) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]<br> at org.hornetq.core.paging.cursor.PagedReferenceImpl.getMessage(PagedReferenceImpl.java:53) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]<br> at org.hornetq.core.server.impl.QueueImpl.decrementRefCount(QueueImpl.java:2934) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]<br> at org.hornetq.core.server.impl.QueueImpl.access$1000(QueueImpl.java:93) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]<br> at org.hornetq.core.server.impl.QueueImpl$RefsOperation.afterCommit(QueueImpl.java:2899) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]<br> at org.hornetq.core.transaction.impl.TransactionImpl.afterCommit(TransactionImpl.java:633) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]<br> at org.hornetq.core.transaction.impl.TransactionImpl.access$100(TransactionImpl.java:37) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]<br> at org.hornetq.core.transaction.impl.TransactionImpl$2.done(TransactionImpl.java:332) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]<br> at org.hornetq.core.persistence.impl.journal.OperationContextImpl$1.run(OperationContextImpl.java:276) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]<br> at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105) [hornetq-core-client-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]<br> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_121]<br> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_121]<br> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_121]<br>{code}<br><br></span> <span class="diffaddedchars" style="background-color:#ddfade;"><br>afterwards, than the Arjuna transaction reaper kicks in, and eventually, looks like hornet is gone:<br>https://gist.github.com/matzew/37816eafdf236e0f590365d21f267dcd#file-some_stack-txt-L50-L53<br><br></span> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0px; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff">
<table id="actions-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px">
<tbody>
<tr>
<td id="actions-pattern-container" valign="middle" style="padding: 0px; border-collapse: collapse; padding: 10px 0 10px 24px; vertical-align: middle; padding-left: 0">
<table align="left" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td class="actions-pattern-action-icon-container" style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0px; vertical-align: middle"> <a href="https://issues.jboss.org/browse/AGPUSH-2067#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none"> <img class="actions-pattern-action-icon-image" src="cid:jira-generated-image-static-comment-icon-5de09b0c-a7bd-4acd-a903-7f90e2f60880" alt="Add Comment" title="Add Comment" height="16" width="16" border="0" style="vertical-align: middle"> </a> </td>
<td class="actions-pattern-action-text-container" style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left: 5px"> <a href="https://issues.jboss.org/browse/AGPUSH-2067#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none">Add Comment</a> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<tr>
<td class="email-content-rounded-bottom mobile-expand" style="padding: 0px; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 5px; line-height: 5px; background-color: #ffffff; border-top: 0; border-left: 1px solid #cccccc; border-bottom: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom-right-radius: 5px; border-bottom-left-radius: 5px; mso-line-height-rule: exactly" height="5" bgcolor="#ffffff"> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="footer-pattern" style="padding: 0px; border-collapse: collapse; padding: 12px 20px">
<table id="footer-pattern-container" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td id="footer-pattern-text" class="mobile-resize-text" width="100%" style="padding: 0px; border-collapse: collapse; color: #999999; font-size: 12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px"> This message was sent by Atlassian JIRA <span id="footer-build-information">(v7.2.3#72005-<span title="73be91d2b96fc29303a7eb6820acf420e5d0ed65" data-commit-id="73be91d2b96fc29303a7eb6820acf420e5d0ed65}">sha1:73be91d</span>)</span> </td>
<td id="footer-pattern-logo-desktop-container" valign="top" style="padding: 0px; border-collapse: collapse; padding-left: 20px; vertical-align: top">
<table style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td id="footer-pattern-logo-desktop-padding" style="padding: 0px; border-collapse: collapse; padding-top: 3px"> <img id="footer-pattern-logo-desktop" src="cid:jira-generated-image-static-footer-desktop-logo-ff09dc9d-2d23-4cec-8bc5-85f0c3416014" alt="Atlassian logo" title="Atlassian logo" width="169" height="36" class="image_fix"> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table>
</body>
</html>