Under high load, we notice OOME from HornetQ and EJB layer. We need to investigate more, and do log file analytics to see what's really causing the problems.
With our tests, it's fairly easy to reproduce the I see IllegalStateExceptions "IllegalStateException : Invalid messageNumber" issue, like: {code} 09:58:19,192 WARN [org.hornetq.core.server] (Thread-23 (HornetQ-server-HornetQServerImpl::serverUUID=938696ea-0959-11e7-a89e-7f7ac77e215d-378692510)) Invalid messageNumber passed = PagePositionImpl [pageNr=146, messageNr=332, recordID=-1] on null: java.lang.IllegalStateException: Invalid messageNumber passed = PagePositionImpl [pageNr=146, messageNr=332, recordID=-1] on null at org.hornetq.core.paging.cursor.impl.PageCursorProviderImpl.getMessage(PageCursorProviderImpl.java:123) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1] at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl.queryMessage(PageSubscriptionImpl.java:609) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1] at org.hornetq.core.paging.cursor.PagedReferenceImpl.getPagedMessage(PagedReferenceImpl.java:67) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1] at org.hornetq.core.paging.cursor.PagedReferenceImpl.getMessage(PagedReferenceImpl.java:53) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1] at org.hornetq.core.server.impl.QueueImpl.decrementRefCount(QueueImpl.java:2934) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1] at org.hornetq.core.server.impl.QueueImpl.access$1000(QueueImpl.java:93) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1] at org.hornetq.core.server.impl.QueueImpl$RefsOperation.afterCommit(QueueImpl.java:2899) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1] at org.hornetq.core.transaction.impl.TransactionImpl.afterCommit(TransactionImpl.java:633) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1] at org.hornetq.core.transaction.impl.TransactionImpl.access$100(TransactionImpl.java:37) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1] at org.hornetq.core.transaction.impl.TransactionImpl$2.done(TransactionImpl.java:332) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1] at org.hornetq.core.persistence.impl.journal.OperationContextImpl$1.run(OperationContextImpl.java:276) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1] at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105) [hornetq-core-client-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_121] {code}
afterwards, than the Arjuna transaction reaper kicks in, and eventually, looks like hornet is gone: https://gist.github.com/matzew/37816eafdf236e0f590365d21f267dcd#file-some_stack-txt-L50-L53
|
|