Under high load, we notice OOME from HornetQ and EJB layer. We need to investigate more, and do log file analytics to see what's really causing the problems .
With our tests, it's fairly easy to reproduce the
I see IllegalStateExceptions:
{code}
09:58:19,192 WARN [org.hornetq.core.server] (Thread-23 (HornetQ-server-HornetQServerImpl::serverUUID=938696ea-0959-11e7-a89e-7f7ac77e215d-378692510)) Invalid messageNumber passed = PagePositionImpl [pageNr=146, messageNr=332, recordID=-1] on null: java.lang.IllegalStateException: Invalid messageNumber passed = PagePositionImpl [pageNr=146, messageNr=332, recordID=-1] on null
at org.hornetq.core.paging.cursor.impl.PageCursorProviderImpl.getMessage(PageCursorProviderImpl.java:123) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]
at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl.queryMessage(PageSubscriptionImpl.java:609) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]
at org.hornetq.core.paging.cursor.PagedReferenceImpl.getPagedMessage(PagedReferenceImpl.java:67) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]
at org.hornetq.core.paging.cursor.PagedReferenceImpl.getMessage(PagedReferenceImpl.java:53) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]
at org.hornetq.core.server.impl.QueueImpl.decrementRefCount(QueueImpl.java:2934) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]
at org.hornetq.core.server.impl.QueueImpl.access$1000(QueueImpl.java:93) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]
at org.hornetq.core.server.impl.QueueImpl$RefsOperation.afterCommit(QueueImpl.java:2899) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]
at org.hornetq.core.transaction.impl.TransactionImpl.afterCommit(TransactionImpl.java:633) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]
at org.hornetq.core.transaction.impl.TransactionImpl.access$100(TransactionImpl.java:37) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]
at org.hornetq.core.transaction.impl.TransactionImpl$2.done(TransactionImpl.java:332) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]
at org.hornetq.core.persistence.impl.journal.OperationContextImpl$1.run(OperationContextImpl.java:276) [hornetq-server-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]
at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105) [hornetq-core-client-2.3.25.SP14-redhat-1.jar:2.3.25.SP14-redhat-1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_121]
{code}
|
|
