[jboss-jira] [JBoss JIRA] (WFLY-5480) SPEC JMS 2007 benchmark fails with AMQ214013: Failed to decode packet
Flavia Rainone (JIRA)
issues at jboss.org
Mon Oct 26 10:07:00 EDT 2015
[ https://issues.jboss.org/browse/WFLY-5480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121957#comment-13121957 ]
Flavia Rainone commented on WFLY-5480:
--------------------------------------
I found out the following: the bug is caused by a race condition, probably at org.apache.activemq.artemis.core.message.impl.MessageImpl.
If I put a breakpoint at the line that reads the problematic index (the line that reads the value of 2130706436), the index value is always read correctly with the breakpoint on (the read number varies, but is usually approximately 2280). Would the bug be on the server side, the client would always read an incorrect number no matter if there is a breakpoint or not.
Reading the code of MessageImpl, I saw that there is already some code synchronizing the access to the buffer to handle race conditions involving the problematic buffer... now I wonder if this bug is related.
At the MessagingImpl constructor:
// This MUST be synchronized using the monitor on the other message to prevent it running concurrently
// with getEncodedBuffer(), otherwise can introduce race condition when delivering concurrently to
// many subscriptions and bridging to other nodes in a cluster
synchronized (other) {
In case somebody is wondering what is end of body position about (the value that is sometimes read as being 2130706436), we can see a description at the line that encodes it:
// Clebert: I've started sending this on encoding due to conversions between protocols
// and making sure we are not losing the buffer start position between protocols
this.endOfBodyPosition = bodySize;
A more deep investigation on the side of amq code will be need to find the cause of all this (I'll need to edit the code to dig deeper).
Btw, I am using wildfly master to run the tests.
> SPEC JMS 2007 benchmark fails with AMQ214013: Failed to decode packet
> ---------------------------------------------------------------------
>
> Key: WFLY-5480
> URL: https://issues.jboss.org/browse/WFLY-5480
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Ondřej Kalman
> Assignee: Flavia Rainone
> Priority: Blocker
>
> But I have another problem, when I removed trace logs from config I'm not able to run benchmark on localhost to the end, because clients starts getting :
> AMQ214013: Failed to decode packet
> java.lang.IndexOutOfBoundsException: readerIndex: 2130706436 (expected: 0 <= readerIndex <= writerIndex(2943))
> at io.netty.buffer.AbstractByteBuf.readerIndex(AbstractByteBuf.java:73)
> at io.netty.buffer.WrappedByteBuf.readerIndex(WrappedByteBuf.java:99)
> at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readerIndex(ChannelBufferWrapper.java:405)
> at org.apache.activemq.artemis.core.message.impl.MessageImpl.decode(MessageImpl.java:1052)
> at org.apache.activemq.artemis.core.message.impl.MessageImpl.decodeFromBuffer(MessageImpl.java:459)
> at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.SessionReceiveMessage.decode(SessionReceiveMessage.java:94)
> at org.apache.activemq.artemis.core.protocol.ClientPacketDecoder.decode(ClientPacketDecoder.java:42)
> at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:371)
> at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1374)
> at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteCha
> nnel.java:131)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at java.lang.Thread.run(Thread.java:745)
> and
> Thread-9 (ActiveMQ-client-global-threads-1266561810): Uncaught exception.
> java.lang.NumberFormatException: null
> at java.lang.Integer.parseInt(Integer.java:542)
> at java.lang.Integer.valueOf(Integer.java:766)
> at org.apache.activemq.artemis.utils.TypedProperties.getIntProperty(TypedProperties.java:280)
> at org.apache.activemq.artemis.core.message.impl.MessageImpl.getIntProperty(MessageImpl.java:811)
> at org.apache.activemq.artemis.jms.client.ActiveMQMessage.getIntProperty(ActiveMQMessage.java:578)
> at org.spec.jms.agents.SPECWorkerThread.receivedMessage(SPECWorkerThread.java:849)
> at org.spec.jms.agents.SPECWorkerThread.onMessage(SPECWorkerThread.java:820)
> at org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:100)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1089)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:47)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1224)
> at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Thread-3 (ActiveMQ-client-global-threads-1266561810): Uncaught exception.
> javax.jms.IllegalStateException: AMQ119027: Could not find reference on consumer ID=0, messageId = 104,833 queue = 127\.0\.0\.1_VM1_SPAgent7_0.SP_CallForOffersEH_7_EHID_1PF0
> at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:410)
> at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendACK(ActiveMQSessionContext.java:461)
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.acknowledge(ClientSessionImpl.java:765)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.doAck(ClientConsumerImpl.java:1212)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.flushAcks(ClientConsumerImpl.java:830)
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.flushAcks(ClientSessionImpl.java:1852)
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.commit(ClientSessionImpl.java:501)
> at org.apache.activemq.artemis.core.client.impl.DelegatingSession.commit(DelegatingSession.java:159)
> at org.apache.activemq.artemis.jms.client.ActiveMQSession.commit(ActiveMQSession.java:218)
> at org.spec.jms.eventhandler.sp.SP_CallForOffersEH.handleMessage(SP_CallForOffersEH.java:306)
> at org.spec.jms.agents.SPECWorkerThread.onMessage(SPECWorkerThread.java:821)
> at org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:100)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1089)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:47)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1224)
> at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecut
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
More information about the jboss-jira
mailing list