[JBoss JIRA] (WFCORE-1301) IO Subsystem improperly shuts down the worker
by Tomaz Cerar (JIRA)
Tomaz Cerar created WFCORE-1301:
-----------------------------------
Summary: IO Subsystem improperly shuts down the worker
Key: WFCORE-1301
URL: https://issues.jboss.org/browse/WFCORE-1301
Project: WildFly Core
Issue Type: Bug
Reporter: Tomaz Cerar
Assignee: Tomaz Cerar
Fix For: 2.0.7.Final
The stop operation of the I/O subsystem's worker service should be asynchronous, and should not complete until the worker is fully shut down.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 3 months
[JBoss JIRA] (WFCORE-1300) RejectedExecutionException: XNIO007007 on ER4 shutdown
by Tomaz Cerar (JIRA)
Tomaz Cerar created WFCORE-1300:
-----------------------------------
Summary: RejectedExecutionException: XNIO007007 on ER4 shutdown
Key: WFCORE-1300
URL: https://issues.jboss.org/browse/WFCORE-1300
Project: WildFly Core
Issue Type: Bug
Components: Domain Management, Remoting, Server
Reporter: Tomaz Cerar
Assignee: Tomaz Cerar
Priority: Blocker
Fix For: 2.0.7.Final
WFLY 10 CR5 logs Error message on shutdown (both standalone and domain mode). This happens regularly, platforms like Fedora 23, RHEL 6, RHEL 7. WFLY 10 CR5 does not have clean start & stop.
*reproduce*
\# start WFLY 10 CR5
{noformat}
$ ./standalone.sh
...
10:30:46,536 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 10.0.0.CR5 (WildFly Core 2.0.5.Final) started in 3999ms - Started 267 of 569 services (386 services are lazy, passive or on-demand)
{noformat}
\# shutdown via CLI
{noformat}
$ ./jboss-cli.sh -c shutdown
{noformat}
\# check console where WFLY 10 CR5 was started
{noformat}
10:30:53,937 ERROR [org.jboss.remoting.resource] (MSC service thread 1-3) Close action for Connection handler for Remoting connection 4181b134 to /127.0.0.1:51703 failed to execute (resource may be left in an indeterminate state): java.util.concurrent.RejectedExecutionException: XNIO007007: Thread is terminating
at org.xnio.nio.WorkerThread.execute(WorkerThread.java:588)
at org.jboss.remoting3.remote.RemoteConnection$RemoteWriteListener.send(RemoteConnection.java:327)
at org.jboss.remoting3.remote.RemoteConnection.send(RemoteConnection.java:142)
at org.jboss.remoting3.remote.RemoteConnectionHandler.sendCloseRequestBody(RemoteConnectionHandler.java:289)
at org.jboss.remoting3.remote.RemoteConnectionHandler.sendCloseRequestBody(RemoteConnectionHandler.java:278)
at org.jboss.remoting3.remote.RemoteConnectionHandler.sendCloseRequest(RemoteConnectionHandler.java:235)
at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAction(RemoteConnectionHandler.java:413)
at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:372)
at org.jboss.remoting3.ConnectionImpl.closeAction(ConnectionImpl.java:54)
at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:372)
at org.jboss.remoting3.EndpointImpl.closeAction(EndpointImpl.java:225)
at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:372)
at org.jboss.as.remoting.EndpointService.stop(EndpointService.java:98)
at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
{noformat}
10:30:53,936 ERROR [org.jboss.remoting.resource] (Remoting "rs:MANAGEMENT" task-8) Close action for Channel ID 1158edde (inbound) of Remoting connection 4181b134 to /127.0.0.1:51703 failed to execute (resource may be left in an indeterminate state): java.util.concurrent.RejectedExecutionException: XNIO007007: Thread is terminating
at org.xnio.nio.WorkerThread.execute(WorkerThread.java:588)
at org.jboss.remoting3.remote.RemoteConnection$RemoteWriteListener.send(RemoteConnection.java:327)
at org.jboss.remoting3.remote.RemoteConnection.send(RemoteConnection.java:138)
at org.jboss.remoting3.remote.RemoteConnectionChannel.closeReadsAndWrites(RemoteConnectionChannel.java:261)
at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:534)
at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:372)
at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAllChannels(RemoteConnectionHandler.java:437)
at org.jboss.remoting3.remote.RemoteConnectionHandler.handleConnectionClose(RemoteConnectionHandler.java:114)
at org.jboss.remoting3.remote.RemoteReadListener$1$1.run(RemoteReadListener.java:57)
at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:717)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 3 months
[JBoss JIRA] (WFLY-5480) SPEC JMS 2007 benchmark fails with AMQ214013: Failed to decode packet
by Jason Greene (JIRA)
[ https://issues.jboss.org/browse/WFLY-5480?page=com.atlassian.jira.plugin.... ]
Jason Greene commented on WFLY-5480:
------------------------------------
[~flavia.rainone] try updating to latest wildfly, we are currently using wildly-010 which required changes in our integration to be compatible with. I would wager a good guess that latest 1.1.x is probably drop-in able on wf master (unless something else changed).
> SPEC JMS 2007 benchmark fails with AMQ214013: Failed to decode packet
> ---------------------------------------------------------------------
>
> Key: WFLY-5480
> URL: https://issues.jboss.org/browse/WFLY-5480
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Ondřej Kalman
> Assignee: Flavia Rainone
> Priority: Blocker
>
> But I have another problem, when I removed trace logs from config I'm not able to run benchmark on localhost to the end, because clients starts getting :
> AMQ214013: Failed to decode packet
> java.lang.IndexOutOfBoundsException: readerIndex: 2130706436 (expected: 0 <= readerIndex <= writerIndex(2943))
> at io.netty.buffer.AbstractByteBuf.readerIndex(AbstractByteBuf.java:73)
> at io.netty.buffer.WrappedByteBuf.readerIndex(WrappedByteBuf.java:99)
> at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readerIndex(ChannelBufferWrapper.java:405)
> at org.apache.activemq.artemis.core.message.impl.MessageImpl.decode(MessageImpl.java:1052)
> at org.apache.activemq.artemis.core.message.impl.MessageImpl.decodeFromBuffer(MessageImpl.java:459)
> at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.SessionReceiveMessage.decode(SessionReceiveMessage.java:94)
> at org.apache.activemq.artemis.core.protocol.ClientPacketDecoder.decode(ClientPacketDecoder.java:42)
> at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:371)
> at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1374)
> at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteCha
> nnel.java:131)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at java.lang.Thread.run(Thread.java:745)
> and
> Thread-9 (ActiveMQ-client-global-threads-1266561810): Uncaught exception.
> java.lang.NumberFormatException: null
> at java.lang.Integer.parseInt(Integer.java:542)
> at java.lang.Integer.valueOf(Integer.java:766)
> at org.apache.activemq.artemis.utils.TypedProperties.getIntProperty(TypedProperties.java:280)
> at org.apache.activemq.artemis.core.message.impl.MessageImpl.getIntProperty(MessageImpl.java:811)
> at org.apache.activemq.artemis.jms.client.ActiveMQMessage.getIntProperty(ActiveMQMessage.java:578)
> at org.spec.jms.agents.SPECWorkerThread.receivedMessage(SPECWorkerThread.java:849)
> at org.spec.jms.agents.SPECWorkerThread.onMessage(SPECWorkerThread.java:820)
> at org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:100)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1089)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:47)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1224)
> at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Thread-3 (ActiveMQ-client-global-threads-1266561810): Uncaught exception.
> javax.jms.IllegalStateException: AMQ119027: Could not find reference on consumer ID=0, messageId = 104,833 queue = 127\.0\.0\.1_VM1_SPAgent7_0.SP_CallForOffersEH_7_EHID_1PF0
> at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:410)
> at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendACK(ActiveMQSessionContext.java:461)
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.acknowledge(ClientSessionImpl.java:765)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.doAck(ClientConsumerImpl.java:1212)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.flushAcks(ClientConsumerImpl.java:830)
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.flushAcks(ClientSessionImpl.java:1852)
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.commit(ClientSessionImpl.java:501)
> at org.apache.activemq.artemis.core.client.impl.DelegatingSession.commit(DelegatingSession.java:159)
> at org.apache.activemq.artemis.jms.client.ActiveMQSession.commit(ActiveMQSession.java:218)
> at org.spec.jms.eventhandler.sp.SP_CallForOffersEH.handleMessage(SP_CallForOffersEH.java:306)
> at org.spec.jms.agents.SPECWorkerThread.onMessage(SPECWorkerThread.java:821)
> at org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:100)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1089)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:47)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1224)
> at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecut
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 3 months
[JBoss JIRA] (WFLY-5480) SPEC JMS 2007 benchmark fails with AMQ214013: Failed to decode packet
by Flavia Rainone (JIRA)
[ https://issues.jboss.org/browse/WFLY-5480?page=com.atlassian.jira.plugin.... ]
Flavia Rainone commented on WFLY-5480:
--------------------------------------
[~clebert] I am running the tests with activeMQ updated, this is the project I'm using: git://github.com/rh-messaging/jboss-activemq-artemis.git, branch jboss-1.1.0-x
I've been updating regularly... just updated with the latest commits, and the error still occurs.
> SPEC JMS 2007 benchmark fails with AMQ214013: Failed to decode packet
> ---------------------------------------------------------------------
>
> Key: WFLY-5480
> URL: https://issues.jboss.org/browse/WFLY-5480
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Ondřej Kalman
> Assignee: Flavia Rainone
> Priority: Blocker
>
> But I have another problem, when I removed trace logs from config I'm not able to run benchmark on localhost to the end, because clients starts getting :
> AMQ214013: Failed to decode packet
> java.lang.IndexOutOfBoundsException: readerIndex: 2130706436 (expected: 0 <= readerIndex <= writerIndex(2943))
> at io.netty.buffer.AbstractByteBuf.readerIndex(AbstractByteBuf.java:73)
> at io.netty.buffer.WrappedByteBuf.readerIndex(WrappedByteBuf.java:99)
> at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readerIndex(ChannelBufferWrapper.java:405)
> at org.apache.activemq.artemis.core.message.impl.MessageImpl.decode(MessageImpl.java:1052)
> at org.apache.activemq.artemis.core.message.impl.MessageImpl.decodeFromBuffer(MessageImpl.java:459)
> at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.SessionReceiveMessage.decode(SessionReceiveMessage.java:94)
> at org.apache.activemq.artemis.core.protocol.ClientPacketDecoder.decode(ClientPacketDecoder.java:42)
> at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:371)
> at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1374)
> at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteCha
> nnel.java:131)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at java.lang.Thread.run(Thread.java:745)
> and
> Thread-9 (ActiveMQ-client-global-threads-1266561810): Uncaught exception.
> java.lang.NumberFormatException: null
> at java.lang.Integer.parseInt(Integer.java:542)
> at java.lang.Integer.valueOf(Integer.java:766)
> at org.apache.activemq.artemis.utils.TypedProperties.getIntProperty(TypedProperties.java:280)
> at org.apache.activemq.artemis.core.message.impl.MessageImpl.getIntProperty(MessageImpl.java:811)
> at org.apache.activemq.artemis.jms.client.ActiveMQMessage.getIntProperty(ActiveMQMessage.java:578)
> at org.spec.jms.agents.SPECWorkerThread.receivedMessage(SPECWorkerThread.java:849)
> at org.spec.jms.agents.SPECWorkerThread.onMessage(SPECWorkerThread.java:820)
> at org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:100)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1089)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:47)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1224)
> at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Thread-3 (ActiveMQ-client-global-threads-1266561810): Uncaught exception.
> javax.jms.IllegalStateException: AMQ119027: Could not find reference on consumer ID=0, messageId = 104,833 queue = 127\.0\.0\.1_VM1_SPAgent7_0.SP_CallForOffersEH_7_EHID_1PF0
> at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:410)
> at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendACK(ActiveMQSessionContext.java:461)
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.acknowledge(ClientSessionImpl.java:765)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.doAck(ClientConsumerImpl.java:1212)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.flushAcks(ClientConsumerImpl.java:830)
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.flushAcks(ClientSessionImpl.java:1852)
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.commit(ClientSessionImpl.java:501)
> at org.apache.activemq.artemis.core.client.impl.DelegatingSession.commit(DelegatingSession.java:159)
> at org.apache.activemq.artemis.jms.client.ActiveMQSession.commit(ActiveMQSession.java:218)
> at org.spec.jms.eventhandler.sp.SP_CallForOffersEH.handleMessage(SP_CallForOffersEH.java:306)
> at org.spec.jms.agents.SPECWorkerThread.onMessage(SPECWorkerThread.java:821)
> at org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:100)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1089)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:47)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1224)
> at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecut
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 3 months
[JBoss JIRA] (WFLY-6001) ISPN-6007 workaround fails for caches using optimistic locking
by Paul Ferraro (JIRA)
Paul Ferraro created WFLY-6001:
----------------------------------
Summary: ISPN-6007 workaround fails for caches using optimistic locking
Key: WFLY-6001
URL: https://issues.jboss.org/browse/WFLY-6001
Project: WildFly
Issue Type: Bug
Components: Clustering
Affects Versions: 10.0.0.CR5
Reporter: Paul Ferraro
Assignee: Paul Ferraro
Priority: Critical
Fix For: 10.0.0.Final
Ultimately, this workaround should be removed following the upgrade to Infinispan 8.1.1.Final - which incorporates a fix for ISPN-6007.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 3 months
[JBoss JIRA] (WFLY-5480) SPEC JMS 2007 benchmark fails with AMQ214013: Failed to decode packet
by Flavia Rainone (JIRA)
[ https://issues.jboss.org/browse/WFLY-5480?page=com.atlassian.jira.plugin.... ]
Flavia Rainone commented on WFLY-5480:
--------------------------------------
Replacing MessageImpl.getEncodedBuffer implementation by:
{{ActiveMQBuffer copied = buff.copy(0, buff.capacity());
copied.setIndex(0, endOfMessagePosition);
buffer.setIndex(0, endOfMessagePosition);
bufferUsed = true;
ActiveMQBuffer oldBuffer = buffer;
buffer = copied;
return oldBuffer;}}
Makes the bug go away. With the code above, the buffer returned is not kept, meaning that there is no window of time where one thread is receiving the buffer and being able to perform operations on it while the other thread is still going to copy the buffer. The con of the above is that it is not a reasonable implementation for getEncodedBuffer, as it creates an extra copy for every encoded message.
Still, I think that this is an indicator that we are dealing with a race condition scenario here.
> SPEC JMS 2007 benchmark fails with AMQ214013: Failed to decode packet
> ---------------------------------------------------------------------
>
> Key: WFLY-5480
> URL: https://issues.jboss.org/browse/WFLY-5480
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Ondřej Kalman
> Assignee: Flavia Rainone
> Priority: Blocker
>
> But I have another problem, when I removed trace logs from config I'm not able to run benchmark on localhost to the end, because clients starts getting :
> AMQ214013: Failed to decode packet
> java.lang.IndexOutOfBoundsException: readerIndex: 2130706436 (expected: 0 <= readerIndex <= writerIndex(2943))
> at io.netty.buffer.AbstractByteBuf.readerIndex(AbstractByteBuf.java:73)
> at io.netty.buffer.WrappedByteBuf.readerIndex(WrappedByteBuf.java:99)
> at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readerIndex(ChannelBufferWrapper.java:405)
> at org.apache.activemq.artemis.core.message.impl.MessageImpl.decode(MessageImpl.java:1052)
> at org.apache.activemq.artemis.core.message.impl.MessageImpl.decodeFromBuffer(MessageImpl.java:459)
> at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.SessionReceiveMessage.decode(SessionReceiveMessage.java:94)
> at org.apache.activemq.artemis.core.protocol.ClientPacketDecoder.decode(ClientPacketDecoder.java:42)
> at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:371)
> at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1374)
> at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteCha
> nnel.java:131)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at java.lang.Thread.run(Thread.java:745)
> and
> Thread-9 (ActiveMQ-client-global-threads-1266561810): Uncaught exception.
> java.lang.NumberFormatException: null
> at java.lang.Integer.parseInt(Integer.java:542)
> at java.lang.Integer.valueOf(Integer.java:766)
> at org.apache.activemq.artemis.utils.TypedProperties.getIntProperty(TypedProperties.java:280)
> at org.apache.activemq.artemis.core.message.impl.MessageImpl.getIntProperty(MessageImpl.java:811)
> at org.apache.activemq.artemis.jms.client.ActiveMQMessage.getIntProperty(ActiveMQMessage.java:578)
> at org.spec.jms.agents.SPECWorkerThread.receivedMessage(SPECWorkerThread.java:849)
> at org.spec.jms.agents.SPECWorkerThread.onMessage(SPECWorkerThread.java:820)
> at org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:100)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1089)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:47)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1224)
> at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Thread-3 (ActiveMQ-client-global-threads-1266561810): Uncaught exception.
> javax.jms.IllegalStateException: AMQ119027: Could not find reference on consumer ID=0, messageId = 104,833 queue = 127\.0\.0\.1_VM1_SPAgent7_0.SP_CallForOffersEH_7_EHID_1PF0
> at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:410)
> at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendACK(ActiveMQSessionContext.java:461)
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.acknowledge(ClientSessionImpl.java:765)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.doAck(ClientConsumerImpl.java:1212)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.flushAcks(ClientConsumerImpl.java:830)
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.flushAcks(ClientSessionImpl.java:1852)
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.commit(ClientSessionImpl.java:501)
> at org.apache.activemq.artemis.core.client.impl.DelegatingSession.commit(DelegatingSession.java:159)
> at org.apache.activemq.artemis.jms.client.ActiveMQSession.commit(ActiveMQSession.java:218)
> at org.spec.jms.eventhandler.sp.SP_CallForOffersEH.handleMessage(SP_CallForOffersEH.java:306)
> at org.spec.jms.agents.SPECWorkerThread.onMessage(SPECWorkerThread.java:821)
> at org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:100)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1089)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:47)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1224)
> at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecut
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 3 months
[JBoss JIRA] (WFLY-5480) SPEC JMS 2007 benchmark fails with AMQ214013: Failed to decode packet
by Flavia Rainone (JIRA)
[ https://issues.jboss.org/browse/WFLY-5480?page=com.atlassian.jira.plugin.... ]
Flavia Rainone edited comment on WFLY-5480 at 1/14/16 2:11 PM:
---------------------------------------------------------------
The wrong size number appears at position 13 of the buffer after the buffer is copied, at MessageImpl.getEncodedBuffer. This wrong size is sent to the client and causes the Exception seen in the description above.
This method, MessageImpl.getEncodedBuffer, is invoked concurrently by two separate threads, both are sending a SessionReceiveMessage to the client (SessionReceiveMessage.encode).
In one of the threads, the buffer is sent and usedBuffer is set to true. In the other thread, after checking that usedBuffer is true, a new copy of the buffer is taken. The problem is in the copied buffer. While the original buffer, right after the copy, still contains the correct size at position 13, the new buffer, the copy, contains a garbage int at position 13.
I am trying to find whereas the problem is in netty, responsible for doing the copy, or is a race condition involving another thread changing the buffer contents, even if temporarily, and causing the copy to go bad.
Maybe the messaging guys have a more specific opinion on what is causing this, let me know if that's the case.
was (Author: flavia.rainone):
The wrong size number appears at position 13 of the buffer after the buffer is copied, at MessageImpl.getEncodedBuffer. This wrong size is sent to the client and causes the Exception seen in the description above.
This method, MessageImpl.getEncodedBuffer, is invoked concurrently by two separate threads, both are sending a SessionReceiveMessage to the client (SessionReceiveMessage.encode).
In one of the threads, the buffer is sent and usedBuffer is set to true. In the other thread, after checking that usedBuffer is true, a new copy of the buffer is taken. The problem is in the copied buffer. While the original buffer, right after the copy, still contains the correct size at position 13, the new buffer, the copy, contains a garbage int at position 13.
I am trying to find whereas the problem is in netty, responsible for doing the copy, or is a race condition involving another thread changing the buffer contents, even if temporarily, and causing the copy to go bad.
Too bad I haven't found yet where is the native implementation of Unsafe at netty, the class responsible for performing the copy.
Maybe the messaging guys have a more specific opinion on what is causing this, let me know if that's the case.
> SPEC JMS 2007 benchmark fails with AMQ214013: Failed to decode packet
> ---------------------------------------------------------------------
>
> Key: WFLY-5480
> URL: https://issues.jboss.org/browse/WFLY-5480
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Ondřej Kalman
> Assignee: Flavia Rainone
> Priority: Blocker
>
> But I have another problem, when I removed trace logs from config I'm not able to run benchmark on localhost to the end, because clients starts getting :
> AMQ214013: Failed to decode packet
> java.lang.IndexOutOfBoundsException: readerIndex: 2130706436 (expected: 0 <= readerIndex <= writerIndex(2943))
> at io.netty.buffer.AbstractByteBuf.readerIndex(AbstractByteBuf.java:73)
> at io.netty.buffer.WrappedByteBuf.readerIndex(WrappedByteBuf.java:99)
> at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readerIndex(ChannelBufferWrapper.java:405)
> at org.apache.activemq.artemis.core.message.impl.MessageImpl.decode(MessageImpl.java:1052)
> at org.apache.activemq.artemis.core.message.impl.MessageImpl.decodeFromBuffer(MessageImpl.java:459)
> at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.SessionReceiveMessage.decode(SessionReceiveMessage.java:94)
> at org.apache.activemq.artemis.core.protocol.ClientPacketDecoder.decode(ClientPacketDecoder.java:42)
> at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:371)
> at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1374)
> at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteCha
> nnel.java:131)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at java.lang.Thread.run(Thread.java:745)
> and
> Thread-9 (ActiveMQ-client-global-threads-1266561810): Uncaught exception.
> java.lang.NumberFormatException: null
> at java.lang.Integer.parseInt(Integer.java:542)
> at java.lang.Integer.valueOf(Integer.java:766)
> at org.apache.activemq.artemis.utils.TypedProperties.getIntProperty(TypedProperties.java:280)
> at org.apache.activemq.artemis.core.message.impl.MessageImpl.getIntProperty(MessageImpl.java:811)
> at org.apache.activemq.artemis.jms.client.ActiveMQMessage.getIntProperty(ActiveMQMessage.java:578)
> at org.spec.jms.agents.SPECWorkerThread.receivedMessage(SPECWorkerThread.java:849)
> at org.spec.jms.agents.SPECWorkerThread.onMessage(SPECWorkerThread.java:820)
> at org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:100)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1089)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:47)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1224)
> at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Thread-3 (ActiveMQ-client-global-threads-1266561810): Uncaught exception.
> javax.jms.IllegalStateException: AMQ119027: Could not find reference on consumer ID=0, messageId = 104,833 queue = 127\.0\.0\.1_VM1_SPAgent7_0.SP_CallForOffersEH_7_EHID_1PF0
> at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:410)
> at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendACK(ActiveMQSessionContext.java:461)
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.acknowledge(ClientSessionImpl.java:765)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.doAck(ClientConsumerImpl.java:1212)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.flushAcks(ClientConsumerImpl.java:830)
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.flushAcks(ClientSessionImpl.java:1852)
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.commit(ClientSessionImpl.java:501)
> at org.apache.activemq.artemis.core.client.impl.DelegatingSession.commit(DelegatingSession.java:159)
> at org.apache.activemq.artemis.jms.client.ActiveMQSession.commit(ActiveMQSession.java:218)
> at org.spec.jms.eventhandler.sp.SP_CallForOffersEH.handleMessage(SP_CallForOffersEH.java:306)
> at org.spec.jms.agents.SPECWorkerThread.onMessage(SPECWorkerThread.java:821)
> at org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:100)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1089)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:47)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1224)
> at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecut
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 3 months