[undertow-dev] Undertow 1.0.x - Handling of large binary messages is probably broken

[Andrej Golovnin]

Hi Stuart,

I think that the handling of large binary messages is broken in Undertow 1.0.x branch.
If you copy the test classes from the pull request 206 to the 1.0.x branch and
execute the test BinaryEndpointTest, then you should get this exception:

00:51:25,401 ERROR (XNIO-1 I/O-1) [org.xnio.listener] <ChannelListeners.java:94> XNIO001007: A channel event listener threw an exception: java.lang.IndexOutOfBoundsException
	at java.nio.Buffer.checkIndex(Buffer.java:532)
	at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:253)
	at io.undertow.websockets.core.protocol.version07.Masker.mask(Masker.java:54)
	at io.undertow.websockets.core.protocol.version07.Masker.afterRead(Masker.java:67)
	at io.undertow.websockets.core.FixedPayloadFrameSourceChannel.afterRead(FixedPayloadFrameSourceChannel.java:117)
	at io.undertow.websockets.core.FixedPayloadFrameSourceChannel.read(FixedPayloadFrameSourceChannel.java:78)
	at io.undertow.websockets.core.BufferedBinaryMessage$1.handleEvent(BufferedBinaryMessage.java:87)
	at io.undertow.websockets.core.BufferedBinaryMessage$1.handleEvent(BufferedBinaryMessage.java:79)
	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
	at io.undertow.server.protocol.framed.AbstractFramedChannel$FrameReadListener.invokeReadListener(AbstractFramedChannel.java:647)
	at io.undertow.server.protocol.framed.AbstractFramedChannel$FrameReadListener.handleEvent(AbstractFramedChannel.java:620)
	at io.undertow.server.protocol.framed.AbstractFramedChannel$FrameReadListener.handleEvent(AbstractFramedChannel.java:614)
	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
	at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
	at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:87)
	at org.xnio.nio.NioHandle$1.run(NioHandle.java:50)
	at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:552)
	at org.xnio.nio.WorkerThread.run(WorkerThread.java:461)


I was not able to find the cause for this exception. It is possible that my test is just wrong.
In that please let me know, what I'm doing wrong. If the test is OK, then I hope you can find the bug
and fix it before the release of WildFly 8.1.0.

Best regards,
Andrej Golovnin