[JBoss JIRA] Created: (NETTY-230) 'bad handshake record MAC' error and IndexOutOfBoundsException on SSL closure

[Trustin Lee (JIRA)]

'bad handshake record MAC' error and IndexOutOfBoundsException on SSL closure
-----------------------------------------------------------------------------

                 Key: NETTY-230
                 URL: https://jira.jboss.org/jira/browse/NETTY-230
             Project: Netty
          Issue Type: Bug
          Components: Handler
            Reporter: Trustin Lee
            Assignee: Trustin Lee
             Fix For: 3.1.4.GA


The following steps trigger an SSLException and an IndexOutOfBoundsException randomly:

        // 1) An SSL packet is received from the wire.
        // 2) SslHandler.decode() deciphers the packet and calls the user code.
        // 3) The user closes the channel in the same thread.
        // 4) The same thread triggers a channelDisconnected() event.
        // 5) FrameDecoder.cleanup() is called, and it calls SslHandler.decode().
        // 6) SslHandler.decode() will feed the same packet with what was
        //    deciphered at the step 2 again if the readerIndex was not advanced
        //    before calling the user code.

And here's an example stack trace:

javax.net.ssl.SSLHandshakeException: bad handshake record MAC
	at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
	at com.sun.net.ssl.internal.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1390)
	at com.sun.net.ssl.internal.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1358)
	at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:902)
	at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:810)
	at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:686)
	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
	at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:748)
	at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:477)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:282)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
	at org.jboss.netty.handler.ssl.SslHandler.channelDisconnected(SslHandler.java:401)
	at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:502)
	at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:584)
	at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:119)
	at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
	at org.jboss.netty.channel.Channels.close(Channels.java:1081)
	at org.jboss.netty.handler.ssl.SslHandler$ClosingChannelFutureListener.operationComplete(SslHandler.java:906)
	at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:366)
	at org.jboss.netty.channel.DefaultChannelFuture.addListener(DefaultChannelFuture.java:139)
	at org.jboss.netty.handler.ssl.SslHandler.closeOutboundAndChannel(SslHandler.java:874)
	at org.jboss.netty.handler.ssl.SslHandler.handleDownstream(SslHandler.java:348)
	at org.jboss.netty.channel.Channels.close(Channels.java:1065)
	at org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:178)
	at org.jboss.netty.channel.ChannelFutureListener$1.operationComplete(ChannelFutureListener.java:46)
	at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:366)
	at org.jboss.netty.channel.DefaultChannelFuture.addListener(DefaultChannelFuture.java:139)
	........ USER CODE that initiates the closure ........
	at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:366)
	at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:352)
	at org.jboss.netty.channel.DefaultChannelFuture.setSuccess(DefaultChannelFuture.java:303)
	at org.jboss.netty.handler.ssl.SslHandler.setHandshakeSuccess(SslHandler.java:844)
	at org.jboss.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:689)
	at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:761)
	at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:477)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:282)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:214)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332)
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
	at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)
	at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)
java.lang.IndexOutOfBoundsException
	at org.jboss.netty.buffer.AbstractChannelBuffer.skipBytes(AbstractChannelBuffer.java:343)
	at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:479)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:282)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:214)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332)
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
	at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)
	at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)

The IndexOutOfBoundsException seems to be triggered by the first exception for some reason.  To fix this issue, we have to advance the buffer's readerIndex before calling unwrap().

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira