Logstash TCP-Input throws "Bad record MAC" when trying to connect with WildFly over SSL/TLS ------------------------------------------------------------------------------------------- Key: WFLY-7338 URL: https://issues.jboss.org/browse/WFLY-7338 Project: WildFly Issue Type: Bug Components: Logging Reporter: Patrick Kleindienst Assignee: James Perkins Labels: jboss, logging, ssl I use the jboss-logmanger-ext library for transfering log records to Logstash over a secure socket. For that purpose, my Logstash TCP-Input config authenticates with WildFly by means of a self-signed certificate. However, some time after SSL handshake has started, the following exception is thrown: {code:java} LogManager error of type FLUSH_FAILURE: Error on flush java.net.SocketException: Socket is closed at sun.security.ssl.SSLSocketImpl.getOutputStream(SSLSocketImpl.java:2240) at org.jboss.logmanager.handlers.TcpOutputStream.flush(TcpOutputStream.java:210) at org.jboss.logmanager.handlers.UninterruptibleOutputStream.flush(UninterruptibleOutputStream.java:110) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) at org.jboss.logmanager.ext.handlers.SocketHandler.safeFlush(SocketHandler.java:340) at org.jboss.logmanager.ext.handlers.SocketHandler.flush(SocketHandler.java:169) at org.jboss.logmanager.ExtHandler.doPublish(ExtHandler.java:104) at org.jboss.logmanager.ext.handlers.SocketHandler.doPublish(SocketHandler.java:159) at org.jboss.logmanager.ExtHandler.publish(ExtHandler.java:76) at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:314) at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:322) at org.jboss.logmanager.Logger.logRaw(Logger.java:850) at org.jboss.logmanager.Logger.log(Logger.java:596) at org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) at org.jboss.stdio.WriterOutputStream.finish(WriterOutputStream.java:143) at org.jboss.stdio.WriterOutputStream.flush(WriterOutputStream.java:164) at java.io.PrintStream.write(PrintStream.java:482) at org.jboss.stdio.StdioContext$DelegatingPrintStream.write(StdioContext.java:264) at java.io.PrintStream.write(PrintStream.java:480) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291) at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104) at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185) at java.io.PrintStream.newLine(PrintStream.java:546) at java.io.PrintStream.println(PrintStream.java:696) at sun.misc.HexDumpEncoder.encodeLineSuffix(HexDumpEncoder.java:116) at sun.misc.CharacterEncoder.encodeBuffer(CharacterEncoder.java:297) at sun.security.ssl.CipherBox.encrypt(CipherBox.java:306) at sun.security.ssl.OutputRecord.encrypt(OutputRecord.java:264) at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:859) at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:847) at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) at org.jboss.logmanager.handlers.TcpOutputStream.write(TcpOutputStream.java:182) at org.jboss.logmanager.handlers.UninterruptibleOutputStream.write(UninterruptibleOutputStream.java:84) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) at org.jboss.logmanager.ext.handlers.SocketHandler.safeFlush(SocketHandler.java:340) at org.jboss.logmanager.ext.handlers.SocketHandler.flush(SocketHandler.java:169) at org.jboss.logmanager.ExtHandler.doPublish(ExtHandler.java:104) at org.jboss.logmanager.ext.handlers.SocketHandler.doPublish(SocketHandler.java:159) at org.jboss.logmanager.ExtHandler.publish(ExtHandler.java:76) at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:314) at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:322) at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:322) at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:322) at org.jboss.logmanager.Logger.logRaw(Logger.java:850) at org.jboss.logmanager.Logger.log(Logger.java:802) at org.jboss.logging.JBossLogManagerLogger.doLogf(JBossLogManagerLogger.java:53) at org.jboss.logging.Logger.logf(Logger.java:2398) at org.jboss.msc.service.ServiceLogger_$logger.greeting(ServiceLogger_$logger.java:65) at org.jboss.msc.service.ServiceContainerImpl.<clinit>(ServiceContainerImpl.java:93) at org.jboss.msc.service.ServiceContainer$Factory.create(ServiceContainer.java:258) at org.jboss.as.server.BootstrapImpl$ShutdownHook.register(BootstrapImpl.java:214) {code} On the Logstash side, the following error message appears in the logs: {code} :message=>"An error occurred. Closing connection", :exception=>#<IOError: bad record MAC> {code} Afterwards, WildFly hangs forever without deploying my webapp or doing anything else. Before that happens, the handshake goes through these phases: * *** ClientHello, TLSv1.2 * *** ServerHello, TLSv1.2 * %% Initialized: [Session-1, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256] * Found trusted certificate * *** ECDH ServerKeyExchange * *** ServerHelloDone * *** ECDHClientKeyExchange * SESSION KEYGEN: * CONNECTION KEYGEN: * *** Finished When disabling SSL both on WildFly and Logstash side, everything works fine.