mod_cluster and HTTP2 enabled (the default) not working ------------------------------------------------------- Key: WFLY-11639 URL: https://issues.jboss.org/browse/WFLY-11639 Project: WildFly Issue Type: Bug Components: mod_cluster Affects Versions: 15.0.1.Final Environment: Mac OS X 10.14.3 Tested on both AdoptOpenJDK 1.8.0_202-b08 AdoptOpenJDK 11.0.1+13 Reporter: Andreas Asplund Assignee: Radoslav Husar Priority: Major Fix For: 16.0.0.CR1 mod cluster and HTTP2 enabled (the default) when using listener default in the modcluster subsystem. When turning off HTTP2 the clustering starts working again. Nothing is printed in the logs by default so DEBUG logging has to be turned on for undertow. Steps to reproduce: {code} unzip wildfly-dist-15.0.1.Final.zip # Terminal 1 cd wildfly-15.0.1.Final/bin/ ./standalone.sh -c standalone-load-balancer.xml -Djava.net.preferIPv4Stack=true -b 127.0.0.1 -Djboss.modcluster.multicast.address=230.0.0.4 # Terminal 2 cd wildfly-15.0.1.Final/bin/ ./standalone.sh -c standalone-ha.xml -Djboss.node.name=node1 -Djboss.socket.binding.port-offset=100 -Djboss.modcluster.multicast.address=230.0.0.4 # Terminal 3 cd wildfly-15.0.1.Final/bin/ ./standalone.sh -c standalone-ha.xml -Djboss.node.name=node2 -Djboss.socket.binding.port-offset=200 -Djboss.modcluster.multicast.address=230.0.0.4 # Terminal 4 cd wildfly-15.0.1.Final/bin/ ./jboss-cli.sh -c /subsystem=logging/logger=io.undertow:add(level=ALL) /subsystem=logging/console-handler=CONSOLE:write-attribute(name=level, value=ALL) connect localhost:10090 /subsystem=logging/logger=io.undertow:add(level=ALL) /subsystem=logging/console-handler=CONSOLE:write-attribute(name=level, value=ALL) /subsystem=modcluster/proxy=default:write-attribute(name=listener,value=default) reload connect localhost:10190 /subsystem=logging/logger=io.undertow:add(level=ALL) /subsystem=logging/console-handler=CONSOLE:write-attribute(name=level, value=ALL) /subsystem=modcluster/proxy=default:write-attribute(name=listener,value=default) reload {code} The following can be seen in the logs on the proxied servers: {code} 16:30:23,409 TRACE [io.undertow.request] (default I/O-7) Opened connection with /127.0.0.1:50002 16:30:23,410 DEBUG [io.undertow.request.error-response] (default I/O-7) Setting error code 500 for exchange HttpServerExchange{ OPTIONS * request {HTTP2-Settings=[AAEAABAAAAIAAAABAAQAAP//AAUAAEAA], Connection=[Upgrade, HTTP2-Settings], Upgrade=[h2c], User-Agent=[mod_cluster ping], Host=[127.0.0.1]} response {}}: java.lang.RuntimeException at io.undertow.server.HttpServerExchange.setStatusCode(HttpServerExchange.java:1410) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:393) at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255) at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136) at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59) at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66) at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89) at org.xnio.nio.WorkerThread.run(WorkerThread.java:591) 16:30:23,410 DEBUG [io.undertow.request.io] (default I/O-7) UT005013: An IOException occurred: java.io.IOException: Invalid base64 character encountered: 47 at io.undertow.util.FlexBase64$Decoder.nextByte(FlexBase64.java:1048) at io.undertow.util.FlexBase64$Decoder.nextByte(FlexBase64.java:1015) at io.undertow.util.FlexBase64$Decoder.decode(FlexBase64.java:1277) at io.undertow.util.FlexBase64$Decoder.decode(FlexBase64.java:1347) at io.undertow.util.FlexBase64$Decoder.decode(FlexBase64.java:1413) at io.undertow.util.FlexBase64$Decoder.access$500(FlexBase64.java:983) at io.undertow.util.FlexBase64.decodeURL(FlexBase64.java:320) at io.undertow.server.protocol.http2.Http2UpgradeHandler.handleHttp2Upgrade(Http2UpgradeHandler.java:158) at io.undertow.server.protocol.http2.Http2UpgradeHandler.handleUpgradeBody(Http2UpgradeHandler.java:107) at io.undertow.server.protocol.http2.Http2UpgradeHandler.handleRequest(Http2UpgradeHandler.java:97) at io.undertow.server.handlers.DisallowedMethodsHandler.handleRequest(DisallowedMethodsHandler.java:61) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255) at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136) at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59) at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66) at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89) at org.xnio.nio.WorkerThread.run(WorkerThread.java:591) 16:30:23,411 TRACE [io.undertow.server.HttpServerExchange] (default I/O-7) Starting to write response for HttpServerExchange{ OPTIONS * request {HTTP2-Settings=[AAEAABAAAAIAAAABAAQAAP//AAUAAEAA], Connection=[Upgrade, HTTP2-Settings], Upgrade=[h2c], User-Agent=[mod_cluster ping], Host=[127.0.0.1]} response {Connection=[keep-alive], Content-Length=[0], Date=[Wed, 23 Jan 2019 15:30:23 GMT]}} {code}