[jboss-jira] [JBoss JIRA] Updated: (JBAS-5815) Bug in DomainServerSocketFactory - SSL clientAuth

Bug in DomainServerSocketFactory - SSL clientAuth -------------------------------------------------- Key: JBAS-5815 URL: https://jira.jboss.org/jira/browse/JBAS-5815 Project: JBoss Application Server Issue Type: Bug Security Level: Public(Everyone can see) Components: Security Reporter: Scott M Stark Assignee: Stefan Guilhen Fix For: JBossAS-5.1.0.GA Daniel Straub <ds(a)ctrlaltdel.de&gt; reports: I had to enable some settings on the RMISSLServerSocketFactory, but the solution for this - shown in wiki.jboss.org/wiki/JRMPInvoker or JBAS-1983 doesn't work. This ends with a nullpointer exception because the the initialization of securityDomain failed. To deal with this, I derive a class from the RMISSLServerSocketFactory like this public class ServerSocketFactory extends RMISSLServerSocketFactory { public ServerSocketFactory() { super(); setNeedsClientAuth(true); //setWantsClientAuth(false); } } and use this as RMIServerSocketFactory of the JRMPInvoker. But this solution also doesn't work ;-( There is another problem in the DomainServerSocketFactory : public ServerSocket createServerSocket(int port, int backlog, InetAddress ifAddress) throws IOException { initSSLContext(); SSLServerSocketFactory factory = sslCtx.getServerSocketFactory(); SSLServerSocket socket = (SSLServerSocket) factory.createServerSocket(port, backlog, ifAddress); SSLSessionContext ctx = sslCtx.getServerSessionContext(); System.out.println(ctx); if( log.isTraceEnabled() ) { String[] supportedProtocols = socket.getSupportedProtocols(); log.debug("Supported protocols: " + Arrays.asList(supportedProtocols)); String[] supportedCipherSuites = socket.getSupportedCipherSuites(); log.debug("Supported CipherSuites: " + Arrays.asList(supportedCipherSuites)); } socket.setNeedClientAuth(needsClientAuth); socket.setWantClientAuth(wantsClientAuth); ... - to make a long story short, the "bug" is in the implementation of SSLServerSocket. This class uses only one instance variable to store the setting of clientAuth ("doClientAuth"). socket.setNeedClientAuth(needsClientAuth) set these to the value "2". fine. but the next call socket.setWantClientAuth(wantsClientAuth) set these to "1" if wantsClientAuth is true, otherwise to "0". in both cases, the first call is override. bad. Here is the decompiled class (com.sun.net.ssl.internal.ssl. SSLServerSocketImpl) : ... public void setNeedClientAuth(boolean flag) { doClientAuth = ((byte)(flag ? 2 : 0)); } public boolean getNeedClientAuth() { return doClientAuth == 2; } public void setWantClientAuth(boolean flag) { doClientAuth = ((byte)(flag ? 1 : 0)); } public boolean getWantClientAuth() { return doClientAuth == 1; } ... well, what for a strange implementation ... I modified my ServerSockeFactory > @Override public ServerSocket createServerSocket(int port) throws IOException { SSLServerSocket sslSocket = (SSLServerSocket) super.createServerSocket(port); sslSocket.setNeedClientAuth(true); return sslSocket; } and now the client authentification works. But can we provide a fix for this problems (initialization of RMISSLServerSocketFactory and SSLServerSocket - e.g if needsClientAuth, why set also wantsClientAuth) ?