JMS feature does not work after cluster failover ------------------------------------------------ Key: JBSEAM-4212 URL: https://jira.jboss.org/jira/browse/JBSEAM-4212 Project: Seam Issue Type: Bug Components: Remoting Affects Versions: 2.0.2.SP1 Environment: JBoss 4.2.3.GA (clustered) Reporter: Scott McNab Assignee: Shane Bryzak We have configured Seam to work in a clustered environment by using a custom JMSConnectionProvider that fetches the UIL2ConnectionFactory from HA-JNDI. This arrangements works perfectly fine while both servers are up - i.e. a web client connected to the slave server will correctly receive Asynchronous Seam notifications from a JMS topic hosted on the master server. However, when the master server fails over to the slave, the Seam JMS support on the slave stops working. We get the following exception: 06-02 12:52:45.405 ERROR [seam.remoting.messaging.SubscriptionRegistry] (http-192.168.100.140-80-2:) org.jboss.mq.SpyJMSException: Cannot get the Topic from the provider; - nested throwable: (java.io.IOE xception: Client is not connected) 06-02 12:52:45.406 ERROR [jboss.seam.remoting.Remoting] (http-192.168.100.140-80-2:) Error java.lang.NullPointerException at org.jboss.seam.remoting.messaging.SubscriptionRequest.marshal(SubscriptionRequest.java:31) at org.jboss.seam.remoting.SubscriptionHandler.marshalResponse(SubscriptionHandler.java:107) at org.jboss.seam.remoting.SubscriptionHandler.handle(SubscriptionHandler.java:96) at org.jboss.seam.remoting.Remoting.getResource(Remoting.java:111) at org.jboss.seam.servlet.SeamResourceServlet.doGet(SeamResourceServlet.java:75) at org.jboss.seam.servlet.SeamResourceServlet.doPost(SeamResourceServlet.java:92) at javax.servlet.http.HttpServlet.service(HttpServlet.java:710) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83) at org.jboss.seam.debug.hot.HotDeployFilter.doFilter(HotDeployFilter.java:68) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:58) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.jboss.seam.web.MultipartFilter.doFilter(MultipartFilter.java:85) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182) at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:97) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157) at org.jboss.web.tomcat.service.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:677) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446) at java.lang.Thread.run(Thread.java:595) We have diagnosed this issue and have developed a fix. The problem is as follows: - once failover occurs, the topicConnection field in the SubscriptionRegistry object becomes invalid, as the connection to the master server has died. - every subsequent call to SubscriptionRegistry.subscribe() will fail with SpyJMSException: Cannot get the Topic from the provider; Client is not connected. - even if the master server recovers, this connection is dead and all Seam JMS subscriptions will no longer work. The fix is to catch the Exception and retry with a new topicConnection if a subscribe request fails. The following patch on jboss-seam-2.0.2.SP1/src/remoting/org/jboss/seam/remoting/messaging/SubscriptionRegistry.java works for us: Index: SubscriptionRegistry.java =================================================================== --- SubscriptionRegistry.java (revision 5528) +++ SubscriptionRegistry.java (working copy) @@ -139,7 +139,21 @@ RemoteSubscriber sub = new RemoteSubscriber(UUID.randomUUID().toString(), topicName); try { - sub.subscribe(getTopicConnection()); + try { + sub.subscribe(getTopicConnection()); + } catch (Exception e) { + // If we failed during the first attempt, we might have a dead JMS connection. + // Clear the topic connection and try again. + if (topicConnection != null) { + try { + topicConnection.close(); + } catch (Exception e1) { + // Ignore any exception during the close, since this connection is probably dead anyway + } + } + topicConnection = null; + sub.subscribe(getTopicConnection()); + } subscriptions.put(sub.getToken(), sub); // Save the client's token in their session context -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira