<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<body link="#355491" alink="#4262a1" vlink="#355491" style="background: #e2e2e2; margin: 0; padding: 20px;">
<div>
<table cellpadding="0" bgcolor="#FFFFFF" border="0" cellspacing="0" style="border: 1px solid #dadada; margin-bottom: 30px; width: 100%; -moz-border-radius: 6px; -webkit-border-radius: 6px;">
<tbody>
<tr>
<td>
<table border="0" cellpadding="0" cellspacing="0" bgcolor="#FFFFFF" style="border: solid 2px #ccc; background: #dadada; width: 100%; -moz-border-radius: 6px; -webkit-border-radius: 6px;">
<tbody>
<tr>
<td bgcolor="#000000" valign="middle" height="58px" style="border-bottom: 1px solid #ccc; padding: 20px; -moz-border-radius-topleft: 3px; -moz-border-radius-topright: 3px; -webkit-border-top-right-radius: 5px; -webkit-border-top-left-radius: 5px;">
<h1 style="color: #333333; font: bold 22px Arial, Helvetica, sans-serif; margin: 0; display: block !important;">
<!-- To have a header image/logo replace the name below with your img tag -->
<!-- Email clients will render the images when the message is read so any image -->
<!-- must be made available on a public server, so that all recipients can load the image. -->
<a href="http://community.jboss.org/index.jspa" style="text-decoration: none; color: #E1E1E1">JBoss Community</a></h1>
</td>
</tr>
<tr>
<td bgcolor="#FFFFFF" style="font: normal 12px Arial, Helvetica, sans-serif; color:#333333; padding: 20px; -moz-border-radius-bottomleft: 4px; -moz-border-radius-bottomright: 4px; -webkit-border-bottom-right-radius: 5px; -webkit-border-bottom-left-radius: 5px;"><h3 style="margin: 10px 0 5px; font-size: 17px; font-weight: normal;">
HornetQ live-Backup failover on JBoss AS 7
</h3>
<span style="margin-bottom: 10px;">
created by <a href="http://community.jboss.org/people/hsichuan">Celia Wang</a> in <i>JBoss AS 7 Development</i> - <a href="http://community.jboss.org/message/645871#645871">View the full discussion</a>
</span>
<hr style="margin: 20px 0; border: none; background-color: #dadada; height: 1px;">
<div class="jive-rendered-content"><p>Hello everyone, I am not a native English speaker and I'm new to here. I really need some help, please help me~thanks.</p><p>I had spent a lot of time on the HornetQ User Manual but it seems like that there are some problems happened.</p><p>What I want to do is that live server can failover to backup server when client is sending messages.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>My problem is: when I killed the live server, my backup server could not take over and become live.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>I have 4 servers - </p><p>A: HornetQ Live Server on JBoss AS7 (jboss-as-7.0.1.Final)</p><p>     deploy mdb-ear-1.0.0.ear (andytaylor-HornetQ-AS7-examples-1a5935b  , it's a mdb example)</p><p>B: HornetQ Backup Server on JBoss AS7</p><p>     deploy mdb-ear-1.0.0.ear , but backup server is lock, this deployment will not be active until backup server become live</p><p>C: a client running MDB example, send 2000 of messages to live server  (andytaylor-HornetQ-AS7-examples-1a5935b/mdb/mdb-client)</p><p>D: shared store - NFS, to store A&B's message journal</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>The method of executing server:</p><p>A: ./standalone.sh --server-config=standalone-preview-ha-live.xml</p><p>B: ./standalone.sh --server-config=standalone-preview-ha-back.xml</p><p>C: mvn -Pclient test</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>Here is the configurations changed on A & B</p><p>standalone-preview-ha-live.xml & standalone-preview-ha-back.xml are modified from standalone-preview-ha.xml</p><p>where the font color is red is my modified for configurations, others are the same as standalone-preview-ha.xml</p><p><strong>A: live server - </strong></p><p><span style="color: #0000ff;"><subsystem xmlns="urn:jboss:domain:messaging:1.0"></span></p><p><span style="color: #0000ff;">            <!-- Default journal file size is 10Mb, reduced here to 100k for faster first boot --></span></p><p><span style="color: #0000ff;">            <journal-file-size>102400</journal-file-size></span></p><p><span style="color: #0000ff;">            <journal-min-files>2</journal-min-files></span></p><p><span style="color: #0000ff;">            <journal-type>NIO</journal-type></span></p><p><span style="color: #0000ff;">            <!-- disable messaging persistence --></span></p><p><span style="color: #0000ff;">            <persistence-enabled>false</persistence-enabled></span></p><p><span style="color: #0000ff;"><!--add hornetq-configuration set live here--> </span></p><p><span style="color: #ff0000;">          <clustered>true</clustered></span></p><p><span style="color: #ff0000;">          <failover-on-shutdown>false</failover-on-shutdown></span></p><p><span style="color: #ff0000;">          <shared-store>true</shared-store></span></p><p><span style="color: #0000ff;"><!--end here--></span></p><p><span style="color: #0000ff;">---</span></p><p><span style="color: #0000ff;">skip<br/></span></p><p><span style="color: #0000ff;">---</span></p><p>         <span style="color: #0000ff;"> <jms-destinations></span></p><p><span style="color: #0000ff;">                <jms-queue name="testQueue"></span></p><p><span style="color: #0000ff;">                   <entry name="queue/test"/></span></p><p><span style="color: #0000ff;">                </jms-queue></span></p><p><span style="color: #ff0000;">                                 <jms-queue name="mdbQueue"></span></p><p><span style="color: #ff0000;">                    <entry name="queue/mdbQueue"/></span></p><p><span style="color: #ff0000;">                </jms-queue></span></p><p><span style="color: #ff0000;">                <jms-queue name="mdbReplyQueue"></span></p><p><span style="color: #ff0000;">                    <entry name="queue/mdbReplyQueue"/></span></p><p><span style="color: #ff0000;">                </jms-queue></span></p><p><span style="color: #0000ff;">                <jms-topic name="testTopic"></span></p><p><span style="color: #0000ff;">                   <entry name="topic/test"/></span></p><p><span style="color: #0000ff;">                </jms-topic></span></p><p><span style="color: #0000ff;">            </jms-destinations></span></p><p><span style="color: #0000ff;">---</span></p><p><span style="color: #0000ff;">skip</span></p><p><span style="color: #0000ff;">---</span></p><p><span style="color: #0000ff;">     <interfaces></span></p><p><span style="color: #0000ff;">        <interface name="management"></span></p><p><span style="color: #0000ff;">            <inet-address value="<span style="color: #ff6600;">140.XXX.XX.231</span>"/></span></p><p><span style="color: #0000ff;">        </interface></span></p><p><span style="color: #0000ff;">        <interface name="public"></span></p><p><span style="color: #0000ff;">           <inet-address value="<span style="color: #ff0000;">140.XXX.XX.231</span>"/></span></p><p><span style="color: #0000ff;">        </interface></span></p><p><span style="color: #0000ff;">    </interfaces></span></p><p><strong style=": ; color: #000000;">B: backup server - </strong></p><p>        <span style="color: #0000ff;"><subsystem xmlns="urn:jboss:domain:messaging:1.0"></span></p><p><span style="color: #0000ff;">            <!-- Default journal file size is 10Mb, reduced here to 100k for faster first boot --></span></p><p><span style="color: #0000ff;">            <journal-file-size>102400</journal-file-size></span></p><p><span style="color: #0000ff;">            <journal-min-files>2</journal-min-files></span></p><p><span style="color: #0000ff;">            <journal-type>NIO</journal-type></span></p><p><span style="color: #0000ff;">            <!-- disable messaging persistence --></span></p><p><span style="color: #0000ff;">            <persistence-enabled>false</persistence-enabled></span></p><p><span style="color: #0000ff;"><!--add hornetq-configuration set backup here --></span></p><p><span style="color: #ff0000;">    <clustered>true</clustered></span></p><p><span style="color: #ff0000;">              <failover-on-shutdown>false</failover-on-shutdown></span></p><p><span style="color: #ff0000;">             <backup>true</backup></span></p><p><span style="color: #ff0000;">    <shared-store>true</shared-store></span></p><p><span style="color: #0000ff;"><!--end here--></span></p><p><span style="color: #0000ff;">            <connectors></span></p><p><span style="color: #0000ff;">               <netty-connector name="netty" socket-binding="messaging" /></span></p><p><span style="color: #0000ff;">               <netty-connector name="netty-throughput" socket-binding="messaging-throughput"></span></p><p><span style="color: #0000ff;">                  <param key="batch-delay" value="50"/></span></p><p><span style="color: #0000ff;">               </netty-connector></span></p><p><span style="color: #0000ff;">               <in-vm-connector name="in-vm" server-id="0" /></span></p><p><span style="color: #0000ff;">            </connectors></span></p><p><span style="color: #0000ff;">---</span></p><p><span style="color: #0000ff;">skip</span></p><p><span style="color: #0000ff;">---<br/></span></p><p>         <span style="color: #0000ff;"> <jms-destinations></span></p><p><span style="color: #0000ff;">                <jms-queue name="testQueue"></span></p><p><span style="color: #0000ff;">                   <entry name="queue/test"/></span></p><p><span style="color: #0000ff;">                </jms-queue></span></p><p><span style="color: #ff0000;">                <jms-queue name="mdbQueue"></span></p><p><span style="color: #ff0000;">                    <entry name="queue/mdbQueue"/></span></p><p><span style="color: #ff0000;">                </jms-queue></span></p><p><span style="color: #ff0000;">                <jms-queue name="mdbReplyQueue"></span></p><p><span style="color: #ff0000;">                    <entry name="queue/mdbReplyQueue"/></span></p><p><span style="color: #ff0000;">                </jms-queue></span></p><p><span style="color: #0000ff;">                <jms-topic name="testTopic"></span></p><p><span style="color: #0000ff;">                   <entry name="topic/test"/></span></p><p><span style="color: #0000ff;">                </jms-topic></span></p><p><span style="color: #0000ff;">            </jms-destinations></span></p><p><span style="color: #0000ff;">---</span></p><p><span style="color: #0000ff;">skip</span></p><p><span style="color: #0000ff;">---</span></p><p><span style="color: #0000ff;">     <interfaces></span></p><p><span style="color: #0000ff;">        <interface name="management"></span></p><p><span style="color: #0000ff;">            <inet-address value="<span style="color: #ff6600;">140.XXX.XX.232</span>"/></span></p><p><span style="color: #0000ff;">        </interface></span></p><p><span style="color: #0000ff;">        <interface name="public"></span></p><p><span style="color: #0000ff;">           <inet-address value="<span style="color: #ff0000;">140.XXX.XX.232</span>"/></span></p><p><span style="color: #0000ff;">        </interface></span></p><p><span style="color: #0000ff;">    </interfaces></span></p><p><strong style=": ; color: #000000;">C: I modified "MDBClient.java", in order to send messages to the live server.</strong></p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p><span style="color: #333333;">          final int numMessages = 2000; </span></p><p><span style="color: #333333;">          HashMap map = new HashMap();</span></p><p><span style="color: #333333;">          map.put("host", "140.XXX.XX.231");</span></p><p><span style="color: #333333;">          map.put("port", 5445);</span></p><p>          TransportConfiguration transportConfiguration = new TransportConfiguration(NettyConnectorFactory.class.getName(),map);</p><p>---skip---     </p><p>     TextMessage message=null;</p><p>      //Step 7. Create a Text Message</p><p>     for (int i = 0; i < numMessages; i++)</p><p>     {</p><p>                   message = session.createTextMessage("This is a text message"+ i);</p><p>          //Step 8. Send the Message</p><p>                   producer.send(message);</p><p>                   System.out.println("Sent message: " + message.getText());</p><p>     }</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p><strong>D: shared store - NFS</strong></p><p>The NFS directory is /home/user/jboss-as-7.0.1.Final/standalone/data</p><p>==================================</p><p>Now, after live & backup server is started, I executed "MDBClient.java", sending 2000 messages to the live server.</p><p>Then, I went to the live server and killed its process id.</p><p>jps</p><p>kill XXXXX</p><p>My backup server did not become live, but after some minutes, there are a lot of WARN on live server's boot.log & backup server's server.log</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>WARN  [com.arjuna.ats.jta] ARJUNA16038: No XAResource to recover < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff8c7452e7:7386542f:4f0d2d17:ebf, node_name=1, branch_uid=0:ffff8c7452e7:7386542f:4f0d2d17:ec0, eis_name=unknown eis name ></p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p><strong style=": ; color: #000000;">Here is the server.log on live server.</strong></p><p>14:32:55,668 INFO  [org.apache.catalina.core.AprLifecycleListener] (MSC service thread 1-4) The Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /opt/jdk1.6.0_24/jre/lib/amd64/server:/opt/jdk1.6.0_24/jre/lib/amd64:/opt/jdk1.6.0_24/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib</p><p>14:32:55,701 INFO  [org.jboss.as.jmx.JMXConnectorService] (MSC service thread 1-1) Starting remote JMX connector</p><p>14:32:55,707 INFO  [org.jboss.as.remoting] (MSC service thread 1-7) Listening on /140.XXX.XX.231:9999</p><p>14:32:55,738 INFO  [org.jboss.as.ee] (Controller Boot Thread) Activating EE subsystem</p><p>14:32:55,742 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-5) Starting Coyote HTTP/1.1 on http-WMC-Cloud-A-140.XXX.XX.231-8080</p><p>14:32:55,751 INFO  [org.jboss.ws.common.management.AbstractServerConfig] (MSC service thread 1-6) JBoss Web Services - Stack CXF Server 4.0.0.Beta3</p><p>14:32:55,889 INFO  [org.jboss.as.modcluster] (MSC service thread 1-4) Mod_cluster uses default load balancer provider</p><p>14:32:55,963 INFO  [org.jboss.modcluster.ModClusterService] (MSC service thread 1-4) Initializing mod_cluster 1.1.3.Final</p><p>14:32:55,982 INFO  [org.jboss.as.connector] (MSC service thread 1-2) Starting JCA Subsystem (JBoss IronJacamar 1.0.3.Final)</p><p>14:32:55,986 INFO  [org.jboss.modcluster.advertise.impl.AdvertiseListenerImpl] (MSC service thread 1-4) Listening to proxy advertisements on 224.0.1.105:23364</p><p>14:32:56,045 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-5) Bound data source [java:jboss/datasources/ExampleDS]</p><p>14:32:56,056 INFO  [org.jboss.as.jacorb] (MSC service thread 1-3) CORBA ORB Service Started</p><p>14:32:56,126 \u8cc7\u8a0a    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-5)<span style="color: #ff0000;"> live server is starting with configuration HornetQ Configuration (clustered=true,backup=false,sharedStore=true,journalDirectory=/home/wmc/jboss-as-7.0.1.Final/standalone/data/messagingjournal,bindingsDirectory=/home/wmc/jboss-as-7.0.1.Final/standalone/data/messagingbindings,largeMessagesDirectory=/home/wmc/jboss-as-7.0.1.Final/standalone/data/messaginglargemessages,pagingDirectory=/home/wmc/jboss-as-7.0.1.Final/standalone/data/messagingpaging)</span></p><p>14:32:56,161 \u8b66\u544a    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-5) Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this.</p><p>14:32:56,172 INFO  [org.jboss.as.connector.services.ResourceAdapterActivatorService$ResourceAdapterActivator] (MSC service thread 1-1) Registered connection factory java:/JmsXA</p><p>14:32:56,193 INFO  [org.hornetq.ra.HornetQResourceAdapter] (MSC service thread 1-1) HornetQ resource adaptor started</p><p>14:32:56,193 INFO  [org.jboss.as.connector.services.ResourceAdapterActivatorService$ResourceAdapterActivator] (MSC service thread 1-1) IJ020002: Deployed: file://RaActivatorhornetq-ra</p><p>14:32:56,209 INFO  [org.jboss.as.jacorb] (MSC service thread 1-3) CORBA Naming Service Started</p><p>14:32:56,304 INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-5) Started Netty Acceptor version 3.2.3.Final-r${buildNumber} WMC-Cloud-A:5455 for CORE protocol</p><p>14:32:56,305 INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-5) Started Netty Acceptor version 3.2.3.Final-r${buildNumber} WMC-Cloud-A:5445 for CORE protocol</p><p>14:32:56,306 \u8cc7\u8a0a    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-5) <span style="color: #ff0000;">Server is now live</span></p><p>14:32:56,306 \u8cc7\u8a0a    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-5) <span style="color: #ff0000;">HornetQ Server version 2.2.7.Final (HQ_2_2_7_FINAL_AS7, 121) [b4c2c38f-3c18-11e1-8dd2-e41f13246ff6] started</span></p><p>14:32:56,308 \u8cc7\u8a0a    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-8) trying to deploy queue jms.topic.testTopic</p><p>14:32:56,384 INFO  [org.jboss.as.messaging.jms.AS7BindingRegistry] (MSC service thread 1-8) Bound messaging object to jndi name java:/topic/test</p><p>14:32:56,395 INFO  [org.jboss.as.messaging.jms.AS7BindingRegistry] (MSC service thread 1-2) Bound messaging object to jndi name java:/RemoteConnectionFactory</p><p>14:32:56,397 INFO  [org.jboss.as.messaging.jms.AS7BindingRegistry] (MSC service thread 1-1) Bound messaging object to jndi name java:/ConnectionFactory</p><p>14:32:56,397 \u8cc7\u8a0a    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-5) trying to deploy queue jms.queue.mdbQueue</p><p>14:32:56,421 INFO  [org.jboss.as.messaging.jms.AS7BindingRegistry] (MSC service thread 1-5) Bound messaging object to jndi name java:/queue/mdbQueue</p><p>14:32:56,422 \u8cc7\u8a0a    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-7) trying to deploy queue jms.queue.testQueue</p><p>14:32:56,435 INFO  [org.jboss.as.messaging.jms.AS7BindingRegistry] (MSC service thread 1-7) Bound messaging object to jndi name java:/queue/test</p><p>14:32:56,435 \u8cc7\u8a0a    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-4) trying to deploy queue jms.queue.mdbReplyQueue</p><p>14:32:56,446 INFO  [org.jboss.as.messaging.jms.AS7BindingRegistry] (MSC service thread 1-4) Bound messaging object to jndi name java:/queue/mdbReplyQueue</p><p>14:32:56,623 INFO  [org.jboss.as.deployment] (MSC service thread 1-4) Started FileSystemDeploymentService for directory /home/wmc/jboss-as-7.0.1.Final/standalone/deployments</p><p>14:32:56,641 INFO  [org.jboss.as] (Controller Boot Thread) JBoss AS 7.0.1.Final "Zap" started in 2665ms - Started 128 of 208 services (80 services are passive or on-demand)</p><p>14:32:56,652 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) Starting deployment of "my-webapp.war"</p><p>14:32:56,652 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-8) Starting deployment of "mdb-ear-1.0.0.ear"</p><p>14:32:56,692 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-3) Starting deployment of "mdb-jar-1.0.0.jar"</p><p>14:32:56,717 INFO  [org.jboss.as.jpa] (MSC service thread 1-3) added javax.persistence.api dependency to my-webapp.war</p><p>14:32:56,720 INFO  [org.jboss.as.jpa] (MSC service thread 1-5) added javax.persistence.api dependency to mdb-ear-1.0.0.ear</p><p>14:32:56,723 INFO  [org.jboss.as.jpa] (MSC service thread 1-5) added javax.persistence.api dependency to mdb-jar-1.0.0.jar</p><p>14:32:56,875 INFO  [org.jboss.web] (MSC service thread 1-4) registering web context: /my-webapp</p><p>14:32:56,923 INFO  [org.jboss.as.server.controller] (DeploymentScanner-threads - 2) Deployed "mdb-ear-1.0.0.ear"</p><p>14:32:56,923 INFO  [org.jboss.as.server.controller] (DeploymentScanner-threads - 2) Deployed "my-webapp.war"</p><p>14:33:05,921 INFO  [org.jboss.modcluster.ModClusterService] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Engine [jboss.web] will use jvmRoute: d883de1f-fe17-3214-8442-4e4880a8781a</p><p><span style="color: #ff0000;">---after client send messages---</span></p><p>14:34:10,802 INFO  [stdout] (Thread-3 (group:HornetQ-client-global-threads-2000502626)) message This is a text message2</p><p>14:34:10,805 INFO  [stdout] (Thread-3 (group:HornetQ-client-global-threads-2000502626)) connection created</p><p>14:34:10,815 INFO  [stdout] (Thread-9 (group:HornetQ-client-global-threads-2000502626)) message This is a text message8</p><p>14:34:10,815 INFO  [stdout] (Thread-9 (group:HornetQ-client-global-threads-2000502626)) connection created</p><p>14:34:10,815 INFO  [stdout] (Thread-2 (group:HornetQ-client-global-threads-2000502626)) message This is a text message1</p><p>14:34:10,816 INFO  [stdout] (Thread-2 (group:HornetQ-client-global-threads-2000502626)) connection created</p><p><span style="color: #ff0000;">---skip---kill the live server</span></p><p>14:36:32,377 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-5) Stopping Coyote HTTP/1.1 on http-WMC-Cloud-A-140.116.82.231-8080</p><p>14:36:32,378 INFO  [org.jboss.as.messaging.jms.AS7BindingRegistry] (MSC service thread 1-7) Unbound messaging object from jndi name java:/RemoteConnectionFactory</p><p>14:36:32,380 INFO  [org.jboss.as.messaging.jms.AS7BindingRegistry] (MSC service thread 1-6) Unbound messaging object from jndi name java:/topic/test</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p><strong>boot.log on live server (because live server is shutdown)</strong></p><p>14:32:55,322 INFO  [org.jboss.as.logging] Removing bootstrap log handlers</p><p><span style="color: #ff0000;">The followings are the WARNs and ERRORs after the live server killed.</span></p><p>14:36:32,398 INFO  [org.jboss.as.logging] Restored bootstrap log handlers</p><p>14:36:32,415 INFO  [org.jboss.modcluster.ModClusterService] All active sessions drained from context [/my-webapp] in 0 seconds</p><p>14:36:32,421 INFO  [org.jboss.as.jmx.JMXConnectorService] JMX remote connector stopped</p><p>14:36:32,423 INFO  [stdout] message This is a text message415</p><p>14:36:32,423 INFO  [stdout] connection created</p><p>14:36:32,426 INFO  [org.jboss.modcluster.ModClusterService] All active sessions drained from context [/my-webapp] in 0 seconds</p><p>14:36:32,429 INFO  [stdout] session created</p><p>14:36:32,430 ERROR [stderr] javax.jms.InvalidDestinationException: Destination mdbReplyQueue does not exist</p><p>14:36:32,430 ERROR [stderr]           at org.hornetq.jms.client.HornetQSession.createProducer(HornetQSession.java:337)</p><p>14:36:32,430 ERROR [stderr]           at org.hornetq.ra.HornetQRASession.createProducer(HornetQRASession.java:1163)</p><p>14:36:32,430 ERROR [stderr]           at org.hornetq.example.MDBSendExample.onMessage(MDBSendExample.java:77)</p><p><span style="color: #000000;">--skip--</span></p><p>14:36:32,608 \u8cc7\u8a0a    [org.hornetq.core.server.impl.HornetQServerImpl] HornetQ Server version 2.2.7.Final (HQ_2_2_7_FINAL_AS7, 121) [b4c2c38f-3c18-11e1-8dd2-e41f13246ff6] stopped</p><p>14:37:02,575 WARN  [com.arjuna.ats.jta] ARJUNA16086: TransactionImple.enlistResource setTransactionTimeout on XAResource < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff8c7452e7:7386542f:4f0d2d17:ee7, node_name=1, branch_uid=0:ffff8c7452e7:7386542f:4f0d2d17:ee8, eis_name=unknown eis name > threw: XAException.XAER_RMERR: javax.transaction.xa.XAException</p><p>--skip</p><p>14:37:02,599 ERROR [org.hornetq.core.client.impl.ClientSessionImpl] Caught jmsexecptione : HornetQException[errorCode=3 message=Timed out waiting for response when sending packet 52]</p><p>--skip</p><p>14:37:02,605 WARN  [com.arjuna.ats.jta] ARJUNA16041: prepare on < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff8c7452e7:7386542f:4f0d2d17:edf, node_name=1, branch_uid=0:ffff8c7452e7:7386542f:4f0d2d17:ee0, eis_name=unknown eis name > (org.hornetq.core.client.impl.DelegatingSession@4aa594e1) failed with exception XAException.XAER_RMERR: javax.transaction.xa.XAException</p><p>--skip</p><p>14:37:02,610 WARN  [com.arjuna.ats.arjuna] ARJUNA12073: BasicAction.End() - prepare phase of action-id 0:ffff8c7452e7:7386542f:4f0d2d17:edf failed.</p><p>14:37:02,610 WARN  [com.arjuna.ats.arjuna] ARJUNA12075: Action Aborting</p><p>14:37:02,633 WARN  [org.hornetq.core.client.impl.ClientSessionImpl] <span style="color: #ff0000;">failover occured during commit throwing XAException.XA_RETRY</span></p><p>14:37:02,634 WARN  [com.arjuna.ats.jta] ARJUNA16036: commit on < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff8c7452e7:7386542f:4f0d2d17:eb7, node_name=1, branch_uid=0:ffff8c7452e7:7386542f:4f0d2d17:eb8, eis_name=unknown eis name > (org.hornetq.core.client.impl.DelegatingSession@58fe210a) failed with exception $XAException.XA_RETRY: javax.transaction.xa.XAException</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p><strong style=": ; color: #000000;">Here is the server.log on backup server.</strong></p><p>14:35:36,363 \u8cc7\u8a0a    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-5) <span style="color: #ff0000;">backup server is starting</span> with configuration HornetQ Configuration (clustered=true,backup=true,sharedStore=true,journalDirectory=/home/wmc/jboss-as-7.0.1.Final/standalone/data/messagingjournal,bindingsDirectory=/home/wmc/jboss-as-7.0.1.Final/standalone/data/messagingbindings,largeMessagesDirectory=/home/wmc/jboss-as-7.0.1.Final/standalone/data/messaginglargemessages,pagingDirectory=/home/wmc/jboss-as-7.0.1.Final/standalone/data/messagingpaging)</p><p>14:35:36,365 \u8cc7\u8a0a    [org.hornetq.jms.server.impl.JMSServerManagerImpl] (MSC service thread 1-5) Caching command for createQueue for mdbReplyQueue since the JMS Server is not active yet</p><p>14:35:36,367 \u8cc7\u8a0a    [org.hornetq.jms.server.impl.JMSServerManagerImpl] (MSC service thread 1-5) Caching command for createConnectionFactory for RemoteConnectionFactory since the JMS Server is not active yet</p><p>14:35:36,367 \u8cc7\u8a0a    [org.hornetq.jms.server.impl.JMSServerManagerImpl] (MSC service thread 1-5) Caching command for createQueue for testQueue since the JMS Server is not active yet</p><p>14:35:36,367 \u8cc7\u8a0a    [org.hornetq.jms.server.impl.JMSServerManagerImpl] (MSC service thread 1-5) Caching command for createQueue for mdbQueue since the JMS Server is not active yet</p><p>14:35:36,368 \u8cc7\u8a0a    [org.hornetq.jms.server.impl.JMSServerManagerImpl] (MSC service thread 1-6) Caching command for createConnectionFactory for InVmConnectionFactory since the JMS Server is not active yet</p><p>14:35:36,368 INFO  [org.hornetq.core.server.impl.FileLockNodeManager] (Thread-18) Waiting to become backup node</p><p>14:35:36,368 \u8cc7\u8a0a    [org.hornetq.jms.server.impl.JMSServerManagerImpl] (MSC service thread 1-2) Caching command for createTopic for testTopic since the JMS Server is not active yet</p><p>14:35:36,396 INFO  [org.jboss.as.jacorb] (MSC service thread 1-1) CORBA Naming Service Started</p><p>14:35:36,401 INFO  [org.jboss.as.connector.services.ResourceAdapterActivatorService$ResourceAdapterActivator] (MSC service thread 1-3) Registered connection factory java:/JmsXA</p><p>14:35:36,407 INFO  [org.hornetq.ra.HornetQResourceAdapter] (MSC service thread 1-3) HornetQ resource adaptor started</p><p>14:35:36,408 INFO  [org.jboss.as.connector.services.ResourceAdapterActivatorService$ResourceAdapterActivator] (MSC service thread 1-3) IJ020002: Deployed: file://RaActivatorhornetq-ra</p><p>14:35:36,427 INFO  [org.hornetq.core.server.impl.FileLockNodeManager] (Thread-18) ** got backup lock</p><p>14:35:36,443 \u8b66\u544a    [org.hornetq.core.server.impl.HornetQServerImpl] (Thread-18) Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this.</p><p>14:35:36,487 \u8cc7\u8a0a    [org.hornetq.core.server.impl.HornetQServerImpl] (Thread-18) <span style="color: #ff0000;">HornetQ Backup Server version 2.2.7.Final (HQ_2_2_7_FINAL_AS7, 121) [b4c2c38f-3c18-11e1-8dd2-e41f13246ff6] started, waiting live to fail before it gets active</span></p><p>14:35:36,843 INFO  [org.jboss.as.deployment] (MSC service thread 1-5) Started FileSystemDeploymentService for directory /home/wmc/jboss-as-7.0.1.Final/standalone/deployments</p><p>14:35:36,860 INFO  [org.jboss.as] (Controller Boot Thread) JBoss AS 7.0.1.Final "Zap" started in 2572ms - Started 128 of 208 services (80 services are passive or on-demand)</p><p>14:35:46,162 INFO  [org.jboss.modcluster.ModClusterService] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Engine [jboss.web] will use jvmRoute: a8c846d5-8c74-327b-ba50-723177f332e2</p><p><span style="color: #ff0000;">--the following are the WARNs after live server killed, </span></p><p>14:37:56,286 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA16037: <span style="color: #ff0000;">Could not find new XAResource to use for recovering non-serializable XAResource XAResourceRecord</span> < resource:null, txid:< formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff8c7452e7:7386542f:4f0d2d17:ed7, node_name=1, branch_uid=0:ffff8c7452e7:7386542f:4f0d2d17:ed8, eis_name=unknown eis name >, heuristic: TwoPhaseOutcome.FINISH_OK com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@78221c75 ></p><p>14:37:56,287 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA16037: Could not find new XAResource to use for recovering non-serializable XAResource XAResourceRecord < resource:null, txid:< formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff8c7452e7:7386542f:4f0d2d17:ed7, node_name=1, branch_uid=0:ffff8c7452e7:7386542f:4f0d2d17:edc, eis_name=unknown eis name >, heuristic: TwoPhaseOutcome.FINISH_OK com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@5271b383 ></p><p>14:37:56,288 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA16038: No XAResource to recover < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff8c7452e7:7386542f:4f0d2d17:ed7, node_name=1, branch_uid=0:ffff8c7452e7:7386542f:4f0d2d17:ed8, eis_name=unknown eis name ></p><p>14:37:56,288 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA16038: No XAResource to recover < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff8c7452e7:7386542f:4f0d2d17:ed7, node_name=1, branch_uid=0:ffff8c7452e7:7386542f:4f0d2d17:edc, eis_name=unknown eis name ></p><p><span style="color: #ff0000;">--it will receive the same WARN repeately---</span></p><p>=================================</p><p>The live & backup server will always receive the same WARNs repeately, it seems like a deadlock.</p><p>However, the backup server did not become live.</p><p>Do I miss any other configurations on live & backup server?</p><p>Does anyone knows how to successfully set live & backup server failover?</p><p>I'm sorry about that I post a long description of my problem, if there is any question about my problem, just ask. </p><p>Thanks for your help! </p></div>
<div style="background-color: #f4f4f4; padding: 10px; margin-top: 20px;">
<p style="margin: 0;">Reply to this message by <a href="http://community.jboss.org/message/645871#645871">going to Community</a></p>
<p style="margin: 0;">Start a new discussion in JBoss AS 7 Development at <a href="http://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2225">Community</a></p>
</div></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
</body>
</html>