7:35 p.m.
I have two clustered JMS servers running JBoss [Zion] 4.0.4.GA (build:
CVSTag=JBoss_4_0_4_GA date=200605151000); the servers run linux (debian). JMS data is
persisted to a postgres8.1 database; each server has the same configuration (same JMS
destinations; same datasource for the JMS persistence manager, etc.).
I have been doing some testing to verify that when the master node (node1) goes down, the
other node (node2) assumes the role of master. Also testing to be sure that when node1
comes back up all still processes without problems. I have seen a bunch of exceptions
that lead me to believe that both servers are trying to be master when node1 is brought
back up; I am pretty sure that this is not possible (andI see that at any given moment
only one has assumed the role of MasterNode when I check the HASingletonDeployer in the
JMX console for each server); so I am hoping that there is a better explanation for what I
am seeing.
Here are the steps that I have carried out:
1. Bring up two clustered JBoss servers; each one has an identical JMS configuration; each
one has the same ear file deployed
2. Start a client that continously produces messages for a Topic; messages begin to be
persisted; note that the client uses HA-JNDI (it has a list of both servers)
3. There are 3 MDBs; two have durable subscriptions; one sleeps for a couple of seconds
during processing
4. Kill the master node (node1) [e.g., stop the JBoss server]; other node (node2) assumes
the responsibility of master; client recognizes this change and starts to connect to the
new master (node2)
5. bring up node1; exceptions start to fly on node1 (none on node2); the exceptions do not
stop; client sees no exceptions
Here is an example of the exception(s):
| 2006-08-15 17:14:13,343 ERROR [org.jboss.jms.asf.StdServerSession] failed to
commit/rollback
| org.jboss.tm.JBossRollbackException: Unable to commit,
tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=dingle/535, BranchQual=, localId=535]
status=STATUS_NO_TRANSACTION; - nested throwable: (org.jboss.mq.SpyXAException: - nested
throwable: (org.jboss.mq.SpyTransactionRolledBackException:
| Transaction was rolled back.; - nested throwable: (org.jboss.mq.SpyJMSException: Could
not mark the message as deleted in the database: update affected 0 rows)))
| at org.jboss.tm.TransactionImpl.commit(TransactionImpl.java:372)
| at org.jboss.tm.TxManager.commit(TxManager.java:240)
| at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:351)
| at
org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:902)
| at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:170)
| at org.jboss.mq.SpySession.run(SpySession.java:323)
| at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194)
| at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:748)
| at java.lang.Thread.run(Thread.java:595)
| Caused by: org.jboss.mq.SpyXAException: - nested throwable:
(org.jboss.mq.SpyTransactionRolledBackException: Transaction was rolled back.; - nested
throwable: (org.jboss.mq.SpyJMSException: Could not mark the message as deleted in the
database: update affected 0 rows))
| at org.jboss.mq.SpyXAResource.commit(SpyXAResource.java:102)
| at org.jboss.tm.TransactionImpl$Resource.commit(TransactionImpl.java:2253)
| at org.jboss.tm.TransactionImpl.commitResources(TransactionImpl.java:1784)
| at org.jboss.tm.TransactionImpl.commit(TransactionImpl.java:340)
| ... 8 more
| Caused by: org.jboss.mq.SpyTransactionRolledBackException: Transaction was rolled
back.; - nested throwable: (org.jboss.mq.SpyJMSException: Could
| not mark the message as deleted in the database: update affected 0 rows)
| at
org.jboss.mq.server.JMSDestinationManager.transact(JMSDestinationManager.java:435)
| at
org.jboss.mq.server.JMSServerInterceptorSupport.transact(JMSServerInterceptorSupport.java:200)
| at
org.jboss.mq.security.ServerSecurityInterceptor.transact(ServerSecurityInterceptor.java:197)
| at
org.jboss.mq.server.TracingInterceptor.transact(TracingInterceptor.java:422)
| at org.jboss.mq.server.JMSServerInvoker.transact(JMSServerInvoker.java:201)
| at org.jboss.mq.il.jvm.JVMServerIL.transact(JVMServerIL.java:342)
| at org.jboss.mq.Connection.send(Connection.java:1110)
| at org.jboss.mq.SpyXAResourceManager.commit(SpyXAResourceManager.java:164)
| at org.jboss.mq.SpyXAResource.commit(SpyXAResource.java:98)
| ... 11 more
| Caused by: org.jboss.mq.SpyJMSException: Could not mark the message as deleted in the
database: update affected 0 rows
| at
org.jboss.mq.pm.jdbc2.PersistenceManager.remove(PersistenceManager.java:1204)
| at org.jboss.mq.server.BasicQueue.acknowledge(BasicQueue.java:578)
| at org.jboss.mq.server.JMSTopic.acknowledge(JMSTopic.java:348)
| at org.jboss.mq.server.ClientConsumer.acknowledge(ClientConsumer.java:334)
| at
org.jboss.mq.server.JMSDestinationManager.acknowledge(JMSDestinationManager.java:483)
| at
org.jboss.mq.server.JMSDestinationManager.transact(JMSDestinationManager.java:427)
| ... 19 more
| 2006-08-15 17:14:13,348 WARN [org.jboss.tm.TransactionImpl] XAException:
tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=dingle/536, BranchQual=, localId=536]
errorCode=XAER_RMERR
| org.jboss.mq.SpyXAException: - nested throwable:
(org.jboss.mq.SpyTransactionRolledBackException: Transaction was rolled back.; - nested
throwable: (org.jboss.mq.SpyJMSException: Could not mark the message as deleted in the
database: update affected 0 rows))
| at org.jboss.mq.SpyXAResource.commit(SpyXAResource.java:102)
| at org.jboss.tm.TransactionImpl$Resource.commit(TransactionImpl.java:2253)
| at org.jboss.tm.TransactionImpl.commitResources(TransactionImpl.java:1784)
| at org.jboss.tm.TransactionImpl.commit(TransactionImpl.java:340)
| at org.jboss.tm.TxManager.commit(TxManager.java:240)
| at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:351)
| at
org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:902)
| at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:170)
| at org.jboss.mq.SpySession.run(SpySession.java:323)
| at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194)
| at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:748)
| at java.lang.Thread.run(Thread.java:595)
| Caused by: org.jboss.mq.SpyTransactionRolledBackException: Transaction was rolled
back.; - nested throwable: (org.jboss.mq.SpyJMSException: Could
| not mark the message as deleted in the database: update affected 0 rows)
| at
org.jboss.mq.server.JMSDestinationManager.transact(JMSDestinationManager.java:435)
| at
org.jboss.mq.server.JMSServerInterceptorSupport.transact(JMSServerInterceptorSupport.java:200)
| at
org.jboss.mq.security.ServerSecurityInterceptor.transact(ServerSecurityInterceptor.java:197)
| at
org.jboss.mq.server.TracingInterceptor.transact(TracingInterceptor.java:422)
| at org.jboss.mq.server.JMSServerInvoker.transact(JMSServerInvoker.java:201)
| at org.jboss.mq.il.jvm.JVMServerIL.transact(JVMServerIL.java:342)
| at org.jboss.mq.Connection.send(Connection.java:1110)
| at org.jboss.mq.SpyXAResourceManager.commit(SpyXAResourceManager.java:164)
| at org.jboss.mq.SpyXAResource.commit(SpyXAResource.java:98)
| ... 11 more
| Caused by: org.jboss.mq.SpyJMSException: Could not mark the message as deleted in the
database: update affected 0 rows
| at
org.jboss.mq.pm.jdbc2.PersistenceManager.remove(PersistenceManager.java:1204)
| at org.jboss.mq.server.BasicQueue.acknowledge(BasicQueue.java:578)
| at org.jboss.mq.server.JMSTopic.acknowledge(JMSTopic.java:348)
| at org.jboss.mq.server.ClientConsumer.acknowledge(ClientConsumer.java:334)
| at
org.jboss.mq.server.JMSDestinationManager.acknowledge(JMSDestinationManager.java:483)
| at
org.jboss.mq.server.JMSDestinationManager.transact(JMSDestinationManager.java:427)
| ... 19 more
|
6. kill node2; exceptions on node1 disappear after a short time
7. if I bring node2 back up then the exceptions start to fly on node2 (and do not stop);
same exceptions as copy/pasted above
It seems like both servers attempt to become the master node when node1 is restarted (Step
5), hence the issue with the database error. I checked the HASingletonDeployer in the JMX
console for each server and only one of the servers is the MasterNode at any point in
time.
Any thoughts on this? It appears that both servers are attempting to access the same
message(s).
Thanks for any help.
View the original post :
http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3965385#...
Reply to the post :
http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&a...