[jboss-user] [JBoss Messaging] - Re: MySQL in Cluster question
jbabad
do-not-reply at jboss.com
Fri Feb 6 05:34:50 EST 2009
OK, here are some errors that I'm seeing ....
09:44:14,178 WARN [JDBCSupport] SQLException caught, SQLState HY000 code:1296- assuming deadlock detected, try:1
| java.sql.SQLException: Got error 266 'Time-out in NDB, probably caused by deadlock' from NDB
| at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1055)
| at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
| at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3515)
| at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3447)
| at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1951)
| at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2101)
| at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2554)
| at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1761)
| at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2046)
| at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1964)
| at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1949)
| at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:365)
| at org.jboss.messaging.core.impl.JDBCPersistenceManager.cacheID(JDBCPersistenceManager.java:1637)
| at org.jboss.messaging.core.impl.JDBCPersistenceManager$1AddReferenceRunner.doTransaction(JDBCPersistenceManager.java:1370)
| at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner2.execute(JDBCSupport.java:474)
| at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner2.executeWithRetry(JDBCSupport.java:512)
| at org.jboss.messaging.core.impl.JDBCPersistenceManager.addReference(JDBCPersistenceManager.java:1408)
| at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:226)
| at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2212)
| at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:499)
| at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:755)
| at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:397)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeTarget(SessionAdvised$send_7280680627620114891.java)
| at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
| at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
| at sun.reflect.GeneratedMethodAccessor283.invoke(Unknown Source)
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:597)
| at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
| at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
| at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
| at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
| at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:908)
| at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:742)
| at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:695)
| at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:549)
| at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:230)
| 09:44:14,189 WARN [JDBCSupport] Trying again after a pause
| 09:44:14,282 WARN [JDBCSupport] Update worked after retry
|
Also, this one...
| 09:45:15,657 WARN [JDBCSupport] SQLException caught, SQLState HY000 code:1296- assuming deadlock detected, try:1
| java.sql.SQLException: Got error 286 'Node failure caused abort of transaction' from NDB
| at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1055)
| at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
| at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3515)
| at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3447)
| at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1951)
| at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2101)
| at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2554)
| at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1761)
| at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2046)
| at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1964)
| at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1949)
| at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:365)
| at org.jboss.messaging.core.impl.JDBCPersistenceManager$1AddReferenceRunner.doTransaction(JDBCPersistenceManager.java:1357)
| at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner2.execute(JDBCSupport.java:474)
| at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner2.executeWithRetry(JDBCSupport.java:512)
| at org.jboss.messaging.core.impl.JDBCPersistenceManager.addReference(JDBCPersistenceManager.java:1408)
| at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:226)
| at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2212)
| at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:499)
| at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:755)
| at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:397)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeTarget(SessionAdvised$send_7280680627620114891.java)
| at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
| at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
| at sun.reflect.GeneratedMethodAccessor283.invoke(Unknown Source)
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:597)
| at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
| at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
| at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
| at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
| at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
| at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:908)
| at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:742)
| at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:695)
| at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:549)
| at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:230)
| 09:45:15,671 WARN [JDBCSupport] Trying again after a pause
| 09:45:16,160 WARN [JDBCSupport] Update worked after retry
|
First one about deadlock - I was seeing this more often, but then increased
the MySQL TransactionDeadlockDetectionTimeout to 3s, so not happening as often.
The second one I've not seen/noticed before,
I'm still trying to dig out the client-side error - but the error is related to failing over.
| 11:08:13,730 INFO @Thread-10 failureDetected(FailoverCommandCenter.java:117) [FailoverCommandCenter] JBoss Messaging server failure detected - waiting for failover to complete...
|
When this error appears on the sender app, it appears as though Jboss1 has failed over its JBM ConnectionFactory to Jboss2.
As another thing to add to the mix...
When messages are stuck - I have seen the following:
Shut down Jboss1 - which then forces a failover onto Jboss2. The receiver application is then able to receive the messages. This is odd since I can see the messages in the JBM_MSG table across the MySQL cluster.
For information:
For Jboss1 and Jboss2 I've set up the datasources so that they can fail over i.e. for Jboss1 the url is mysql1|mysql2 and for jboss2 it's mysql2|mysql1
I've noticed that in the case where Jboss1 fails over from mysql1 to mysql2
(ie both are now pointing to the same MySQL instance) I don't see the issues about stuck messages.
If all this sounds confused - you're right :) as I've not got my head round the root of the problem. If I try to summarise it seems to be that under load JBM/MySQL:
1. Either thinks that one of the MySQL data nodes is no longer responding which triggers a failover;
2. Or, gets into a state whereby a sender app can still push messages into the cluster, but a receiver app can't pick them up unless:
- either there's been a failover to the JBoss that the receiver is connecting to,
- or... the receiver connects to the same JBoss instance as the sender.
Thanks.
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4207647#4207647
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4207647
More information about the jboss-user
mailing list