[jboss-user] [JBoss Messaging] New message: "Problem with cluster failover."

Krystian Siuda do-not-reply at jboss.com
Thu Jan 7 13:21:23 EST 2010 

User development,

A new message was posted in the thread "Problem with cluster failover.":

http://community.jboss.org/message/518812#518812

Author  : Krystian Siuda
Profile : http://community.jboss.org/people/ksiuda

Message:
--------------------------------------------------------------
Hi,
 
I'm crashtests of an JBoss cluster dedicated to be an JMS server.
I wrote an simple (2 threaded) JMS client that connects to the server and simultaneously sends&receives uniq messages checking whether they are duplicated/lost etc...
The scenario I'm currently stuck with is:
- i put 2 server nodes up
- i start the client (mentioned before)
- i see the traffic
- kill (sig kill) the 1st node
- i still see the traffic after failover
- kill (sig kill) the 2nd node
- the traffic stops
- after some time client starts to throw exceptions like:
javax.jms.IllegalStateException: The object is closed
     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:157)
     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
     at org.jboss.jms.client.delegate.ClientConsumerDelegate.receive(ClientConsumerDelegate.java)
     at org.jboss.jms.client.JBossMessageConsumer.receive(JBossMessageConsumer.java:86)
     at auto.Main.receive(Main.java:92)
     at auto.Main.receiveWrap(Main.java:138)
     at auto.Main$2.run(Main.java:195)
     at java.lang.Thread.run(Thread.java:619)

- and after some more time:
javax.jms.JMSException: Maximum number of failover attempts exceeded. Cannot find a server to failover onto.
     at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:234)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:597)
     at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
     at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
     at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
     at org.jboss.jms.client.JBossConnectionFactory.createConnection(JBossConnectionFactory.java:87)
     at auto.Main.receive(Main.java:86)
     at auto.Main.receiveWrap(Main.java:138)
     at auto.Main$2.run(Main.java:195)
     at java.lang.Thread.run(Thread.java:619)

- i put the 1st node up
- client throws:
org.jboss.jms.exception.MessagingShutdownException: Cannot handle invocation since messaging server is not active (it is either starting up or shutting down)
     at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:133)
     at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:891)
     at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:744)
     at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:697)
     at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:551)
     at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232)
     at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:211)
     at org.jboss.remoting.Client.invoke(Client.java:1724)
     at org.jboss.remoting.Client.invoke(Client.java:629)
(...)

- after some time it throws:
org.jboss.jms.exception.MessagingJMSException: Failed to invoke
     at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:271)
     at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:191)
     at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeTarget(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
     at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:81)
     at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect_z_handleCreateConnectionDelegate_26293492.invoke(StateCreationAspect_z_handleCreateConnectionDelegate_26293492.java)
     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
     at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
     at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:134)
     at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:597)
     at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
     at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
     at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
     at org.jboss.jms.client.JBossConnectionFactory.createConnection(JBossConnectionFactory.java:87)
(...)

- and finally the client start to listen again
 
Everything looks fine after that.
But all of the stacktraces come from the "receiver" thread. None of them come from "sender" thread.
And after all that fatal failovers none message can be send.
 
The "sender" thread is blocked. Here is the snapshoot from VisualVM:
(...)
"sender" prio=10 tid=0x8f62c800 nid=0x4a1a in Object.wait() [0x8f3c7000]
   java.lang.Thread.State: WAITING (on object monitor)
     at java.lang.Object.wait(Native Method)
     - waiting on <0x947601a0> (a org.jboss.jms.client.container.ClosedInterceptor)
     at java.lang.Object.wait(Object.java:485)
     at org.jboss.jms.client.container.ClosedInterceptor.checkCloseAlreadyDone(ClosedInterceptor.java:245)
     - locked <0x947601a0> (a org.jboss.jms.client.container.ClosedInterceptor)
     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:142)
     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
     at org.jboss.jms.client.delegate.ClientProducerDelegate.close(ClientProducerDelegate.java)
     at org.jboss.jms.client.container.ClosedInterceptor.maintainRelatives(ClosedInterceptor.java:306)
     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:165)
     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
     at org.jboss.jms.client.delegate.ClientSessionDelegate.closing(ClientSessionDelegate.java)
     at org.jboss.jms.client.container.ClosedInterceptor.maintainRelatives(ClosedInterceptor.java:305)
     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:165)
     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
     at org.jboss.jms.client.delegate.ClientConnectionDelegate.closing(ClientConnectionDelegate.java)
     at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:208)
     at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:124)
     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
     at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
     at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:269)
     at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect_z_handleSend_26293492.invoke(ProducerAspect_z_handleSend_26293492.java)
     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
     at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
     at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
     at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)
     at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
     at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136)
     at auto.Main.send(Main.java:45)
     at auto.Main.sendWrap(Main.java:70)
     at auto.Main$1.run(Main.java:189)
     at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
     - None
(...)

 
I'm using:
Java JRE 6 (build 1.6.0_17-b04)
JBossMessaging 1.4.3.GA
JBoss 5.1.0.GA
 
The client is ussing port 1099 to lookup the JNDI (is this significant?).
 
The client code is extended version of this: http://pastebin.com/m410ce7d0 (the connection is created and maintainted in the very same way as in this example)
 
Is it an configuration issue or jboss error?

--------------------------------------------------------------

To reply to this message visit the message page: http://community.jboss.org/message/518812#518812

More information about the jboss-user mailing list