[JBoss JIRA] Created: (JBESB-2184) Endless loop during cluster failover, caused by JmsCourier.jmsConnectRetry
by Joakim Sandström (JIRA)
Endless loop during cluster failover, caused by JmsCourier.jmsConnectRetry
--------------------------------------------------------------------------
Key: JBESB-2184
URL: https://jira.jboss.org/jira/browse/JBESB-2184
Project: JBoss ESB
Issue Type: Bug
Security Level: Public (Everyone can see)
Components: Transports
Affects Versions: 4.4
Environment: 4 node JBoss ESB 4.4 cluster, clustered JBM 1.4.0 SP3, replicated MySQL 5.0
Reporter: Joakim Sandström
JmsCourier ends up in endless loop during cluster failover, due to a bug in JmsCourier.jmsConnectRetry.
The 'failed to route' JMSException happens outside the JmsCourier.jmsConnectRetry, thus JmsCourier.jmsConnectRetry is invoked over and over again, and maxRetry is always reset
in the method.
final int maxRetry = 5; // TODO Magic number here!!!
for (int i1 = 0; i1 < maxRetry; i1++) {
2008-11-10 16:01:29,303 ERROR [org.jboss.messaging.util.ExceptionUtil] SessionEndpoint[16-txdx5dnf-1-198q5dnf-8hclqf-kn414q1] send [le-nshn6dnf-1-198q5dnf-8hclqf-kn41
4q1]
javax.jms.JMSException: Failed to route Reference[446473]:RELIABLE to myChannel
at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:743)
at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:795)
at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
at org.jboss.remoting.Client.invoke(Client.java:1634)
at org.jboss.remoting.Client.invoke(Client.java:548)
at org.jboss.remoting.Client.invoke(Client.java:536)
at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:477)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:632)
at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect29.invoke(SessionAspect29.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:278)
at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect47.invoke(ProducerAspect47.java)
at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)
at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136)
at org.jboss.internal.soa.esb.couriers.JmsCourier.deliver(JmsCourier.java:252)
at org.jboss.internal.soa.esb.couriers.JmsCourier.deliver(JmsCourier.java:213)
at org.jboss.internal.soa.esb.couriers.TwoWayCourierImpl.deliver(TwoWayCourierImpl.java:201)
at org.jboss.soa.esb.client.ServiceInvoker$EPRInvoker.attemptDelivery(ServiceInvoker.java:636)
at org.jboss.soa.esb.client.ServiceInvoker$EPRInvoker.access$2(ServiceInvoker.java:570)
at org.jboss.soa.esb.client.ServiceInvoker.post(ServiceInvoker.java:371)
at org.jboss.soa.esb.client.ServiceInvoker.deliverAsync(ServiceInvoker.java:252)
at org.jboss.soa.esb.client.MessageMulticaster.sendToSubset(MessageMulticaster.java:144)
at org.jboss.soa.esb.actions.ContentBasedWiretap.routeMessage(ContentBasedWiretap.java:187)
at org.jboss.soa.esb.actions.ContentBasedWiretap.process(ContentBasedWiretap.java:153)
at org.jboss.soa.esb.actions.ContentBasedRouter.process(ContentBasedRouter.java:58)
at org.jboss.soa.esb.listeners.message.ActionProcessingPipeline.process(ActionProcessingPipeline.java:316)
at org.jboss.soa.esb.listeners.message.MessageAwareListener$TransactionalRunner.run(MessageAwareListener.java:530)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
2008-11-10 16:01:29,306 DEBUG [org.jboss.internal.soa.esb.couriers.JmsCourier] JMS error. Attempting JMS reconnect.
javax.jms.JMSException: Failed to route Reference[446473]:RELIABLE to myChannel
at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:743)
at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:795)
at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
at org.jboss.remoting.Client.invoke(Client.java:1634)
at org.jboss.remoting.Client.invoke(Client.java:548)
at org.jboss.remoting.Client.invoke(Client.java:536)
at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:477)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:632)
at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect29.invoke(SessionAspect29.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:278)
at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect47.invoke(ProducerAspect47.java)
at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)
at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136)
at org.jboss.internal.soa.esb.couriers.JmsCourier.deliver(JmsCourier.java:252)
at org.jboss.internal.soa.esb.couriers.JmsCourier.deliver(JmsCourier.java:213)
at org.jboss.internal.soa.esb.couriers.TwoWayCourierImpl.deliver(TwoWayCourierImpl.java:201)
at org.jboss.soa.esb.client.ServiceInvoker$EPRInvoker.attemptDelivery(ServiceInvoker.java:636)
at org.jboss.soa.esb.client.ServiceInvoker$EPRInvoker.access$2(ServiceInvoker.java:570)
at org.jboss.soa.esb.client.ServiceInvoker.post(ServiceInvoker.java:371)
at org.jboss.soa.esb.client.ServiceInvoker.deliverAsync(ServiceInvoker.java:252)
at org.jboss.soa.esb.client.MessageMulticaster.sendToSubset(MessageMulticaster.java:144)
at org.jboss.soa.esb.actions.ContentBasedWiretap.routeMessage(ContentBasedWiretap.java:187)
at org.jboss.soa.esb.actions.ContentBasedWiretap.process(ContentBasedWiretap.java:153)
at org.jboss.soa.esb.actions.ContentBasedRouter.process(ContentBasedRouter.java:58)
at org.jboss.soa.esb.listeners.message.ActionProcessingPipeline.process(ActionProcessingPipeline.java:316)
at org.jboss.soa.esb.listeners.message.MessageAwareListener$TransactionalRunner.run(MessageAwareListener.java:530)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
2008-11-10 16:01:29,335 DEBUG [org.jboss.internal.soa.esb.couriers.JmsCourier] JMSCourier deliveryMode: 2, peristent:true
2008-11-10 16:01:29,335 ERROR [org.jboss.messaging.util.ExceptionUtil] SessionEndpoint[16-txdx5dnf-1-198q5dnf-8hclqf-kn414q1] send [me-jthn6dnf-1-198q5dnf-8hclqf-kn41
4q1]
javax.jms.JMSException: Failed to route Reference[446474]:RELIABLE to myChannel
at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:743)
at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
17 years, 5 months
[JBoss JIRA] Created: (JBESB-2183) Wrong ReplyTo after message delivery failure
by Joakim Sandström (JIRA)
Wrong ReplyTo after message delivery failure
--------------------------------------------
Key: JBESB-2183
URL: https://jira.jboss.org/jira/browse/JBESB-2183
Project: JBoss ESB
Issue Type: Bug
Security Level: Public (Everyone can see)
Components: Transports
Affects Versions: 4.4
Environment: 4 node JBoss ESB 4.4 cluster, clustered JBM 1.4.0 SP3, replicated MySQL 5.0
Reporter: Joakim Sandström
ServiceInvoker does not reset ReplyTO when message delivery fails, which causes the next EPR (for the same service) to get the wrong ReplyTO address
Steps to reproduce:
Node1: 10.5.0.229
Node2: 10.5.0.230
1. uddi contains all cluster node EPR's
2. kill the node (10.5.0.230) behind the first EPR for a service in uddi
3. invoke the service
4. ServiceInvoker sends a mesage to jms://10.5.0.230:1100/queue/myserviceEsbChannel, which failes because the node is down
4.1 ReplyTo is jms://10.5.0.230:1100/queue/myserviceEsbChannel_reply
5. ServiceInvoker sends a message to the next EPR jms://10.5.0.229:1100/queue/myserviceEsbChannel, which succeeds
5.1 ReplyTo IS STILL jms://10.5.0.230:1100/queue/myserviceEsbChannel_reply
6. Message gets delivered successfully, but the reply fails
First EPR, message sent to node2, reply to node2:
deliver message [header: [ To: JMSEpr [ PortReference < <wsa:Address jms://10.5.0.230:1100/queue/myserviceEsbChannel/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.initial : org.jnp.interfaces.NamingContextFactory/>, <wsa:ReferenceProperties jbossesb:java.naming.provider.url : 10.5.0.230:1100/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.url.pkgs : org.jboss.naming:org.jnp.interfaces/>, <wsa:ReferenceProperties jbossesb:destination-type : queue/>, <wsa:ReferenceProperties jbossesb:specification-version : 1.1/>, <wsa:ReferenceProperties jbossesb:connection-factory :ClusteredConnectionFactory/>, <wsa:ReferenceProperties jbossesb:persistent : true/>, <wsa:ReferenceProperties jbossesb:acknowledge-mode : AUTO_ACKNOWLEDGE/>, <wsa:ReferenceProperties jbossesb:transacted : false/>, <wsa:ReferenceProperties jbossesb:type : urn:jboss/esb/epr/type/jms/> > ]ReplyTo: JMSEpr [ PortReference < <wsa:Address jms://10.5.0.230:1100/queue/myserviceEsbChannel_reply/>, <wsa:ReferenceProperties jbossesb:java.naming.provider.url : 10.5.0.230:1100/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.initial : org.jnp.interfaces.NamingContextFactory/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.url.pkgs : org.jboss.naming:org.jnp.interfaces/>, <wsa:ReferenceProperties jbossesb:destination-type : queue/>, <wsa:ReferenceProperties jbossesb:specification-version : 1.1/>, <wsa:ReferenceProperties jbossesb:connection-factory : ClusteredConnectionFactory/>, <wsa:ReferenceProperties jbossesb:message-selector : jbossESBresponseUUID='de1ddfaf-6d45-4d03-b9ab-97b72244457c'/>, <wsa:ReferenceProperties jbossesb:persistent : true/>, <wsa:ReferenceProperties jbossesb:acknowledge-mode : AUTO_ACKNOWLEDGE/>, <wsa:ReferenceProperties jbossesb:transacted : false/>, <wsa:ReferenceProperties jbossesb:type : urn:jboss/esb/epr/type/jms/> > ] ]] to Service [mycategory:myservice].
Second EPR, message sent to node1, reply to is still node2:
deliver message [header: [ To: JMSEpr [ PortReference < <wsa:Address jms://10.5.0.229:1100/queue/myserviceEsbChannel/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.initial : org.jnp.interfaces.NamingContextFactory/>, <wsa:ReferenceProperties jbossesb:java.naming.provider.url : 10.5.0.229:1100/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.url.pkgs : org.jboss.naming:org.jnp.interfaces/>, <wsa:ReferenceProperties jbossesb:destination-type : queue/>, <wsa:ReferenceProperties jbossesb:specification-version : 1.1/>, <wsa:ReferenceProperties jbossesb:connection-factory :ClusteredConnectionFactory/>, <wsa:ReferenceProperties jbossesb:persistent : true/>, <wsa:ReferenceProperties jbossesb:acknowledge-mode : AUTO_ACKNOWLEDGE/>, <wsa:ReferenceProperties jbossesb:transacted : false/>, <wsa:ReferenceProperties jbossesb:type : urn:jboss/esb/epr/type/jms/> > ]ReplyTo: JMSEpr [ PortReference < <wsa:Address jms://10.5.0.230:1100/queue/myserviceEsbChannel_reply/>, <wsa:ReferenceProperties jbossesb:java.naming.provider.url : 10.5.0.230:1100/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.initial : org.jnp.interfaces.NamingContextFactory/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.url.pkgs : org.jboss.naming:org.jnp.interfaces/>, <wsa:ReferenceProperties jbossesb:destination-type : queue/>, <wsa:ReferenceProperties jbossesb:specification-version : 1.1/>, <wsa:ReferenceProperties jbossesb:connection-factory : ClusteredConnectionFactory/>, <wsa:ReferenceProperties jbossesb:message-selector : jbossESBresponseUUID='de1ddfaf-6d45-4d03-b9ab-97b72244457c'/>, <wsa:ReferenceProperties jbossesb:persistent : true/>, <wsa:ReferenceProperties jbossesb:acknowledge-mode : AUTO_ACKNOWLEDGE/>, <wsa:ReferenceProperties jbossesb:transacted : false/>, <wsa:ReferenceProperties jbossesb:type : urn:jboss/esb/epr/type/jms/> > ] ]] to Service [mycategory:myservice].
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
17 years, 5 months
[JBoss JIRA] Created: (JBESB-866) jUDDI retains invalid EPRs
by Jiri Pechanec (JIRA)
jUDDI retains invalid EPRs
--------------------------
Key: JBESB-866
URL: http://jira.jboss.com/jira/browse/JBESB-866
Project: JBoss ESB
Issue Type: Bug
Security Level: Public (Everyone can see)
Components: Registry and Repository
Affects Versions: 4.2 Milestone Release 3
Reporter: Jiri Pechanec
Assigned To: Mark Little
If the ESB is terminated (crashes) it will retain EPRs in the database. Unfortunately when the EPR is changed e.g. by renaming of underlaying queue, both old and new EPRs are stored in jUDDI registry on the next start. Then under heavier load both EPRs are selected with the result of lot error messages related to unavailable EPR.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
17 years, 5 months