[JBoss JIRA] (WFLY-10078) [Artemis 2.x Upgrade] Replicated HA: Live and Backup do not form cluster and initial synchronization is not triggered
by Martyn Taylor (JIRA)
[ https://issues.jboss.org/browse/WFLY-10078?page=com.atlassian.jira.plugin... ]
Martyn Taylor commented on WFLY-10078:
--------------------------------------
Could be related to WFLY-10109
> [Artemis 2.x Upgrade] Replicated HA: Live and Backup do not form cluster and initial synchronization is not triggered
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-10078
> URL: https://issues.jboss.org/browse/WFLY-10078
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Erich Duda
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: activemq, feature-branch-blocker
> Attachments: server1-trace.log, server1.log, server2-trace.log, server2.log
>
>
> Across all replicated HA tests I often see an issue that initial synchronization between Live and Backup was not triggered.
> *Scenario*
> # There are two Wildfly servers configured as replicated Live-Backup pair.
> # Live server is killed/shutdown
> *Expectation:* Backup server becomes active.
> *Reality:* Backup server does not become active, because initial synchronization with Live server was not triggered.
> *Users impact:* Replicated HA feature doesn't work properly.
> *Blocker* priority was set because this is regression against previous Wildfly releases.
> *Detail description of the issue*
> In the trace log there is last message from the {{SharedNothingBackupActivation}} which says that it is waiting on cluster connection. It is interested that JGroups subsystem is booting after this message was logged. However I am not sure if this could cause the issue.
> {code}
> 13:42:06,489 DEBUG [org.apache.activemq.artemis.core.client.impl.Topology] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=null) Topology@6a834ff0[owner=ServerLocatorImpl [initialConnectors=[],
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] is sending topology to org.apache.activemq.artemis.core.server.impl.NamedLiveN
> odeLocatorForReplication@38150570
> 13:42:06,489 TRACE [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=null) Waiting on cluster connection
> 13:42:06,502 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000078: Starting JGroups channel ejb
> 13:42:06,503 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000078: Starting JGroups channel ejb
> 13:42:06,504 INFO [org.infinispan.CLUSTER] (MSC service thread 1-6) ISPN000094: Received new cluster view for channel ejb: [node-1|1] (2) [node-1, node-2]
> 13:42:06,506 INFO [org.infinispan.CLUSTER] (MSC service thread 1-7) ISPN000094: Received new cluster view for channel ejb: [node-1|1] (2) [node-1, node-2]
> 13:42:06,506 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000078: Starting JGroups channel ejb
> 13:42:06,506 INFO [org.infinispan.CLUSTER] (MSC service thread 1-8) ISPN000094: Received new cluster view for channel ejb: [node-1|1] (2) [node-1, node-2]
> 13:42:06,510 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000078: Starting JGroups channel ejb
> 13:42:06,510 INFO [org.infinispan.CLUSTER] (MSC service thread 1-3) ISPN000094: Received new cluster view for channel ejb: [node-1|1] (2) [node-1, node-2]
> 13:42:06,516 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000079: Channel ejb local address is node-2, physical addresses are [127.0.0.1:56200]
> 13:42:06,527 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000079: Channel ejb local address is node-2, physical addresses are [127.0.0.1:56200]
> 13:42:06,527 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000079: Channel ejb local address is node-2, physical addresses are [127.0.0.1:56200]
> 13:42:06,529 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000079: Channel ejb local address is node-2, physical addresses are [127.0.0.1:56200]
> 13:42:06,551 INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-7) ISPN000128: Infinispan version: Infinispan 'Gaina' 9.2.0.Final
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 3 months
[JBoss JIRA] (WFLY-10079) [Artemis 2.x Upgrade] Unable to announce backup
by Martyn Taylor (JIRA)
[ https://issues.jboss.org/browse/WFLY-10079?page=com.atlassian.jira.plugin... ]
Martyn Taylor commented on WFLY-10079:
--------------------------------------
Could be related to WFLY-10109
> [Artemis 2.x Upgrade] Unable to announce backup
> -----------------------------------------------
>
> Key: WFLY-10079
> URL: https://issues.jboss.org/browse/WFLY-10079
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Affects Versions: 13.0.0.Beta1
> Reporter: Erich Duda
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: activemq, feature-branch-blocker
> Attachments: server1.log, server2.log, test.log
>
>
> In shared store HA test scenario it sometimes happens that backup repeatedly prints the error message \[1\]. When the live is killed later, the backup doesn't become active.
> *Users impact:* High availability feature does not work properly.
> *Blocker* priority was set because it is regression against previous release of Wildfly.
> In this particular test Netty connectors/acceptors are used.
> \[1\]
> {code}
> 12:07:26,617 WARN [org.apache.activemq.artemis.core.server] (Thread-1 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@ea5272e)) AMQ222137: Unable to announce backup, retrying: ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ119012: Timed out waiting to receive initial broadcast from cluster]
> at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:761) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:637) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:619) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.server.cluster.BackupManager$BackupConnector$1.run(BackupManager.java:246) [artemis-server-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0.jar:2.5.0]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_162]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_162]
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.5.0.jar:2.5.0]
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 3 months
[JBoss JIRA] (WFLY-10101) [Artemis 2.x upgrade] Unexptected crash of server in SOAK test
by Martyn Taylor (JIRA)
[ https://issues.jboss.org/browse/WFLY-10101?page=com.atlassian.jira.plugin... ]
Martyn Taylor reassigned WFLY-10101:
------------------------------------
Assignee: Miroslav Novak (was: Jeff Mesnil)
> [Artemis 2.x upgrade] Unexptected crash of server in SOAK test
> --------------------------------------------------------------
>
> Key: WFLY-10101
> URL: https://issues.jboss.org/browse/WFLY-10101
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Miroslav Novak
> Assignee: Miroslav Novak
> Priority: Blocker
> Labels: feature-branch-blocker
>
> After ~13 hours there is unexpected crash of one server in SOAK test. There is no error/warning in the logs.
> Test Scenario:
> * Start 2 servers
> * Client sends messages to input queue. Messages then go through:
> * One server to another through MDB reading and sending them from remote container through resource adapter
> * Messages are forwarded from one server to another over JMS bridge and back over Core bridge
> * Messages have JMSReplyTo defined with a temporary queue, that is filled with responses for the client
> * Messages are read from the destination with stateless EJB and sent back to clients
> * Client reads the messages after the pass through all the soak modules.
> Pass Criteria: In the last step receiver consumes all messages sent by producer.
> Actual Result:
> After ~13 hours 1st server suddenly crashes. There is no error/warning in server logs.
> Issue was hit with Artemis 2.5.0 with https://github.com/jmesnil/wildfly/tree/WFLY-9407_upgrade_artemis_2.4.0_w... (commit 51dd8102f103ccb0470a3cfc8713d3f9bdb1b65d)
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 3 months
[JBoss JIRA] (WFLY-10105) [Artemis 2.x upgrade] Undelivered messages in cluster test with JDBC persistent store
by Martyn Taylor (JIRA)
[ https://issues.jboss.org/browse/WFLY-10105?page=com.atlassian.jira.plugin... ]
Martyn Taylor reassigned WFLY-10105:
------------------------------------
Assignee: Martyn Taylor (was: Jeff Mesnil)
> [Artemis 2.x upgrade] Undelivered messages in cluster test with JDBC persistent store
> -------------------------------------------------------------------------------------
>
> Key: WFLY-10105
> URL: https://issues.jboss.org/browse/WFLY-10105
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Erich Duda
> Assignee: Martyn Taylor
> Priority: Blocker
> Labels: activemq, feature-branch-blocker
> Attachments: server1-logs.zip, server2-logs.zip, test.log.zip
>
>
> *Scenario:*
> * There are two Wildfly servers in cluster. Both use JDBC persistence store.
> * Producer sends 6000 messages to server 1.
> * After all messages are sent, server 2 is killed and restarted.
> * Receiver receives messages from server 2.
> *Expectation:* All messages sent by producer are received by receiver from server 2.
> *Reality:* Only 5055 messages from 6000 are received.
> *Blocker* priority was set because it is regression against previous versions of Wildfly.
> The issue was hit with Artemis 2.5.0 with https://github.com/jmesnil/wildfly/tree/WFLY-9407_upgrade_artemis_2.4.0_w... (commit 51dd8102f103ccb0470a3cfc8713d3f9bdb1b65d)
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 3 months
[JBoss JIRA] (WFLY-10114) [Artemis 2.x upgrade] Broken managed connections after suspending remote broker
by Martyn Taylor (JIRA)
[ https://issues.jboss.org/browse/WFLY-10114?page=com.atlassian.jira.plugin... ]
Martyn Taylor updated WFLY-10114:
---------------------------------
Labels: activemq feature-branch-blocker (was: feature-branch-blocker)
> [Artemis 2.x upgrade] Broken managed connections after suspending remote broker
> -------------------------------------------------------------------------------
>
> Key: WFLY-10114
> URL: https://issues.jboss.org/browse/WFLY-10114
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Miroslav Novak
> Assignee: Martyn Taylor
> Priority: Blocker
> Labels: activemq, feature-branch-blocker
>
> Test Scenario:
> * Start 2 servers in cluster with deployed InQueue and OutQueue (jms cluster)
> * Send 50000 messages (10KB) to InQueue
> * Start 2 other servers (mdb nodes) which have configured remote JCA to connect to jms cluster
> * Deploy MDB to each mdb node
> ** MDB consumes messages from InQueue and resend to OutQueue
> * Suspend one of JMS servers (node-3) for 5 minutes when MDBs are processing messages
> * Resume JMS servers and wait for MDBs to process all messages
> Pass Criteria: All messages sent to InQueue well get to OutQueue
> Actual Result:
> There are errors on one of the MDB servers node-4 after node-3 is resumed:
> {code}
> 21:35:05,813 ERROR [org.jboss.qa.hornetq.apps.mdb.MdbWithRemoteOutQueueWithOutQueueLookups] (Thread-168 (ActiveMQ-client-global-threads)) Could not create a session: IJ000453: Unable to get managed connection fo
> r java:/JmsXA: javax.jms.JMSException: Could not create a session: IJ000453: Unable to get managed connection for java:/JmsXA
> at org.apache.activemq.artemis.ra.ActiveMQRASessionFactoryImpl.allocateConnection(ActiveMQRASessionFactoryImpl.java:909)
> at org.apache.activemq.artemis.ra.ActiveMQRASessionFactoryImpl.createSession(ActiveMQRASessionFactoryImpl.java:531)
> at org.jboss.qa.hornetq.apps.mdb.MdbWithRemoteOutQueueWithOutQueueLookups.onMessage(MdbWithRemoteOutQueueWithOutQueueLookups.java:89) [lodhLikemdb1.jar:]
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) [:1.8.0_162]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_162]
> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_162]
> at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
> at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:253) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:332) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:238) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPS
> HOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.messagedriven.MessageDrivenComponentDescription$5$1.processInvocation(MessageDrivenComponentDescription.java:243) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNA
> PSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.messagedriven.MessageDrivenComponentDescription$5$1.processInvocation(MessageDrivenComponentDescription.java:243) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNA
> PSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619) [wildfly-elytron-1.2.2.Final.jar:1.2.2.Final]
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
> at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)
> at org.jboss.qa.hornetq.apps.mdb.MdbWithRemoteOutQueueWithOutQueueLookups$$$view1.onMessage(Unknown Source) [lodhLikemdb1.jar:]
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) [:1.8.0_162]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_162]
> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_162]
> at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.doInvoke(MessageEndpointInvocationHandler.java:139) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.ejb3.inflow.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:73) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.qa.hornetq.apps.mdb.MdbWithRemoteOutQueueWithOutQueueLookups$$$endpoint1.onMessage(Unknown Source) [lodhLikemdb1.jar:]
> at org.apache.activemq.artemis.ra.inflow.ActiveMQMessageHandler.onMessage(ActiveMQMessageHandler.java:302)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1002) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1125) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0.jar:2.5.0]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_162]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_162]
> Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/JmsXA
> at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:690)
> at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:430)
> at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:789)
> at org.apache.activemq.artemis.ra.ActiveMQRASessionFactoryImpl.allocateConnection(ActiveMQRASessionFactoryImpl.java:872)
> ... 69 more
> Caused by: javax.resource.ResourceException: IJ000655: No managed connections available within configured blocking timeout (30000 [ms])
> at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.getConnection(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:570)
> at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getTransactionNewConnection(AbstractPool.java:714)
> at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:613)
> at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:624)
> ... 72 more
> {code}
> later on there are errors:
> {code}
> 21:35:52,119 WARN [org.apache.activemq.artemis.core.client] (Thread-218 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,120 WARN [org.apache.activemq.artemis.core.client] (Thread-208 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,122 WARN [org.apache.activemq.artemis.core.client] (Thread-161 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,123 WARN [org.apache.activemq.artemis.core.client] (Thread-172 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,124 WARN [org.apache.activemq.artemis.core.client] (Thread-167 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,124 WARN [org.apache.activemq.artemis.core.client] (Thread-174 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,124 WARN [org.apache.activemq.artemis.core.client] (Thread-176 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,124 WARN [org.apache.activemq.artemis.core.client] (Thread-196 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,124 WARN [org.apache.activemq.artemis.core.client] (Thread-157 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:53,087 WARN [org.apache.activemq.artemis.core.client] (Thread-163 (ActiveMQ-client-global-threads)) AMQ212052: Packet PACKET(SessionXAResponseMessage)[type=55, channelID=10, packetObject=SessionXAResponse
> Message, error=false, message=null, responseCode=0] was answered out of sequence due to a previous server timeout and it's being ignored: java.lang.Exception: trace
> at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:395) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:319) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sessionStop(ActiveMQSessionContext.java:399) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.stop(ClientSessionImpl.java:1019) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.stop(ClientSessionImpl.java:1008) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:912) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:895) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.resetIfNeeded(ClientSessionImpl.java:985) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.ra.inflow.ActiveMQMessageHandler.onMessage(ActiveMQMessageHandler.java:371)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1002) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1125) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0.jar:2.5.0]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_162]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_162]
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.5.0.jar:2.5.0]
> {code}
> However mdb server never recovers from this situation. It seems that all managed connections are left in broken state.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 3 months
[JBoss JIRA] (WFLY-10114) [Artemis 2.x upgrade] Broken managed connections after suspending remote broker
by Martyn Taylor (JIRA)
[ https://issues.jboss.org/browse/WFLY-10114?page=com.atlassian.jira.plugin... ]
Martyn Taylor reassigned WFLY-10114:
------------------------------------
Assignee: Martyn Taylor (was: Jeff Mesnil)
> [Artemis 2.x upgrade] Broken managed connections after suspending remote broker
> -------------------------------------------------------------------------------
>
> Key: WFLY-10114
> URL: https://issues.jboss.org/browse/WFLY-10114
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Miroslav Novak
> Assignee: Martyn Taylor
> Priority: Blocker
> Labels: feature-branch-blocker
>
> Test Scenario:
> * Start 2 servers in cluster with deployed InQueue and OutQueue (jms cluster)
> * Send 50000 messages (10KB) to InQueue
> * Start 2 other servers (mdb nodes) which have configured remote JCA to connect to jms cluster
> * Deploy MDB to each mdb node
> ** MDB consumes messages from InQueue and resend to OutQueue
> * Suspend one of JMS servers (node-3) for 5 minutes when MDBs are processing messages
> * Resume JMS servers and wait for MDBs to process all messages
> Pass Criteria: All messages sent to InQueue well get to OutQueue
> Actual Result:
> There are errors on one of the MDB servers node-4 after node-3 is resumed:
> {code}
> 21:35:05,813 ERROR [org.jboss.qa.hornetq.apps.mdb.MdbWithRemoteOutQueueWithOutQueueLookups] (Thread-168 (ActiveMQ-client-global-threads)) Could not create a session: IJ000453: Unable to get managed connection fo
> r java:/JmsXA: javax.jms.JMSException: Could not create a session: IJ000453: Unable to get managed connection for java:/JmsXA
> at org.apache.activemq.artemis.ra.ActiveMQRASessionFactoryImpl.allocateConnection(ActiveMQRASessionFactoryImpl.java:909)
> at org.apache.activemq.artemis.ra.ActiveMQRASessionFactoryImpl.createSession(ActiveMQRASessionFactoryImpl.java:531)
> at org.jboss.qa.hornetq.apps.mdb.MdbWithRemoteOutQueueWithOutQueueLookups.onMessage(MdbWithRemoteOutQueueWithOutQueueLookups.java:89) [lodhLikemdb1.jar:]
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) [:1.8.0_162]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_162]
> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_162]
> at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
> at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:253) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:332) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:238) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPS
> HOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.messagedriven.MessageDrivenComponentDescription$5$1.processInvocation(MessageDrivenComponentDescription.java:243) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNA
> PSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.messagedriven.MessageDrivenComponentDescription$5$1.processInvocation(MessageDrivenComponentDescription.java:243) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNA
> PSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619) [wildfly-elytron-1.2.2.Final.jar:1.2.2.Final]
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
> at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)
> at org.jboss.qa.hornetq.apps.mdb.MdbWithRemoteOutQueueWithOutQueueLookups$$$view1.onMessage(Unknown Source) [lodhLikemdb1.jar:]
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) [:1.8.0_162]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_162]
> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_162]
> at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.doInvoke(MessageEndpointInvocationHandler.java:139) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.ejb3.inflow.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:73) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.qa.hornetq.apps.mdb.MdbWithRemoteOutQueueWithOutQueueLookups$$$endpoint1.onMessage(Unknown Source) [lodhLikemdb1.jar:]
> at org.apache.activemq.artemis.ra.inflow.ActiveMQMessageHandler.onMessage(ActiveMQMessageHandler.java:302)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1002) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1125) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0.jar:2.5.0]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_162]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_162]
> Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/JmsXA
> at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:690)
> at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:430)
> at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:789)
> at org.apache.activemq.artemis.ra.ActiveMQRASessionFactoryImpl.allocateConnection(ActiveMQRASessionFactoryImpl.java:872)
> ... 69 more
> Caused by: javax.resource.ResourceException: IJ000655: No managed connections available within configured blocking timeout (30000 [ms])
> at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.getConnection(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:570)
> at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getTransactionNewConnection(AbstractPool.java:714)
> at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:613)
> at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:624)
> ... 72 more
> {code}
> later on there are errors:
> {code}
> 21:35:52,119 WARN [org.apache.activemq.artemis.core.client] (Thread-218 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,120 WARN [org.apache.activemq.artemis.core.client] (Thread-208 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,122 WARN [org.apache.activemq.artemis.core.client] (Thread-161 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,123 WARN [org.apache.activemq.artemis.core.client] (Thread-172 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,124 WARN [org.apache.activemq.artemis.core.client] (Thread-167 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,124 WARN [org.apache.activemq.artemis.core.client] (Thread-174 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,124 WARN [org.apache.activemq.artemis.core.client] (Thread-176 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,124 WARN [org.apache.activemq.artemis.core.client] (Thread-196 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,124 WARN [org.apache.activemq.artemis.core.client] (Thread-157 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:53,087 WARN [org.apache.activemq.artemis.core.client] (Thread-163 (ActiveMQ-client-global-threads)) AMQ212052: Packet PACKET(SessionXAResponseMessage)[type=55, channelID=10, packetObject=SessionXAResponse
> Message, error=false, message=null, responseCode=0] was answered out of sequence due to a previous server timeout and it's being ignored: java.lang.Exception: trace
> at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:395) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:319) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sessionStop(ActiveMQSessionContext.java:399) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.stop(ClientSessionImpl.java:1019) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.stop(ClientSessionImpl.java:1008) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:912) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:895) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.resetIfNeeded(ClientSessionImpl.java:985) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.ra.inflow.ActiveMQMessageHandler.onMessage(ActiveMQMessageHandler.java:371)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1002) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1125) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0.jar:2.5.0]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_162]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_162]
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.5.0.jar:2.5.0]
> {code}
> However mdb server never recovers from this situation. It seems that all managed connections are left in broken state.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 3 months
[JBoss JIRA] (WFLY-10131) [Artemis 2.x upgrade] Missing message JMS bridge with DUPS_OK quality of service
by Martyn Taylor (JIRA)
[ https://issues.jboss.org/browse/WFLY-10131?page=com.atlassian.jira.plugin... ]
Martyn Taylor commented on WFLY-10131:
--------------------------------------
May be caused by: WFLY-10067
> [Artemis 2.x upgrade] Missing message JMS bridge with DUPS_OK quality of service
> --------------------------------------------------------------------------------
>
> Key: WFLY-10131
> URL: https://issues.jboss.org/browse/WFLY-10131
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Miroslav Novak
> Assignee: Martyn Taylor
> Labels: activemq
>
> There is missing one message in scenario where JMS bridge (with DUPS_OK) resending messages from one server to another. Durint resending target server is killed and bridge does failover to backup.
> After failover in server logs there are lots of warns/errors:
> {code}
> 05:39:01,697 WARN [org.apache.activemq.artemis.jms.bridge] (Thread-144) AMQ342009: JMS Bridge N/A failed to send + acknowledge batch, closing JMS objects: javax.jms.JMSException: Duplicate message detected - me
> ssage will not be routed. Message information:LargeServerMessage[messageID=2147485500,durable=true,userID=d8ea8e0a-326b-11e8-b74b-fa163e64a220,priority=4, timestamp=Wed Mar 28 05:39:01 EDT 2018,expiration=0, dur
> able=true, address=jms.queue.OutQueue,size=410263,properties=TypedProperties[__AMQ_CID=bd95a355-326b-11e8-b74b-fa163e64a220,_AMQ_ROUTING_TYPE=1,AMQ_BRIDGE_MSG_ID_LIST=ID:c13a41dc-326b-11e8-949b-fa163e64a220,JMSX
> DeliveryCount=10,count=989,color=RED,counter=990,_AMQ_DUPL_ID=fde672e0-e858-4c50-a148-a0b7a4a223591522229901951,_AMQ_LARGE_SIZE=409615]]@1304406481
> at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:423) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:319) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.simpleCommit(ActiveMQSessionContext.java:381) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.commit(ClientSessionImpl.java:862) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.commit(ClientSessionImpl.java:835) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.jms.client.ActiveMQSession.commit(ActiveMQSession.java:231) [artemis-jms-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.sendBatchNonTransacted(JMSBridgeImpl.java:1338) [artemis-jms-server-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.sendBatch(JMSBridgeImpl.java:1298) [artemis-jms-server-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.access$1700(JMSBridgeImpl.java:74) [artemis-jms-server-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl$SourceReceiver.run(JMSBridgeImpl.java:1694) [artemis-jms-server-2.5.0.jar:2.5.0]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_162]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_162]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_162]
> Caused by: ActiveMQDuplicateIdException[errorType=DUPLICATE_ID_REJECTED message=Duplicate message detected - message will not be routed. Message information:LargeServerMessage[messageID=2147485500,durable=true,u
> serID=d8ea8e0a-326b-11e8-b74b-fa163e64a220,priority=4, timestamp=Wed Mar 28 05:39:01 EDT 2018,expiration=0, durable=true, address=jms.queue.OutQueue,size=410263,properties=TypedProperties[__AMQ_CID=bd95a355-326b
> -11e8-b74b-fa163e64a220,_AMQ_ROUTING_TYPE=1,AMQ_BRIDGE_MSG_ID_LIST=ID:c13a41dc-326b-11e8-949b-fa163e64a220,JMSXDeliveryCount=10,count=989,color=RED,counter=990,_AMQ_DUPL_ID=fde672e0-e858-4c50-a148-a0b7a4a2235915
> 22229901951,_AMQ_LARGE_SIZE=409615]]@1304406481]
> ... 13 more
> 05:39:01,699 WARN [org.apache.activemq.artemis.core.server] (Thread-23 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@2076a02)) AMQ222149: Message Reference[3056]:RELIABLE:Co
> reMessage[messageID=3056,durable=true,userID=c1347573-326b-11e8-949b-fa163e64a220,priority=4, timestamp=Wed Mar 28 05:38:21 EDT 2018,expiration=0, durable=true, address=jms.queue.InQueue,size=10673,properties=Ty
> pedProperties[__AMQ_CID=be528a5c-326b-11e8-949b-fa163e64a220,_AMQ_ROUTING_TYPE=1,count=980,counter=981,_AMQ_DUPL_ID=13adf1ec-581f-42e4-835b-0e6e33a4e1191522229901913,color=GREEN]]@511432790 has reached maximum d
> elivery attempts, sending it to Dead Letter Address jms.queue.DLQ from jms.queue.InQueue
> 05:39:01,700 WARN [org.apache.activemq.artemis.core.server] (Thread-23 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@2076a02)) AMQ222149: Message Reference[3058]:RELIABLE:CoreMessage[messageID=3058,durable=true,userID=c1349c84-326b-11e8-949b-fa163e64a220,priority=4, timestamp=Wed Mar 28 05:38:21 EDT 2018,expiration=0, durable=true, address=jms.queue.InQueue,size=20914,properties=TypedProperties[__AMQ_CID=be528a5c-326b-11e8-949b-fa163e64a220,_AMQ_ROUTING_TYPE=1,count=981,counter=982,_AMQ_DUPL_ID=6feb66cd-8616-4dff-8775-3f85a4b9bdbb1522229901914,color=RED]]@1150646585 has reached maximum delivery attempts, sending it to Dead Letter Address jms.queue.DLQ from jms.queue.InQueue
> ....
> 05:39:01,706 WARN [org.apache.activemq.artemis.core.server] (Thread-23 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@2076a02)) AMQ222149: Message Reference[3082]:RELIABLE:CoreMessage[messageID=3082,durable=true,userID=c13d4f1d-326b-11e8-949b-fa163e64a220,priority=4, timestamp=Wed Mar 28 05:38:21 EDT 2018,expiration=0, durable=true, address=jms.queue.InQueue,size=10673,properties=TypedProperties[__AMQ_CID=be528a5c-326b-11e8-949b-fa163e64a220,_AMQ_ROUTING_TYPE=1,count=990,counter=991,_AMQ_DUPL_ID=f802f4b7-e0a1-4dee-acc7-0b11963bec401522229901971,color=GREEN]]@2708478 has reached maximum delivery attempts, sending it to Dead Letter Address jms.queue.DLQ from jms.queue.InQueue
> {code}
> So the missing message ended up in DLQ as it could not be delivered to target queue. It looks like that _AMQ_DUPL_ID might trigger dup message detection in target queueu however such message did not get there.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 3 months
[JBoss JIRA] (WFLY-10115) Regression in JMS bridge with network failures tests
by Martyn Taylor (JIRA)
[ https://issues.jboss.org/browse/WFLY-10115?page=com.atlassian.jira.plugin... ]
Martyn Taylor commented on WFLY-10115:
--------------------------------------
Maybe caused by WFLY-10067
> Regression in JMS bridge with network failures tests
> ----------------------------------------------------
>
> Key: WFLY-10115
> URL: https://issues.jboss.org/browse/WFLY-10115
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Martin Styk
> Assignee: Martyn Taylor
> Priority: Blocker
> Labels: activemq
> Attachments: org.jboss.qa.hornetq.test.bridges.NetworkFailuresJMSBridgesTestCase.zip
>
>
> *Scenario*
> Node A and B are started. JMS bridge with {{once and only once}} delivery and 1 reconnect attempt is deployed on node A. Producer starts sending mix of normal and large messages to InQueue on node A. Jms bridge resends messages from A's InQueue to B's OutQueue. During this time short network failure sequence (network goes down and then up) is executed twice. Receiver receives messages from OutQueue(also from node A). After that, network stays disconnected. Another receiver receives remaining messages from A's InQueue.
> {noformat:title=Jms Bridge}
> <jms-bridge name="myBridge" module="org.apache.activemq.artemis" quality-of-service="ONCE_AND_ONLY_ONCE" failure-retry-interval="1000" max-retries="1" max-batch-size="10" max-batch-time="100" add-messageID-in-header="true">
> <source connection-factory="java:/ConnectionFactory" destination="jms/queue/InQueue"/>
> <target connection-factory="jms/BridgeConnectionFactory" destination="jms/queue/OutQueue">
> <target-context>
> <property name="java.naming.factory.initial" value="org.jboss.naming.remote.client.InitialContextFactory"/>
> <property name="java.naming.provider.url" value="http-remoting://127.0.0.1:9080"/>
> </target-context>
> </target>
> </jms-bridge>
> {noformat}
> *Issue*
> * node A's InQueue and node B's OutQueue is empty
> * There are no prepared transactions on node A
> * some messages *are missing* after test (multiply of 10, which is number of messages in one transaction)
> * this scenario fails *intermittently*
> Issue was hit with Artemis 2.5.0 with https://github.com/jmesnil/wildfly/tree/WFLY-9407_upgrade_artemis_2.4.0_w... (commit 51dd8102f103ccb0470a3cfc8713d3f9bdb1b65d)
> Logs are attached.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 3 months