[JBoss JIRA] (WFLY-10078) [Artemis 2.x Upgrade] Replicated HA: Live and Backup do not form cluster and initial synchronization is not triggered
by Jeff Mesnil (JIRA)
[ https://issues.jboss.org/browse/WFLY-10078?page=com.atlassian.jira.plugin... ]
Jeff Mesnil updated WFLY-10078:
-------------------------------
Labels: activemq feature-branch-blocker (was: feature-branch-blocker)
> [Artemis 2.x Upgrade] Replicated HA: Live and Backup do not form cluster and initial synchronization is not triggered
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-10078
> URL: https://issues.jboss.org/browse/WFLY-10078
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Affects Versions: 13.0.0.Beta1
> Reporter: Erich Duda
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: activemq, feature-branch-blocker
> Attachments: server1-trace.log, server1.log, server2-trace.log, server2.log
>
>
> Across all replicated HA tests I often see an issue that initial synchronization between Live and Backup was not triggered.
> *Scenario*
> # There are two Wildfly servers configured as replicated Live-Backup pair.
> # Live server is killed/shutdown
> *Expectation:* Backup server becomes active.
> *Reality:* Backup server does not become active, because initial synchronization with Live server was not triggered.
> *Users impact:* Replicated HA feature doesn't work properly.
> *Blocker* priority was set because this is regression against previous Wildfly releases.
> *Detail description of the issue*
> In the trace log there is last message from the {{SharedNothingBackupActivation}} which says that it is waiting on cluster connection. It is interested that JGroups subsystem is booting after this message was logged. However I am not sure if this could cause the issue.
> {code}
> 13:42:06,489 DEBUG [org.apache.activemq.artemis.core.client.impl.Topology] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=null) Topology@6a834ff0[owner=ServerLocatorImpl [initialConnectors=[],
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] is sending topology to org.apache.activemq.artemis.core.server.impl.NamedLiveN
> odeLocatorForReplication@38150570
> 13:42:06,489 TRACE [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=null) Waiting on cluster connection
> 13:42:06,502 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000078: Starting JGroups channel ejb
> 13:42:06,503 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000078: Starting JGroups channel ejb
> 13:42:06,504 INFO [org.infinispan.CLUSTER] (MSC service thread 1-6) ISPN000094: Received new cluster view for channel ejb: [node-1|1] (2) [node-1, node-2]
> 13:42:06,506 INFO [org.infinispan.CLUSTER] (MSC service thread 1-7) ISPN000094: Received new cluster view for channel ejb: [node-1|1] (2) [node-1, node-2]
> 13:42:06,506 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000078: Starting JGroups channel ejb
> 13:42:06,506 INFO [org.infinispan.CLUSTER] (MSC service thread 1-8) ISPN000094: Received new cluster view for channel ejb: [node-1|1] (2) [node-1, node-2]
> 13:42:06,510 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000078: Starting JGroups channel ejb
> 13:42:06,510 INFO [org.infinispan.CLUSTER] (MSC service thread 1-3) ISPN000094: Received new cluster view for channel ejb: [node-1|1] (2) [node-1, node-2]
> 13:42:06,516 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000079: Channel ejb local address is node-2, physical addresses are [127.0.0.1:56200]
> 13:42:06,527 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000079: Channel ejb local address is node-2, physical addresses are [127.0.0.1:56200]
> 13:42:06,527 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000079: Channel ejb local address is node-2, physical addresses are [127.0.0.1:56200]
> 13:42:06,529 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000079: Channel ejb local address is node-2, physical addresses are [127.0.0.1:56200]
> 13:42:06,551 INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-7) ISPN000128: Infinispan version: Infinispan 'Gaina' 9.2.0.Final
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (WFLY-10075) [Artemis 2.x upgrade] Stuck messages in artemis.internal.sf.my-cluster... queue after restarting nodes in cluster
by Jeff Mesnil (JIRA)
[ https://issues.jboss.org/browse/WFLY-10075?page=com.atlassian.jira.plugin... ]
Jeff Mesnil updated WFLY-10075:
-------------------------------
Labels: activemq feature-branch-blocker (was: )
> [Artemis 2.x upgrade] Stuck messages in artemis.internal.sf.my-cluster... queue after restarting nodes in cluster
> -----------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-10075
> URL: https://issues.jboss.org/browse/WFLY-10075
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Miroslav Novak
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: activemq, feature-branch-blocker
> Attachments: journal-node-2.txt
>
>
> There are lost messages in scenario where nodes in cluster are cleanly stopped and started again. This issue was hit with Artemis 2.5.0.Final and WF Jeff's integration branch WFLY-9407_upgrade_artemis_2.4.0_with_prefix.
> Test Scenario:
> * start two servers in cluster (JGroups used for discovery)
> * send messages to testQueue0 on node-1 and node-2
> * wait until consumers on both nodes receive 300 messages
> * cleanly shut down 1st and then 2nd server
> * leave servers shut down for one minute
> * start both servers
> * wait until both consumers receive 500 messages
> * stop sending messages and receive all remaining messages
> Pass Criteria: All send messages are received by consumer
> Actual Result: There are lost messages.
> Investigation:
> There are lost messages which were sent to 2nd node. However they got stuck in queue {{.artemis.internal.sf.my-cluster.8a7e9e98-2c36-11e8-9737-fa163ea20b26}} during load balancing to 1st server.
> I'm attaching trace logs from client and servers and content of journal from 2nd server.
> This is regression against Artemis 1.5.5 thus setting blocker priority.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (WFLY-10105) [Artemis 2.x upgrade] Undelivered messages in cluster test with JDBC persistent store
by Jeff Mesnil (JIRA)
[ https://issues.jboss.org/browse/WFLY-10105?page=com.atlassian.jira.plugin... ]
Jeff Mesnil updated WFLY-10105:
-------------------------------
Labels: activemq feature-branch-blocker (was: activemq)
> [Artemis 2.x upgrade] Undelivered messages in cluster test with JDBC persistent store
> -------------------------------------------------------------------------------------
>
> Key: WFLY-10105
> URL: https://issues.jboss.org/browse/WFLY-10105
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Erich Duda
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: activemq, feature-branch-blocker
> Attachments: server1-logs.zip, server2-logs.zip, test.log.zip
>
>
> *Scenario:*
> * There are two Wildfly servers in cluster. Both use JDBC persistence store.
> * Producer sends 6000 messages to server 1.
> * After all messages are sent, server 2 is killed and restarted.
> * Receiver receives messages from server 2.
> *Expectation:* All messages sent by producer are received by receiver from server 2.
> *Reality:* Only 5055 messages from 6000 are received.
> *Blocker* priority was set because it is regression against previous versions of Wildfly.
> The issue was hit with Artemis 2.5.0 with https://github.com/jmesnil/wildfly/tree/WFLY-9407_upgrade_artemis_2.4.0_w... (commit 51dd8102f103ccb0470a3cfc8713d3f9bdb1b65d)
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (WFLY-10101) [Artemis 2.x upgrade] Unexptected crash of server in SOAK test
by Jeff Mesnil (JIRA)
[ https://issues.jboss.org/browse/WFLY-10101?page=com.atlassian.jira.plugin... ]
Jeff Mesnil updated WFLY-10101:
-------------------------------
Labels: feature-branch-blocker (was: )
> [Artemis 2.x upgrade] Unexptected crash of server in SOAK test
> --------------------------------------------------------------
>
> Key: WFLY-10101
> URL: https://issues.jboss.org/browse/WFLY-10101
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Miroslav Novak
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: feature-branch-blocker
>
> After ~13 hours there is unexpected crash of one server in SOAK test. There is no error/warning in the logs.
> Test Scenario:
> * Start 2 servers
> * Client sends messages to input queue. Messages then go through:
> * One server to another through MDB reading and sending them from remote container through resource adapter
> * Messages are forwarded from one server to another over JMS bridge and back over Core bridge
> * Messages have JMSReplyTo defined with a temporary queue, that is filled with responses for the client
> * Messages are read from the destination with stateless EJB and sent back to clients
> * Client reads the messages after the pass through all the soak modules.
> Pass Criteria: In the last step receiver consumes all messages sent by producer.
> Actual Result:
> After ~13 hours 1st server suddenly crashes. There is no error/warning in server logs.
> Issue was hit with Artemis 2.5.0 with https://github.com/jmesnil/wildfly/tree/WFLY-9407_upgrade_artemis_2.4.0_w... (commit 51dd8102f103ccb0470a3cfc8713d3f9bdb1b65d)
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (WFLY-10094) [Artemis 2.x upgrade] libAIO does not get loaded on RHEL 6 x86_64
by Jeff Mesnil (JIRA)
[ https://issues.jboss.org/browse/WFLY-10094?page=com.atlassian.jira.plugin... ]
Jeff Mesnil updated WFLY-10094:
-------------------------------
Labels: activemq feature-branch-blocker (was: activemq)
> [Artemis 2.x upgrade] libAIO does not get loaded on RHEL 6 x86_64
> -----------------------------------------------------------------
>
> Key: WFLY-10094
> URL: https://issues.jboss.org/browse/WFLY-10094
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Miroslav Novak
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: activemq, feature-branch-blocker
>
> LibAIO does not get loaded on RHEL 6 x86_64:
> {code}
> [hudson@rhel6-large-2723 bin]$ sh standalone.sh -c standalone-full-ha.xml -Djboss.socket.binding.port-offset=1000
> =========================================================================
> JBoss Bootstrap Environment
> JBOSS_HOME: /tmp/jboss-eap
> JAVA: java
> JAVA_OPTS: -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
> =========================================================================
> ...
> 10:14:31,918 INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-1) WFLYMSGAMQ0075: AIO wasn't located on this platform, it will fall back to using pure Java NIO. Your platform is Linux, install LibAIO to enable the AIO journal and achieve optimal performance.
> 10:14:32,017 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 75) AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=/tmp/jboss-eap/standalone/data/activemq/journal,bindingsDirectory=/tmp/jboss-eap/standalone/data/activemq/bindings,largeMessagesDirectory=/tmp/jboss-eap/standalone/data/activemq/largemessages,pagingDirectory=/tmp/jboss-eap/standalone/data/activemq/paging)
> 10:14:32,110 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 75) AMQ221013: Using NIO Journal
> ...
> {code}
> libAIO in artemis journal module from WF branch. Not from Artemis 2.5.0.Final tag.
> Wildfly: https://github.com/jmesnil/wildfly/tree/WFLY-9407_upgrade_artemis_2.4.0_w... (06c878a313d3cad323889d017e60fd5533204d1a)
> Artemis: upstreadm master (577b62d5210cdcc0f86ab9bb1b24e944c877dfe7)
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (WFLY-10089) [Artemis 2.x upgrade] Lost message when MDB is resending messages under high load
by Jeff Mesnil (JIRA)
[ https://issues.jboss.org/browse/WFLY-10089?page=com.atlassian.jira.plugin... ]
Jeff Mesnil updated WFLY-10089:
-------------------------------
Labels: activemq feature-branch-blocker (was: activemq)
> [Artemis 2.x upgrade] Lost message when MDB is resending messages under high load
> ---------------------------------------------------------------------------------
>
> Key: WFLY-10089
> URL: https://issues.jboss.org/browse/WFLY-10089
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Miroslav Novak
> Assignee: Martyn Taylor
> Priority: Blocker
> Labels: activemq, feature-branch-blocker
>
> Test scenario:
> * Start cluster A of nodes node-1, node-3
> * Start cluster B of nodes node-2, node-4
> * Send messages to queue on node-1
> * Deploy mdbs to servers in cluster A. This mdb reads messages from local queue, sends them to remote queue on cluster B and inserts them into database
> * Deploy mdbs to servers in cluster B. This mdb reads messages from local queue and inserts them into database
> * Cause CPU overload (for 5 min) on server node-2 when mdbs on cluster1 and 2 are processing mesages
> * Restart failed server
> * Let MDBs to process remaining messages
> Pass Criteria: Number of sent message is equal number of records(lines) in database and messages in
> Actual Result:
> Sometimes happens that one record is missing in database which means that one message was not processed be MDB in cluster 2.
> This looks like broker related regression against Artemis 1.5.
> Wildfly: https://github.com/jmesnil/wildfly/tree/WFLY-9407_upgrade_artemis_2.4.0_w... (06c878a313d3cad323889d017e60fd5533204d1a)
> Artemis tag 2.5.0.Final
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (WFLY-10114) [Artemis 2.x upgrade] Broken managed connections after suspending remote broker
by Jeff Mesnil (JIRA)
[ https://issues.jboss.org/browse/WFLY-10114?page=com.atlassian.jira.plugin... ]
Jeff Mesnil updated WFLY-10114:
-------------------------------
Labels: feature-branch-blocker (was: )
> [Artemis 2.x upgrade] Broken managed connections after suspending remote broker
> -------------------------------------------------------------------------------
>
> Key: WFLY-10114
> URL: https://issues.jboss.org/browse/WFLY-10114
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Miroslav Novak
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: feature-branch-blocker
>
> Test Scenario:
> * Start 2 servers in cluster with deployed InQueue and OutQueue (jms cluster)
> * Send 50000 messages (10KB) to InQueue
> * Start 2 other servers (mdb nodes) which have configured remote JCA to connect to jms cluster
> * Deploy MDB to each mdb node
> ** MDB consumes messages from InQueue and resend to OutQueue
> * Suspend one of JMS servers (node-3) for 5 minutes when MDBs are processing messages
> * Resume JMS servers and wait for MDBs to process all messages
> Pass Criteria: All messages sent to InQueue well get to OutQueue
> Actual Result:
> There are errors on one of the MDB servers node-4 after node-3 is resumed:
> {code}
> 21:35:05,813 ERROR [org.jboss.qa.hornetq.apps.mdb.MdbWithRemoteOutQueueWithOutQueueLookups] (Thread-168 (ActiveMQ-client-global-threads)) Could not create a session: IJ000453: Unable to get managed connection fo
> r java:/JmsXA: javax.jms.JMSException: Could not create a session: IJ000453: Unable to get managed connection for java:/JmsXA
> at org.apache.activemq.artemis.ra.ActiveMQRASessionFactoryImpl.allocateConnection(ActiveMQRASessionFactoryImpl.java:909)
> at org.apache.activemq.artemis.ra.ActiveMQRASessionFactoryImpl.createSession(ActiveMQRASessionFactoryImpl.java:531)
> at org.jboss.qa.hornetq.apps.mdb.MdbWithRemoteOutQueueWithOutQueueLookups.onMessage(MdbWithRemoteOutQueueWithOutQueueLookups.java:89) [lodhLikemdb1.jar:]
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) [:1.8.0_162]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_162]
> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_162]
> at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
> at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:253) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:332) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:238) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPS
> HOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.messagedriven.MessageDrivenComponentDescription$5$1.processInvocation(MessageDrivenComponentDescription.java:243) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNA
> PSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.messagedriven.MessageDrivenComponentDescription$5$1.processInvocation(MessageDrivenComponentDescription.java:243) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNA
> PSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619) [wildfly-elytron-1.2.2.Final.jar:1.2.2.Final]
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
> at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)
> at org.jboss.qa.hornetq.apps.mdb.MdbWithRemoteOutQueueWithOutQueueLookups$$$view1.onMessage(Unknown Source) [lodhLikemdb1.jar:]
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) [:1.8.0_162]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_162]
> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_162]
> at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.doInvoke(MessageEndpointInvocationHandler.java:139) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.ejb3.inflow.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:73) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.qa.hornetq.apps.mdb.MdbWithRemoteOutQueueWithOutQueueLookups$$$endpoint1.onMessage(Unknown Source) [lodhLikemdb1.jar:]
> at org.apache.activemq.artemis.ra.inflow.ActiveMQMessageHandler.onMessage(ActiveMQMessageHandler.java:302)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1002) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1125) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0.jar:2.5.0]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_162]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_162]
> Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/JmsXA
> at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:690)
> at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:430)
> at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:789)
> at org.apache.activemq.artemis.ra.ActiveMQRASessionFactoryImpl.allocateConnection(ActiveMQRASessionFactoryImpl.java:872)
> ... 69 more
> Caused by: javax.resource.ResourceException: IJ000655: No managed connections available within configured blocking timeout (30000 [ms])
> at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.getConnection(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:570)
> at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getTransactionNewConnection(AbstractPool.java:714)
> at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:613)
> at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:624)
> ... 72 more
> {code}
> later on there are errors:
> {code}
> 21:35:52,119 WARN [org.apache.activemq.artemis.core.client] (Thread-218 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,120 WARN [org.apache.activemq.artemis.core.client] (Thread-208 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,122 WARN [org.apache.activemq.artemis.core.client] (Thread-161 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,123 WARN [org.apache.activemq.artemis.core.client] (Thread-172 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,124 WARN [org.apache.activemq.artemis.core.client] (Thread-167 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,124 WARN [org.apache.activemq.artemis.core.client] (Thread-174 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,124 WARN [org.apache.activemq.artemis.core.client] (Thread-176 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,124 WARN [org.apache.activemq.artemis.core.client] (Thread-196 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,124 WARN [org.apache.activemq.artemis.core.client] (Thread-157 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:53,087 WARN [org.apache.activemq.artemis.core.client] (Thread-163 (ActiveMQ-client-global-threads)) AMQ212052: Packet PACKET(SessionXAResponseMessage)[type=55, channelID=10, packetObject=SessionXAResponse
> Message, error=false, message=null, responseCode=0] was answered out of sequence due to a previous server timeout and it's being ignored: java.lang.Exception: trace
> at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:395) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:319) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sessionStop(ActiveMQSessionContext.java:399) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.stop(ClientSessionImpl.java:1019) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.stop(ClientSessionImpl.java:1008) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:912) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:895) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.resetIfNeeded(ClientSessionImpl.java:985) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.ra.inflow.ActiveMQMessageHandler.onMessage(ActiveMQMessageHandler.java:371)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1002) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1125) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0.jar:2.5.0]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_162]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_162]
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.5.0.jar:2.5.0]
> {code}
> However mdb server never recovers from this situation. It seems that all managed connections are left in broken state.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (WFLY-10109) [Artemis 2.x upgrade] MDB connected to remote server does not get activated when using discovery group with JGroups
by Jeff Mesnil (JIRA)
[ https://issues.jboss.org/browse/WFLY-10109?page=com.atlassian.jira.plugin... ]
Jeff Mesnil updated WFLY-10109:
-------------------------------
Labels: feature-branch-blocker (was: )
> [Artemis 2.x upgrade] MDB connected to remote server does not get activated when using discovery group with JGroups
> --------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-10109
> URL: https://issues.jboss.org/browse/WFLY-10109
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Miroslav Novak
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: feature-branch-blocker
>
> Sometimes happens that MDB does not start to consume messages from remote server. There are not warning or error messages in log.
> Test Scenario:
> * Start server with deployed InQueue and OutQueue
> * Start 2nd server with configured pooled-connection-factory to connect to 1st server
> * Deploy MDB to 2nd server - mdb is consuming messages from InQueue and sending to OutQueue
> * Start sending messages to InQueue to 1st server
> * Let MDB processing messages from InQueue
> * Start consuming messages from OutQueue from 1st server
> Pass Criteria: There is the same number of sent and received messages
> Actual Result: Sometimes happens that Mdb does not start to consume messages.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (WFLY-10079) [Artemis 2.x Upgrade] Unable to announce backup
by Jeff Mesnil (JIRA)
[ https://issues.jboss.org/browse/WFLY-10079?page=com.atlassian.jira.plugin... ]
Jeff Mesnil updated WFLY-10079:
-------------------------------
Labels: activemq feature-branch-blocker (was: feature-branch-blocker)
> [Artemis 2.x Upgrade] Unable to announce backup
> -----------------------------------------------
>
> Key: WFLY-10079
> URL: https://issues.jboss.org/browse/WFLY-10079
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Affects Versions: 13.0.0.Beta1
> Reporter: Erich Duda
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: activemq, feature-branch-blocker
> Attachments: server1.log, server2.log, test.log
>
>
> In shared store HA test scenario it sometimes happens that backup repeatedly prints the error message \[1\]. When the live is killed later, the backup doesn't become active.
> *Users impact:* High availability feature does not work properly.
> *Blocker* priority was set because it is regression against previous release of Wildfly.
> In this particular test Netty connectors/acceptors are used.
> \[1\]
> {code}
> 12:07:26,617 WARN [org.apache.activemq.artemis.core.server] (Thread-1 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@ea5272e)) AMQ222137: Unable to announce backup, retrying: ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ119012: Timed out waiting to receive initial broadcast from cluster]
> at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:761) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:637) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:619) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.server.cluster.BackupManager$BackupConnector$1.run(BackupManager.java:246) [artemis-server-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0.jar:2.5.0]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_162]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_162]
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.5.0.jar:2.5.0]
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years