[jboss-jira] [JBoss JIRA] (WFWIP-119) [Artemis 2.x upgrade] Lost message in replicated HA scenario with MDB

Erich Duda (JIRA) issues at jboss.org
Thu Aug 23 03:27:00 EDT 2018 

    [ https://issues.jboss.org/browse/WFWIP-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13622992#comment-13622992 ] 

Erich Duda commented on WFWIP-119:
----------------------------------

[~clebert.suconic] I've looked at \[1\] and \[2\].

\[1\] failed on "getCountOfMessagesOnQueue() failed for queue:OutQueue and container: node-1. Number of tries: 10". Boot of node-1 failed with an following exception. Do you know what could happen?

{code}
21:57:16,375 ERROR [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ224000: Failure in initialisation: java.lang.NullPointerException
        at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1745) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1181) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2620) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2375) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:376) [artemis-jms-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:206) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
        at org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:65) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
        at org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:100) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_181]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
        at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
        at org.jboss.threads.JBossThread.run(JBossThread.java:485) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]

21:57:16,375 ERROR [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ224000: Failure in initialisation: java.lang.NullPointerException
        at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1745) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPS
HOT]
        at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1181) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2620) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2375) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:376) [artemis-jms-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:206) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
        at org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:65) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
        at org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:100) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_181]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
        at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
        at org.jboss.threads.JBossThread.run(JBossThread.java:485) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]

21:57:16,377 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 80) MSC000001: Failed to start service jboss.messaging-activemq.default.jms.manager: org.jboss.msc.service.StartException in service jboss.messaging-activemq.default.jms.manager: java.lang.NullPointerException
        at org.wildfly.extension.messaging.activemq.jms.JMSService.lambda$doStart$0(JMSService.java:142) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.callActivationFailureListeners(ActiveMQServerImpl.java:1919) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:127) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:376) [artemis-jms-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:206) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
        at org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:65) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
        at org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:100) [wildfly-messaging-activemq-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_181]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
        at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
        at org.jboss.threads.JBossThread.run(JBossThread.java:485) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
Caused by: java.lang.NullPointerException
        at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1745) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1181) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2620) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2375) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        ... 14 more
{code}

The \[2\] failed because of the issue you mentioned that receiver didn't receive messages from node-4. However it is not test issue but it is caused by bug in JGroups. I saw in different test that if JGroups are used, Artemis cluster is not formed sometimes. It is reported in WFWIP-90. I will re-run the job with Netty discovery. Here is the new run \[1\]. If you will run the test by yourself be sure that there is that there is {{-Dprepare.param.CLUSTER_TYPE=NETTY_DISCOVERY}} in {{ADDITIONAL_PARAMS}} field.

\[1\] https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messaging-retry-one-test-trigger-face/127/
\[2\] https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messaging-run-one-test-until-fail/207/
\[3\] https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messaging-run-one-test-until-fail/209/

> [Artemis 2.x upgrade] Lost message in replicated HA scenario with MDB
> ---------------------------------------------------------------------
>
>                 Key: WFWIP-119
>                 URL: https://issues.jboss.org/browse/WFWIP-119
>             Project: WildFly WIP
>          Issue Type: Bug
>          Components: Artemis
>            Reporter: Erich Duda
>            Assignee: Martyn Taylor
>            Priority: Blocker
>
> *Scenario*
> * Start live/backup pair in replicated dedicated topology and start one more server which is in Artemis cluster with live.
> * Send messages to queue InQueue.
> * Start another server with MDB connected to cluster with live server which resend messages from InQueue to OutQueue.
> * Stop backup server and then live server.
> * Stop server which was in cluster with live server.
> * Start live server and server in cluster.
> * Start backup server.
> The test fails because one message get lost. It wasn't sent to OutQueue.
> Logs from failed test including traces you can find in \[1\].
> \[1\] https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messaging-run-one-test-until-fail/201/



--
This message was sent by Atlassian JIRA
(v7.5.0#75005)

More information about the jboss-jira mailing list