[JBoss JIRA] (WFWIP-15) [Artemis 2.x upgrade] Broken managed connections after suspending remote broker
by Martin Styk (JIRA)
[ https://issues.jboss.org/browse/WFWIP-15?page=com.atlassian.jira.plugin.s... ]
Martin Styk updated WFWIP-15:
-----------------------------
Component/s: Artemis
(was: JMS)
> [Artemis 2.x upgrade] Broken managed connections after suspending remote broker
> -------------------------------------------------------------------------------
>
> Key: WFWIP-15
> URL: https://issues.jboss.org/browse/WFWIP-15
> Project: WildFly WIP
> Issue Type: Bug
> Components: Artemis
> Reporter: Miroslav Novak
> Assignee: Martyn Taylor
> Priority: Blocker
> Labels: activemq, feature-branch-blocker
>
> Test Scenario:
> * Start 2 servers in cluster with deployed InQueue and OutQueue (jms cluster)
> * Send 50000 messages (10KB) to InQueue
> * Start 2 other servers (mdb nodes) which have configured remote JCA to connect to jms cluster
> * Deploy MDB to each mdb node
> ** MDB consumes messages from InQueue and resend to OutQueue
> * Suspend one of JMS servers (node-3) for 5 minutes when MDBs are processing messages
> * Resume JMS servers and wait for MDBs to process all messages
> Pass Criteria: All messages sent to InQueue well get to OutQueue
> Actual Result:
> There are errors on one of the MDB servers node-4 after node-3 is resumed:
> {code}
> 21:35:05,813 ERROR [org.jboss.qa.hornetq.apps.mdb.MdbWithRemoteOutQueueWithOutQueueLookups] (Thread-168 (ActiveMQ-client-global-threads)) Could not create a session: IJ000453: Unable to get managed connection fo
> r java:/JmsXA: javax.jms.JMSException: Could not create a session: IJ000453: Unable to get managed connection for java:/JmsXA
> at org.apache.activemq.artemis.ra.ActiveMQRASessionFactoryImpl.allocateConnection(ActiveMQRASessionFactoryImpl.java:909)
> at org.apache.activemq.artemis.ra.ActiveMQRASessionFactoryImpl.createSession(ActiveMQRASessionFactoryImpl.java:531)
> at org.jboss.qa.hornetq.apps.mdb.MdbWithRemoteOutQueueWithOutQueueLookups.onMessage(MdbWithRemoteOutQueueWithOutQueueLookups.java:89) [lodhLikemdb1.jar:]
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) [:1.8.0_162]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_162]
> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_162]
> at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
> at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:253) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:332) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:238) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPS
> HOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.messagedriven.MessageDrivenComponentDescription$5$1.processInvocation(MessageDrivenComponentDescription.java:243) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNA
> PSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.messagedriven.MessageDrivenComponentDescription$5$1.processInvocation(MessageDrivenComponentDescription.java:243) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNA
> PSHOT]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619) [wildfly-elytron-1.2.2.Final.jar:1.2.2.Final]
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
> at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)
> at org.jboss.qa.hornetq.apps.mdb.MdbWithRemoteOutQueueWithOutQueueLookups$$$view1.onMessage(Unknown Source) [lodhLikemdb1.jar:]
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) [:1.8.0_162]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_162]
> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_162]
> at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.doInvoke(MessageEndpointInvocationHandler.java:139) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.as.ejb3.inflow.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:73) [wildfly-ejb3-13.0.0.Alpha1-SNAPSHOT.jar:13.0.0.Alpha1-SNAPSHOT]
> at org.jboss.qa.hornetq.apps.mdb.MdbWithRemoteOutQueueWithOutQueueLookups$$$endpoint1.onMessage(Unknown Source) [lodhLikemdb1.jar:]
> at org.apache.activemq.artemis.ra.inflow.ActiveMQMessageHandler.onMessage(ActiveMQMessageHandler.java:302)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1002) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1125) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0.jar:2.5.0]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_162]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_162]
> Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/JmsXA
> at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:690)
> at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:430)
> at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:789)
> at org.apache.activemq.artemis.ra.ActiveMQRASessionFactoryImpl.allocateConnection(ActiveMQRASessionFactoryImpl.java:872)
> ... 69 more
> Caused by: javax.resource.ResourceException: IJ000655: No managed connections available within configured blocking timeout (30000 [ms])
> at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.getConnection(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:570)
> at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getTransactionNewConnection(AbstractPool.java:714)
> at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:613)
> at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:624)
> ... 72 more
> {code}
> later on there are errors:
> {code}
> 21:35:52,119 WARN [org.apache.activemq.artemis.core.client] (Thread-218 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,120 WARN [org.apache.activemq.artemis.core.client] (Thread-208 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,122 WARN [org.apache.activemq.artemis.core.client] (Thread-161 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,123 WARN [org.apache.activemq.artemis.core.client] (Thread-172 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,124 WARN [org.apache.activemq.artemis.core.client] (Thread-167 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,124 WARN [org.apache.activemq.artemis.core.client] (Thread-174 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,124 WARN [org.apache.activemq.artemis.core.client] (Thread-176 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,124 WARN [org.apache.activemq.artemis.core.client] (Thread-196 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:52,124 WARN [org.apache.activemq.artemis.core.client] (Thread-157 (ActiveMQ-client-global-threads)) AMQ212009: resetting session after failure
> 21:35:53,087 WARN [org.apache.activemq.artemis.core.client] (Thread-163 (ActiveMQ-client-global-threads)) AMQ212052: Packet PACKET(SessionXAResponseMessage)[type=55, channelID=10, packetObject=SessionXAResponse
> Message, error=false, message=null, responseCode=0] was answered out of sequence due to a previous server timeout and it's being ignored: java.lang.Exception: trace
> at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:395) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:319) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sessionStop(ActiveMQSessionContext.java:399) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.stop(ClientSessionImpl.java:1019) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.stop(ClientSessionImpl.java:1008) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:912) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:895) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.resetIfNeeded(ClientSessionImpl.java:985) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.ra.inflow.ActiveMQMessageHandler.onMessage(ActiveMQMessageHandler.java:371)
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1002) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1125) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0.jar:2.5.0]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_162]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_162]
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.5.0.jar:2.5.0]
> {code}
> However mdb server never recovers from this situation. It seems that all managed connections are left in broken state.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (WFWIP-40) CLI prompt is rendered at first line without clearing of screen
by Erich Duda (JIRA)
[ https://issues.jboss.org/browse/WFWIP-40?page=com.atlassian.jira.plugin.s... ]
Erich Duda commented on WFWIP-40:
---------------------------------
I hit the issue when I just run the CLI and then I insert {{embed-server}} command.
{code}
./bin/jboss-cli.sh
embed-server
{code}
> CLI prompt is rendered at first line without clearing of screen
> ---------------------------------------------------------------
>
> Key: WFWIP-40
> URL: https://issues.jboss.org/browse/WFWIP-40
> Project: WildFly WIP
> Issue Type: Bug
> Reporter: Erich Duda
> Assignee: Jean-Francois Denise
> Labels: WFCORE-3568
> Attachments: screenshot.png
>
>
> Type following commands.
> {code}
> $ ./bin/jboss-cli.sh
> [disconnected /] embed-server
> {code}
> When I type aforementioned commands the CLI prompt is rendered at first line of terminal, but the screen is not cleared before it. See attached screenshot.
> !screenshot.png|thumbnail!
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (WFWIP-25) [Artemis 2.x Upgrade] Unable to announce backup
by Martin Styk (JIRA)
[ https://issues.jboss.org/browse/WFWIP-25?page=com.atlassian.jira.plugin.s... ]
Martin Styk updated WFWIP-25:
-----------------------------
Component/s: Artemis
(was: JMS)
> [Artemis 2.x Upgrade] Unable to announce backup
> -----------------------------------------------
>
> Key: WFWIP-25
> URL: https://issues.jboss.org/browse/WFWIP-25
> Project: WildFly WIP
> Issue Type: Bug
> Components: Artemis
> Reporter: Erich Duda
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: activemq, feature-branch-blocker
> Attachments: server1.log, server2.log, test.log
>
>
> In shared store HA test scenario it sometimes happens that backup repeatedly prints the error message \[1\]. When the live is killed later, the backup doesn't become active.
> *Users impact:* High availability feature does not work properly.
> *Blocker* priority was set because it is regression against previous release of Wildfly.
> In this particular test Netty connectors/acceptors are used.
> \[1\]
> {code}
> 12:07:26,617 WARN [org.apache.activemq.artemis.core.server] (Thread-1 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@ea5272e)) AMQ222137: Unable to announce backup, retrying: ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ119012: Timed out waiting to receive initial broadcast from cluster]
> at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:761) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:637) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:619) [artemis-core-client-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.core.server.cluster.BackupManager$BackupConnector$1.run(BackupManager.java:246) [artemis-server-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0.jar:2.5.0]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0.jar:2.5.0]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_162]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_162]
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.5.0.jar:2.5.0]
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (WFWIP-17) [Artemis 2.x Upgrade] Replicated HA: Live and Backup do not form cluster and initial synchronization is not triggered
by Martin Styk (JIRA)
[ https://issues.jboss.org/browse/WFWIP-17?page=com.atlassian.jira.plugin.s... ]
Martin Styk updated WFWIP-17:
-----------------------------
Component/s: Artemis
(was: JMS)
> [Artemis 2.x Upgrade] Replicated HA: Live and Backup do not form cluster and initial synchronization is not triggered
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: WFWIP-17
> URL: https://issues.jboss.org/browse/WFWIP-17
> Project: WildFly WIP
> Issue Type: Bug
> Components: Artemis
> Reporter: Erich Duda
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: activemq, feature-branch-blocker
> Attachments: server1-trace.log, server1.log, server2-trace.log, server2.log
>
>
> Across all replicated HA tests I often see an issue that initial synchronization between Live and Backup was not triggered.
> *Scenario*
> # There are two Wildfly servers configured as replicated Live-Backup pair.
> # Live server is killed/shutdown
> *Expectation:* Backup server becomes active.
> *Reality:* Backup server does not become active, because initial synchronization with Live server was not triggered.
> *Users impact:* Replicated HA feature doesn't work properly.
> *Blocker* priority was set because this is regression against previous Wildfly releases.
> *Detail description of the issue*
> In the trace log there is last message from the {{SharedNothingBackupActivation}} which says that it is waiting on cluster connection. It is interested that JGroups subsystem is booting after this message was logged. However I am not sure if this could cause the issue.
> {code}
> 13:42:06,489 DEBUG [org.apache.activemq.artemis.core.client.impl.Topology] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=null) Topology@6a834ff0[owner=ServerLocatorImpl [initialConnectors=[],
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] is sending topology to org.apache.activemq.artemis.core.server.impl.NamedLiveN
> odeLocatorForReplication@38150570
> 13:42:06,489 TRACE [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=null) Waiting on cluster connection
> 13:42:06,502 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000078: Starting JGroups channel ejb
> 13:42:06,503 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000078: Starting JGroups channel ejb
> 13:42:06,504 INFO [org.infinispan.CLUSTER] (MSC service thread 1-6) ISPN000094: Received new cluster view for channel ejb: [node-1|1] (2) [node-1, node-2]
> 13:42:06,506 INFO [org.infinispan.CLUSTER] (MSC service thread 1-7) ISPN000094: Received new cluster view for channel ejb: [node-1|1] (2) [node-1, node-2]
> 13:42:06,506 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000078: Starting JGroups channel ejb
> 13:42:06,506 INFO [org.infinispan.CLUSTER] (MSC service thread 1-8) ISPN000094: Received new cluster view for channel ejb: [node-1|1] (2) [node-1, node-2]
> 13:42:06,510 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000078: Starting JGroups channel ejb
> 13:42:06,510 INFO [org.infinispan.CLUSTER] (MSC service thread 1-3) ISPN000094: Received new cluster view for channel ejb: [node-1|1] (2) [node-1, node-2]
> 13:42:06,516 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000079: Channel ejb local address is node-2, physical addresses are [127.0.0.1:56200]
> 13:42:06,527 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000079: Channel ejb local address is node-2, physical addresses are [127.0.0.1:56200]
> 13:42:06,527 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000079: Channel ejb local address is node-2, physical addresses are [127.0.0.1:56200]
> 13:42:06,529 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000079: Channel ejb local address is node-2, physical addresses are [127.0.0.1:56200]
> 13:42:06,551 INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-7) ISPN000128: Infinispan version: Infinispan 'Gaina' 9.2.0.Final
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (WFWIP-28) [Artemis 2.x upgrade] Unexptected crash of server in SOAK test
by Martin Styk (JIRA)
[ https://issues.jboss.org/browse/WFWIP-28?page=com.atlassian.jira.plugin.s... ]
Martin Styk updated WFWIP-28:
-----------------------------
Component/s: Artemis
(was: JMS)
> [Artemis 2.x upgrade] Unexptected crash of server in SOAK test
> --------------------------------------------------------------
>
> Key: WFWIP-28
> URL: https://issues.jboss.org/browse/WFWIP-28
> Project: WildFly WIP
> Issue Type: Bug
> Components: Artemis
> Reporter: Miroslav Novak
> Assignee: Martyn Taylor
> Priority: Blocker
> Labels: feature-branch-blocker
>
> After ~13 hours there is unexpected crash of one server in SOAK test. There is no error/warning in the logs.
> Test Scenario:
> * Start 2 servers
> * Client sends messages to input queue. Messages then go through:
> * One server to another through MDB reading and sending them from remote container through resource adapter
> * Messages are forwarded from one server to another over JMS bridge and back over Core bridge
> * Messages have JMSReplyTo defined with a temporary queue, that is filled with responses for the client
> * Messages are read from the destination with stateless EJB and sent back to clients
> * Client reads the messages after the pass through all the soak modules.
> Pass Criteria: In the last step receiver consumes all messages sent by producer.
> Actual Result:
> After ~13 hours 1st server suddenly crashes. There is no error/warning in server logs.
> Issue was hit with Artemis 2.5.0 with https://github.com/jmesnil/wildfly/tree/WFLY-9407_upgrade_artemis_2.4.0_w... (commit 51dd8102f103ccb0470a3cfc8713d3f9bdb1b65d)
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (WFWIP-40) CLI prompt is rendered at first line without clearing of screen
by Jean-Francois Denise (JIRA)
[ https://issues.jboss.org/browse/WFWIP-40?page=com.atlassian.jira.plugin.s... ]
Jean-Francois Denise commented on WFWIP-40:
-------------------------------------------
Thanks, I don't really follow what is going on, could you list the steps, scrolling is in the picture?
> CLI prompt is rendered at first line without clearing of screen
> ---------------------------------------------------------------
>
> Key: WFWIP-40
> URL: https://issues.jboss.org/browse/WFWIP-40
> Project: WildFly WIP
> Issue Type: Bug
> Reporter: Erich Duda
> Assignee: Jean-Francois Denise
> Labels: WFCORE-3568
> Attachments: screenshot.png
>
>
> Type following commands.
> {code}
> $ ./bin/jboss-cli.sh
> [disconnected /] embed-server
> {code}
> When I type aforementioned commands the CLI prompt is rendered at first line of terminal, but the screen is not cleared before it. See attached screenshot.
> !screenshot.png|thumbnail!
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (WFWIP-27) [Artemis 2.x Upgrade] Failover does not work with HTTP conncetors/acceptors
by Martin Styk (JIRA)
[ https://issues.jboss.org/browse/WFWIP-27?page=com.atlassian.jira.plugin.s... ]
Martin Styk updated WFWIP-27:
-----------------------------
Component/s: Artemis
(was: JMS)
> [Artemis 2.x Upgrade] Failover does not work with HTTP conncetors/acceptors
> ---------------------------------------------------------------------------
>
> Key: WFWIP-27
> URL: https://issues.jboss.org/browse/WFWIP-27
> Project: WildFly WIP
> Issue Type: Bug
> Components: Artemis
> Reporter: Erich Duda
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: activemq, feature-branch-blocker
> Attachments: clients.log, server1.log, server2.log, standalone-full-ha.xml, standalone-full-ha.xml, test.log
>
>
> Failover does not work with HTTP connectors/acceptors.
> *Scenario:*
> # There are two Wildfly servers configured as Live-Backup pair
> # There is one JMS producer and one JMS receiver which sends/receives messages
> # Live server is several times killed and restarted.
> *Expectation:* Always when the Live server is killed or restarted, clients do failover or failback.
> *Reality*: Sometimes happens that clients don't do failover.
> *Users impact:* One of basics feature of HA, failover, does not work properly.
> *Blocker* priority was set because this is regression against previous releases of Wildfly.
> *Detail description of issue*
> In the trace logs it can be seen that clients send HTTP handshake request to active backup, but the handshake fails. All checks (and logs) say that the backup is active in this time period. I tried to run the test with Netty connectors/acceptors and I didn't see this issue.
> {code:title=Backup log}
> 07:52:47,856 WARN [org.apache.activemq.artemis.core.server] (Thread-2 (ActiveMQ-IO-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@1f01f347)) AMQ222029: Could not locate page transaction 3 995, ignoring message on position PagePositionImpl [pageNr=2, messageNr=55, recordID=-1] on address=jms.queue.testQueue queue=jms.queue.testQueue
> 07:52:47,856 WARN [org.apache.activemq.artemis.core.server] (Thread-2 (ActiveMQ-IO-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@1f01f347)) AMQ222029: Could not locate page transaction 3 995, ignoring message on position PagePositionImpl [pageNr=2, messageNr=56, recordID=-1] on address=jms.queue.testQueue queue=jms.queue.testQueue
> 07:52:47,863 INFO [org.apache.activemq.artemis.core.server] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=null) AMQ221005: Deleting pending large message as it was not completed: Pair[a=2147485068, b=2147485067]
> 07:52:47,863 INFO [org.apache.activemq.artemis.core.server] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=null) AMQ221005: Deleting pending large message as it was not completed: Pair[a=2147485076, b=2147485075]
> 07:52:47,864 INFO [org.apache.activemq.artemis.core.server] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=null) AMQ221005: Deleting pending large message as it was not completed: Pair[a=2147484450, b=2147484449]
> 07:52:47,864 INFO [org.apache.activemq.artemis.core.server] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=null) AMQ221005: Deleting pending large message as it was not completed: Pair[a=2147485072, b=2147485071]
> 07:52:47,864 INFO [org.apache.activemq.artemis.core.server] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=null) AMQ221005: Deleting pending large message as it was not completed: Pair[a=2147484454, b=2147484453]
> 07:52:47,866 WARN [org.apache.activemq.artemis.core.client] (activemq-discovery-group-thread-dg-group1) AMQ212034: There are more than one servers on the network broadcasting the same node id. You will see this message exactly once (per node) if a node is restarted, in which case it can be safely ignored. But if it is logged continuously it means you really do have more than one node on the same network active concurrently with the same node id. This could occur if you have a backup node active at the same time as its live node. nodeID=0dee81c9-2d9d-11e8-ba3f-cc3d825f79a4
> 07:52:47,867 INFO [org.apache.activemq.artemis.core.server] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=null) AMQ221007: Server is now live
> {code}
> {code:title=Clients log}
> 07:52:49,274 Thread-80 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:726] Remote destination: localhost/127.0.0.1:9080
> 07:52:49,274 Thread-79 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=8080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true, activemqServerName=default, httpUpgradeEndpoint=acceptor] failed
> 07:52:49,275 Thread-79 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1081] Trying backup config = TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=9080&host=localhost
> 07:52:49,275 Thread-79 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:447] Connector NettyConnector [host=localhost, port=9080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true, activemqServerName=default, httpUpgradeEndpoint=acceptor] using native epoll
> 07:52:49,275 Thread-80 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:797] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
> GET HTTP/1.1
> host: localhost
> upgrade: activemq-remoting
> connection: upgrade
> activemqServerName: default
> httpUpgradeEndpoint: acceptor
> Sec-ActiveMQRemoting-Key: QWV3bCfgh75NjWH3pZV5Ew==
> 07:52:49,275 Thread-79 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:601] AMQ211002: Started EPOLL Netty Connector version 4.1.16.Final to localhost:9080
> 07:52:49,275 Thread-79 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:726] Remote destination: localhost/127.0.0.1:9080
> 07:52:49,276 Thread-79 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:797] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
> GET HTTP/1.1
> host: localhost
> upgrade: activemq-remoting
> connection: upgrade
> activemqServerName: default
> httpUpgradeEndpoint: acceptor
> Sec-ActiveMQRemoting-Key: P9xBwRk1eZP5QjDWjqYuIg==
> 07:52:49,276 Thread-2 (ActiveMQ-client-netty-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$HttpUpgradeHandler:876] Received msg=DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)
> HTTP/1.1 200 OK
> Connection: keep-alive
> Last-Modified: Thu, 22 Mar 2018 06:47:03 GMT
> Content-Length: 2426
> Content-Type: text/html
> Accept-Ranges: bytes
> Date: Thu, 22 Mar 2018 06:52:49 GMT
> 07:52:49,276 Thread-2 (ActiveMQ-client-netty-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$HttpUpgradeHandler:903] AMQ214023: HTTP Handshake failed, received DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)
> HTTP/1.1 200 OK
> Connection: keep-alive
> Last-Modified: Thu, 22 Mar 2018 06:47:03 GMT
> Content-Length: 2426
> Content-Type: text/html
> Accept-Ranges: bytes
> Date: Thu, 22 Mar 2018 06:52:49 GMT
> 07:52:49,276 Thread-2 (ActiveMQ-client-netty-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$HttpUpgradeHandler:876] Received msg=DefaultHttpContent(data: PooledSlicedByteBuf(ridx: 0, widx: 829, cap: 829/829, unwrapped: PooledUnsafeDirectByteBuf(ridx: 1024, widx: 1024, cap: 1024)), decoderResult: success)
> 07:52:49,276 Thread-2 (ActiveMQ-client-netty-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$HttpUpgradeHandler:903] AMQ214023: HTTP Handshake failed, received DefaultHttpContent(data: PooledSlicedByteBuf(ridx: 0, widx: 829, cap: 829/829, unwrapped: PooledUnsafeDirectByteBuf(ridx: 1024, widx: 1024, cap: 1024)), decoderResult: success)
> 07:52:49,277 Thread-80 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1040] Connector towards NettyConnector [host=localhost, port=9080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true, activemqServerName=default, httpUpgradeEndpoint=acceptor] failed
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (WFWIP-22) [Artemis 2.x upgrade] MDB connected to remote server does not get activated when using discovery group with JGroups
by Martin Styk (JIRA)
[ https://issues.jboss.org/browse/WFWIP-22?page=com.atlassian.jira.plugin.s... ]
Martin Styk updated WFWIP-22:
-----------------------------
Component/s: Artemis
> [Artemis 2.x upgrade] MDB connected to remote server does not get activated when using discovery group with JGroups
> --------------------------------------------------------------------------------------------------------------------
>
> Key: WFWIP-22
> URL: https://issues.jboss.org/browse/WFWIP-22
> Project: WildFly WIP
> Issue Type: Bug
> Components: Artemis, JMS
> Reporter: Miroslav Novak
> Assignee: Martyn Taylor
> Priority: Blocker
> Labels: activemq, feature-branch-blocker
> Attachments: log
>
>
> Sometimes happens that MDB does not start to consume messages from remote server. There are not warning or error messages in log.
> Test Scenario:
> * Start server with deployed InQueue and OutQueue
> * Start 2nd server with configured pooled-connection-factory to connect to 1st server
> * Deploy MDB to 2nd server - mdb is consuming messages from InQueue and sending to OutQueue
> * Start sending messages to InQueue to 1st server
> * Let MDB processing messages from InQueue
> * Start consuming messages from OutQueue from 1st server
> Pass Criteria: There is the same number of sent and received messages
> Actual Result: Sometimes happens that Mdb does not start to consume messages.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years