[jboss-jira] [JBoss JIRA] (WFWIP-89) [Artemis 2.x Upgrade] Clients do not failover to backup when force-failover operation is used with JDBC store

[Francesco Nigro (JIRA)]

    [ https://issues.jboss.org/browse/WFWIP-89?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13616205#comment-13616205 ] 

Francesco Nigro edited comment on WFWIP-89 at 8/8/18 5:27 AM:
--------------------------------------------------------------

I have noticed these logs on the node 2 log:
{code:java}
11:54:36,296 ERROR [org.apache.activemq.artemis.core.server] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=00d22eb4-9712-11e8-bc7a-fa163e5855f4) AMQ224000: Failure in initialisation: java.lang.IllegalStateException: java.sql.SQLException: IJ031013: Interrupted attempting lock: org.jboss.jca.adapters.jdbc.local.LocalManagedConnection at 6f9f8b97
	at org.apache.activemq.artemis.core.server.impl.jdbc.JdbcSharedStateManager.readState(JdbcSharedStateManager.java:338) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
	at org.apache.activemq.artemis.core.server.impl.jdbc.JdbcNodeManager.readSharedState(JdbcNodeManager.java:523) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
	at org.apache.activemq.artemis.core.server.impl.jdbc.JdbcNodeManager.awaitLiveNode(JdbcNodeManager.java:380) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
	at org.apache.activemq.artemis.core.server.impl.SharedStoreBackupActivation.run(SharedStoreBackupActivation.java:77) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:3236) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
Caused by: java.sql.SQLException: IJ031013: Interrupted attempting lock: org.jboss.jca.adapters.jdbc.local.LocalManagedConnection at 6f9f8b97
	at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.tryLock(BaseWrapperManagedConnection.java:402)
	at org.jboss.jca.adapters.jdbc.WrappedConnection.lock(WrappedConnection.java:161)
	at org.jboss.jca.adapters.jdbc.WrappedConnection.setTransactionIsolation(WrappedConnection.java:1007)
	at org.apache.activemq.artemis.core.server.impl.jdbc.JdbcSharedStateManager.readState(JdbcSharedStateManager.java:316) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
	... 4 more
{code}
It shows that the backup node isn't able to come up because isn't able to read the state of the live node due to an issue of the EAP SQL connection pools. 
It isn't a broker issue and it looks like a timing issue with the reaper thread used by the connection pool. 
I believe that is intermittent, given that is dependent by timing.
[~mnovak] I think that it isn't a regression and not caused by Artemis.


was (Author: fnigro):
I have noticed these logs on the node 2 log:
{code:java}
11:54:36,296 ERROR [org.apache.activemq.artemis.core.server] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=00d22eb4-9712-11e8-bc7a-fa163e5855f4) AMQ224000: Failure in initialisation: java.lang.IllegalStateException: java.sql.SQLException: IJ031013: Interrupted attempting lock: org.jboss.jca.adapters.jdbc.local.LocalManagedConnection at 6f9f8b97
	at org.apache.activemq.artemis.core.server.impl.jdbc.JdbcSharedStateManager.readState(JdbcSharedStateManager.java:338) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
	at org.apache.activemq.artemis.core.server.impl.jdbc.JdbcNodeManager.readSharedState(JdbcNodeManager.java:523) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
	at org.apache.activemq.artemis.core.server.impl.jdbc.JdbcNodeManager.awaitLiveNode(JdbcNodeManager.java:380) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
	at org.apache.activemq.artemis.core.server.impl.SharedStoreBackupActivation.run(SharedStoreBackupActivation.java:77) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:3236) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
Caused by: java.sql.SQLException: IJ031013: Interrupted attempting lock: org.jboss.jca.adapters.jdbc.local.LocalManagedConnection at 6f9f8b97
	at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.tryLock(BaseWrapperManagedConnection.java:402)
	at org.jboss.jca.adapters.jdbc.WrappedConnection.lock(WrappedConnection.java:161)
	at org.jboss.jca.adapters.jdbc.WrappedConnection.setTransactionIsolation(WrappedConnection.java:1007)
	at org.apache.activemq.artemis.core.server.impl.jdbc.JdbcSharedStateManager.readState(JdbcSharedStateManager.java:316) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
	... 4 more
{code}
It shows that the backup node isn't able to come up because isn't able to read the state of the live node due to an issue of the EAP SQL connection pools. I think that's not a broker issue and it looks like a timing issue with the reaper thread used by the connection pool.
I believe that is intermittent, given that is dependent by timing.


> [Artemis 2.x Upgrade] Clients do not failover to backup when force-failover operation is used with JDBC store
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: WFWIP-89
>                 URL: https://issues.jboss.org/browse/WFWIP-89
>             Project: WildFly WIP
>          Issue Type: Bug
>          Components: Artemis
>            Reporter: Martin Styk
>            Assignee: Martyn Taylor
>         Attachments: org.jboss.qa.artemis.test.failover.DedicatedFailoverTestCase20.testForceFailoverOperation.zip
>
>
> Dedicated topology (1live, 1 backup) with JDBC store (oracle12c).
> Force failover operation is called on live node. Live artemis server stops, however backup doesn't start.
> This issue is intermittent.
> Attaching logs.



--
This message was sent by Atlassian JIRA
(v7.5.0#75005)