[jboss-jira] [JBoss JIRA] (AS7-3408) Classloading issue with HornetQLoggerFactory - possilbe problem with failback

[Miroslav Novak (JIRA)]

Miroslav Novak created AS7-3408:
-----------------------------------

             Summary: Classloading issue with HornetQLoggerFactory - possilbe problem with failback
                 Key: AS7-3408
                 URL: https://issues.jboss.org/browse/AS7-3408
             Project: Application Server 7
          Issue Type: Bug
          Components: JMS
    Affects Versions: 7.1.0.CR1
            Reporter: Miroslav Novak
            Assignee: Andy Taylor
            Priority: Critical
             Fix For: 7.1.0.Final
         Attachments: console-log-backup-server.txt, console-log-live-server.txt, reproducer.zip

Test scenario:
1. Start two servers AS7/EAP6 servers - live and its backup in dedicated topology - each on different machine
2. Kill live server using "kill -9 ..."
3. Start live server again

In step 3. there are unexpected messages in console log of live and backup server.

>From backup server:
{code}
12:07:26,165 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (Thread-78) HornetQ Server version 2.2.7.Final (HQ_2_2_7_FINAL_AS7, 121) [17700d86-45b2-11e1-a575-d48564b8e1e7] stopped
12:07:26,165 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (Thread-78) unable to restart server, please kill and restart manually: java.lang.IllegalArgumentException: Could not find class org.jboss.as.messaging.HornetQLoggerFactory
	at org.hornetq.utils.ClassloadingUtil$1.run(ClassloadingUtil.java:42) [hornetq-core-2.2.7.Final.jar:]
	at java.security.AccessController.doPrivileged(Native Method) [:1.6.0_22]
	at org.hornetq.utils.ClassloadingUtil.safeInitNewInstance(ClassloadingUtil.java:16) [hornetq-core-2.2.7.Final.jar:]
	at org.hornetq.core.server.impl.HornetQServerImpl.instantiateInstance(HornetQServerImpl.java:1868) [hornetq-core-2.2.7.Final.jar:]
	at org.hornetq.core.server.impl.HornetQServerImpl.initialiseLogging(HornetQServerImpl.java:1301) [hornetq-core-2.2.7.Final.jar:]
	at org.hornetq.core.server.impl.HornetQServerImpl.start(HornetQServerImpl.java:541) [hornetq-core-2.2.7.Final.jar:]
	at org.hornetq.core.server.impl.HornetQServerImpl$SharedStoreBackupActivation$1FailbackChecker$1.run(HornetQServerImpl.java:430) [hornetq-core-2.2.7.Final.jar:]
	at java.lang.Thread.run(Thread.java:679) [:1.6.0_22]
{code}

>From live server:
{code}
12:07:51,993 INFO  [org.jboss.as.messaging] (MSC service thread 1-3) JBAS011601: Bound messaging object to jndi name java:/topic/test
12:07:52,007 INFO  [org.jboss.as] (Controller Boot Thread) JBoss EAP 6.0.0.Alpha2 (AS 7.1.0.CR1-redhat-1) started in 31092ms - Started 155 of 263 services (103 services are passive or on-demand)
12:07:53,292 DEBUG [org.hornetq.core.client.impl.ClientSessionFactoryImpl] (Thread-1 (group:HornetQ-client-global-threads-1954829789)) Trying reconnection attempt 1
12:07:53,292 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (Thread-1 (group:HornetQ-client-global-threads-1954829789)) Started Netty Connector version 3.2.3.Final-r${buildNumber}
12:07:53,292 DEBUG [org.hornetq.core.client.impl.ClientSessionFactoryImpl] (Thread-1 (group:HornetQ-client-global-threads-1954829789)) Trying to connect at the main server using connector :org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5446&host=192-168-10-4
12:07:53,293 DEBUG [org.hornetq.core.client.impl.ClientSessionFactoryImpl] (Thread-1 (group:HornetQ-client-global-threads-1954829789)) Main server is not up. Hopefully there's a backup configured now!
12:07:55,293 DEBUG [org.hornetq.core.client.impl.ClientSessionFactoryImpl] (Thread-1 (group:HornetQ-client-global-threads-1954829789)) Trying reconnection attempt 2
12:07:55,293 DEBUG [org.hornetq.core.remoting.impl.netty.NettyConnector] (Thread-1 (group:HornetQ-client-global-threads-1954829789)) Started Netty Connector version 3.2.3.Final-r${buildNumber}
12:07:55,293 DEBUG [org.hornetq.core.client.impl.ClientSessionFactoryImpl] (Thread-1 (group:HornetQ-client-global-threads-1954829789)) Trying to connect at the main server using connector :org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5446&host=192-168-10-4
12:07:55,294 DEBUG [org.hornetq.core.client.impl.ClientSessionFactoryImpl] (Thread-1 (group:HornetQ-client-global-threads-1954829789)) Main server is not up. Hopefully there's a backup
{code}

I'm not fully sure but it looks like that backup server did not manage to get to "waiting for live to fail" state and live server is not able to detect it.

There is set "DEBUG" level logging for "org.hornetq" in attached logs.

I've attached reproducer.zip - steps to use:
1. Download and unzip "reproducer.zip"
2. Prepare live and backup server - "sh prepare.sh"
3. Start live - "sh start-server1.sh server1_hostname"
4. Start backup - "sh start-server2.sh server2_hostname"
5. Kill live server using "kill -9 server1_process_id"
6. Start live server again - "sh start-server1.sh server1_hostname"

Note:
In reproducer.zip are configuration files standalone-ha-A.xml, standalone-ha-B.xml (A for live, B for backup).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira