[jboss-jira] [JBoss JIRA] (WFLY-2443) JMS service shutsdown and doesn't recover after disk fills up.
Miroslav Novak (Jira)
issues at jboss.org
Wed Nov 21 07:47:00 EST 2018
[ https://issues.jboss.org/browse/WFLY-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13664534#comment-13664534 ]
Miroslav Novak commented on WFLY-2443:
--------------------------------------
The basic strategy for HornetQ/Artemis is to shutdown when anything "bad" happens with underlying file system. Issue must be fixed and Wildfly server restarted to recover.
However for this particular case with disk full issue there was added feature to block all message sends to Artemis until disk is freed. For more information see WFLY-10326.
I'll set this jira as resolved once WFLY-10326 is merged into Wildfly master.
> JMS service shutsdown and doesn't recover after disk fills up.
> --------------------------------------------------------------
>
> Key: WFLY-2443
> URL: https://issues.jboss.org/browse/WFLY-2443
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Environment: RHEL 5.2
> JDK 1.7.0_07
> Reporter: Corey Puffalt
> Assignee: Jeff Mesnil
> Priority: Major
>
> In AS 7.1.2 we had a disk fill up and we see warnings (not an error??):
> {code}
> 06:07:40,641 WARNING [org.hornetq.core.server.impl.HornetQServerImpl] (Thread-1973 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=d7fd8a09-5f3c-11e2-a072-49b352663c05-1554718086-1713338353)) Critical IO Error, shutting down the server. code=6, message=No space left on device
> 06:07:40,892 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (Thread-15478 (HornetQ-client-global-threads-251718996)) Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
> 06:07:40,893 WARN [org.hornetq.jms.server.recovery.RecoveryDiscovery] (Thread-15478 (HornetQ-client-global-threads-251718996)) being disconnected for server shutdown: HornetQException[errorCode=4 message=The connection was disconnected because of server shutdown]
> at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1468) [hornetq-core-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]
> at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [hornetq-core-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_07]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_07]
> at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_07]
> {code}
> After the disk space issue was resolved JMS services didn't recover (though we didn't notice this for awhile) so we didn't restart JBoss as we didn't realize there was an issue. In checking the logs after the fact we see periodic warnings (again, it seems strange this is just a warning instead of an error):
> {code}
> 08:28:51,564 WARN [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Periodic Recovery) Can't connect to XARecoveryConfig [transportConfiguration = [org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0], discoveryConfiguration = null, username=null, password=null] on auto-generated resource recovery: HornetQException[errorCode=2 message=Cannot connect to server(s). Tried with all available servers.]
> at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:774) [hornetq-core-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]
> at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connect(HornetQXAResourceWrapper.java:347) [hornetq-jms-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]
> at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.getDelegate(HornetQXAResourceWrapper.java:262) [hornetq-jms-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]
> at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.recover(HornetQXAResourceWrapper.java:76) [hornetq-jms-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]
> at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecovery(XARecoveryModule.java:503) [jbossjts-4.16.4.Final.jar:]
> at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecoveryForRecoveryHelpers(XARecoveryModule.java:471) [jbossjts-4.16.4.Final.jar:]
> at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:385) [jbossjts-4.16.4.Final.jar:]
> at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:166) [jbossjts-4.16.4.Final.jar:]
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) [jbossjts-4.16.4.Final.jar:]
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [jbossjts-4.16.4.Final.jar:]
> 08:28:51,646 WARN [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Periodic Recovery) Can't connect to any hornetq server on recovery [XARecoveryConfig [transportConfiguration = [org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0], discoveryConfiguration = null, username=null, password=null]]
> 08:28:51,647 WARN [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016027: Local XARecoveryModule.xaRecovery got XA exception XAException.XAER_RMERR: javax.transaction.xa.XAException: Error trying to connect to any providers for xa recovery
> at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.getDelegate(HornetQXAResourceWrapper.java:286) [hornetq-jms-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]
> at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.recover(HornetQXAResourceWrapper.java:76) [hornetq-jms-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]
> at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecovery(XARecoveryModule.java:503) [jbossjts-4.16.4.Final.jar:]
> at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecoveryForRecoveryHelpers(XARecoveryModule.java:471) [jbossjts-4.16.4.Final.jar:]
> at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:385) [jbossjts-4.16.4.Final.jar:]
> at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:166) [jbossjts-4.16.4.Final.jar:]
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) [jbossjts-4.16.4.Final.jar:]
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [jbossjts-4.16.4.Final.jar:]
> Caused by: HornetQException[errorCode=2 message=null]
> at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connect(HornetQXAResourceWrapper.java:396) [hornetq-jms-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]
> at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.getDelegate(HornetQXAResourceWrapper.java:262) [hornetq-jms-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]
> ... 7 more
> {code}
> It seems if the whole server process isn't going to shutdown when the disk fills up then services like JMS should just reject messages if the disk fills up rather than shutting down. That way services nicely recover as soon as there's disk space again. The way it is currently, it's easy to miss that JBoss is only partially inoperable.
> Thanks,
> Corey
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
More information about the jboss-jira
mailing list