12:09 p.m.
Corey Puffalt created WFLY-2443:
-----------------------------------
Summary: JMS service shutsdown and doesn't recover after disk fills up.
Key: WFLY-2443
URL: https://issues.jboss.org/browse/WFLY-2443
Project: WildFly
Issue Type: Bug
Security Level: Public (Everyone can see)
Components: JMS
Environment: RHEL 5.2
JDK 1.7.0_07
Reporter: Corey Puffalt
Assignee: Jeff Mesnil
Priority: Critical
In AS 7.1.2 we had a disk fill up and we see warnings (not an error??):
{code}
06:07:40,641 WARNING [org.hornetq.core.server.impl.HornetQServerImpl] (Thread-1973
(HornetQ-remoting-threads-HornetQServerImpl::serverUUID=d7fd8a09-5f3c-11e2-a072-49b352663c05-1554718086-1713338353))
Critical IO Error, shutting down the server. code=6, message=No space left on device
06:07:40,892 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
(Thread-15478 (HornetQ-client-global-threads-251718996)) Connection failure has been
detected: The connection was disconnected because of server shutdown [code=4]
06:07:40,893 WARN [org.hornetq.jms.server.recovery.RecoveryDiscovery] (Thread-15478
(HornetQ-client-global-threads-251718996)) being disconnected for server shutdown:
HornetQException[errorCode=4 message=The connection was disconnected because of server
shutdown]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1468)
[hornetq-core-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
[rt.jar:1.7.0_07]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
[rt.jar:1.7.0_07]
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_07]
{code}
After the disk space issue was resolved JMS services didn't recover (though we
didn't notice this for awhile) so we didn't restart JBoss as we didn't realize
there was an issue. In checking the logs after the fact we see periodic warnings (again,
it seems strange this is just a warning instead of an error):
{code}
08:28:51,564 WARN [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Periodic
Recovery) Can't connect to XARecoveryConfig [transportConfiguration =
[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0],
discoveryConfiguration = null, username=null, password=null] on auto-generated resource
recovery: HornetQException[errorCode=2 message=Cannot connect to server(s). Tried with all
available servers.]
at
org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:774)
[hornetq-core-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connect(HornetQXAResourceWrapper.java:347)
[hornetq-jms-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.getDelegate(HornetQXAResourceWrapper.java:262)
[hornetq-jms-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.recover(HornetQXAResourceWrapper.java:76)
[hornetq-jms-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]
at
com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecovery(XARecoveryModule.java:503)
[jbossjts-4.16.4.Final.jar:]
at
com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecoveryForRecoveryHelpers(XARecoveryModule.java:471)
[jbossjts-4.16.4.Final.jar:]
at
com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:385)
[jbossjts-4.16.4.Final.jar:]
at
com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:166)
[jbossjts-4.16.4.Final.jar:]
at
com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789)
[jbossjts-4.16.4.Final.jar:]
at
com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371)
[jbossjts-4.16.4.Final.jar:]
08:28:51,646 WARN [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Periodic
Recovery) Can't connect to any hornetq server on recovery [XARecoveryConfig
[transportConfiguration =
[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0],
discoveryConfiguration = null, username=null, password=null]]
08:28:51,647 WARN [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016027: Local
XARecoveryModule.xaRecovery got XA exception XAException.XAER_RMERR:
javax.transaction.xa.XAException: Error trying to connect to any providers for xa
recovery
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.getDelegate(HornetQXAResourceWrapper.java:286)
[hornetq-jms-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.recover(HornetQXAResourceWrapper.java:76)
[hornetq-jms-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]
at
com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecovery(XARecoveryModule.java:503)
[jbossjts-4.16.4.Final.jar:]
at
com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecoveryForRecoveryHelpers(XARecoveryModule.java:471)
[jbossjts-4.16.4.Final.jar:]
at
com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:385)
[jbossjts-4.16.4.Final.jar:]
at
com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:166)
[jbossjts-4.16.4.Final.jar:]
at
com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789)
[jbossjts-4.16.4.Final.jar:]
at
com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371)
[jbossjts-4.16.4.Final.jar:]
Caused by: HornetQException[errorCode=2 message=null]
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connect(HornetQXAResourceWrapper.java:396)
[hornetq-jms-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.getDelegate(HornetQXAResourceWrapper.java:262)
[hornetq-jms-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]
... 7 more
{code}
It seems if the whole server process isn't going to shutdown when the disk fills up
then services like JMS should just reject messages if the disk fills up rather than
shutting down. That way services nicely recover as soon as there's disk space again.
The way it is currently, it's easy to miss that JBoss is only partially inoperable.
Thanks,
Corey
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira