11:24 a.m.
Ove Ranheim created AS7-3919:
--------------------------------
Summary: HornetQ causing deadlock when max-size-bytes reaches its limit
Key: AS7-3919
URL: https://issues.jboss.org/browse/AS7-3919
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.0.Final
Reporter: Ove Ranheim
Assignee: Andy Taylor
Priority: Critical
Fix For: 7.1.1.Final
I have journal-file-size set to 10 Mb and max-size-bytes set 10 Mb, with a address policy
of BLOCK. When my sendMailQueue reaches its limit a deadlock occurs. This is further
emphasized in the JBossAS Forum reference.
Here's a snippet from jstack after HornetQ crashes.
{noformat}
Found one Java-level deadlock:
=============================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x0000000041df8640 (object 0x00000000ea17d6e0, a
java.lang.Object),
which is held by "Thread-873 (HornetQ-client-global-threads-1151364488)"
"Thread-873 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x000000004107c8b8 (object 0x00000000ea17d6d0, a
java.lang.Object),
which is held by "Thread-878 (HornetQ-client-global-threads-1151364488)"
Java stack information for the threads listed above:
===================================================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:461)
- waiting to lock <0x00000000ea17d6e0> (a java.lang.Object)
- locked <0x00000000ea17d6d0> (a java.lang.Object)
at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1294)
at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1238)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:392)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000f0222060> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"Thread-873 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:458)
- waiting to lock <0x00000000ea17d6d0> (a java.lang.Object)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:391)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000ea17d6e0> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Found 1 deadlock.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:36 a.m.
[
https://issues.jboss.org/browse/AS7-3919?page=com.atlassian.jira.plugin.s...
]
Ove Ranheim commented on AS7-3919:
----------------------------------
My messaging configuration is based on standard-full with an additional queue
configuration:
{noformat}
<jms-queue name="sendMailQueue">
<entry name="queue/sendMail"/>
<entry name="java:jboss/exported/jms/queue/sendMail"/>
</jms-queue>
{noformat}
In case this is helpful, here is the console log from our app, just upon the queue address
space reaches its limit.
{noformat}
09:55:48,870 INFO [com.parts.contract.process.bean.JbpmContractProcessExecutor]
(ajp--127.0.0.1-8009-7) Constructing Contract Process Process Executor, session: 551
09:57:24,552 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
(hornetq-failure-check-thread) Connection failure has been detected: Did not receive data
from invm:0. It is likely the client has exited or crashed without closing its connection,
or the network between the server and client has failed. You also might have configured
connection-ttl and client-failure-check-period incorrectly. Please check user manual for
more information. The connection will now be closed. [code=3]
09:57:24,558 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Client connection failed, clearing up resources for session
a1e94ae0-60c5-11e1-be13-0010189e36c0
09:57:24,614 INFO [com.parts.contract.bean.PerformanceContractPrincipals]
(ajp--127.0.0.1-8009-6) Performance Contract Principals POST CONSTRUCT
09:57:24,712 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Cleared up resources for session
a1e94ae0-60c5-11e1-be13-0010189e36c0
09:57:25,298 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
(hornetq-failure-check-thread) Connection failure has been detected: Did not receive data
from invm:0. It is likely the client has exited or crashed without closing its connection,
or the network between the server and client has failed. You also might have configured
connection-ttl and client-failure-check-period incorrectly. Please check user manual for
more information. The connection will now be closed. [code=3]
09:57:25,304 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Client connection failed, clearing up resources for session
7c7a2008-6103-11e1-be13-0010189e36c0
09:57:25,307 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Cleared up resources for session
7c7a2008-6103-11e1-be13-0010189e36c0
09:57:25,308 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
(hornetq-failure-check-thread) Connection failure has been detected: Did not receive data
from invm:0. It is likely the client has exited or crashed without closing its connection,
or the network between the server and client has failed. You also might have configured
connection-ttl and client-failure-check-period incorrectly. Please check user manual for
more information. The connection will now be closed. [code=3]
09:57:25,314 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Client connection failed, clearing up resources for session
53269e60-60f1-11e1-be13-0010189e36c0
09:57:25,316 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Cleared up resources for session
53269e60-60f1-11e1-be13-0010189e36c0
09:57:25,318 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
(hornetq-failure-check-thread) Connection failure has been detected: Did not receive data
from invm:0. It is likely the client has exited or crashed without closing its connection,
or the network between the server and client has failed. You also might have configured
connection-ttl and client-failure-check-period incorrectly. Please check user manual for
more information. The connection will now be closed. [code=3]
09:57:25,323 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Client connection failed, clearing up resources for session
5a5e91e2-6115-11e1-be13-0010189e36c0
09:57:25,325 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Cleared up resources for session
5a5e91e2-6115-11e1-be13-0010189e36c0
09:57:25,327 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
(hornetq-failure-check-thread) Connection failure has been detected: Did not receive data
from invm:0. It is likely the client has exited or crashed without closing its connection,
or the network between the server and client has failed. You also might have configured
connection-ttl and client-failure-check-period incorrectly. Please check user manual for
more information. The connection will now be closed. [code=3]
09:57:25,332 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Client connection failed, clearing up resources for session
ab8c84bc-6101-11e1-be13-0010189e36c0
09:57:25,334 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Cleared up resources for session
ab8c84bc-6101-11e1-be13-0010189e36c0
09:57:25,335 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
(hornetq-failure-check-thread) Connection failure has been detected: Did not receive data
from invm:0. It is likely the client has exited or crashed without closing its connection,
or the network between the server and client has failed. You also might have configured
connection-ttl and client-failure-check-period incorrectly. Please check user manual for
more information. The connection will now be closed. [code=3]
09:57:25,341 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Client connection failed, clearing up resources for session
8d726e7e-6120-11e1-be13-0010189e36c0
09:57:25,343 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Cleared up resources for session
8d726e7e-6120-11e1-be13-0010189e36c0
09:57:25,344 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
(hornetq-failure-check-thread) Connection failure has been detected: Did not receive data
from invm:0. It is likely the client has exited or crashed without closing its connection,
or the network between the server and client has failed. You also might have configured
connection-ttl and client-failure-check-period incorrectly. Please check user manual for
more information. The connection will now be closed. [code=3]
09:57:25,349 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Client connection failed, clearing up resources for session
241439ac-60f3-11e1-be13-0010189e36c0
09:57:25,352 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Cleared up resources for session
241439ac-60f3-11e1-be13-0010189e36c0
09:57:25,353 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
(hornetq-failure-check-thread) Connection failure has been detected: Did not receive data
from invm:0. It is likely the client has exited or crashed without closing its connection,
or the network between the server and client has failed. You also might have configured
connection-ttl and client-failure-check-period incorrectly. Please check user manual for
more information. The connection will now be closed. [code=3]
09:57:25,358 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Client connection failed, clearing up resources for session
3bf091f4-6113-11e1-be13-0010189e36c0
09:57:25,360 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Cleared up resources for session
3bf091f4-6113-11e1-be13-0010189e36c0
09:57:25,706 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
(hornetq-failure-check-thread) Connection failure has been detected: Did not receive data
from invm:0. It is likely the client has exited or crashed without closing its connection,
or the network between the server and client has failed. You also might have configured
connection-ttl and client-failure-check-period incorrectly. Please check user manual for
more information. The connection will now be closed. [code=3]
09:57:25,771 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Client connection failed, clearing up resources for session
0ce188a0-6115-11e1-be13-0010189e36c0
09:57:25,773 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Cleared up resources for session
0ce188a0-6115-11e1-be13-0010189e36c0
09:57:25,775 INFO [com.parts.contract.bean.PerformanceContractPrincipals]
(ajp--127.0.0.1-8009-8) Performance Contract Principals POST CONSTRUCT
09:57:25,775 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
(hornetq-failure-check-thread) Connection failure has been detected: Did not receive data
from invm:0. It is likely the client has exited or crashed without closing its connection,
or the network between the server and client has failed. You also might have configured
connection-ttl and client-failure-check-period incorrectly. Please check user manual for
more information. The connection will now be closed. [code=3]
09:57:25,840 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Client connection failed, clearing up resources for session
1a7d0cbe-60b7-11e1-be13-0010189e36c0
09:57:25,843 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Cleared up resources for session
1a7d0cbe-60b7-11e1-be13-0010189e36c0
09:57:25,845 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
(hornetq-failure-check-thread) Connection failure has been detected: Did not receive data
from invm:0. It is likely the client has exited or crashed without closing its connection,
or the network between the server and client has failed. You also might have configured
connection-ttl and client-failure-check-period incorrectly. Please check user manual for
more information. The connection will now be closed. [code=3]
09:57:25,846 INFO [org.jboss.seam.jms.MessageManagerImpl] (ajp--127.0.0.1-8009-7) Routing
destination HornetQQueue[sendMailQueue] with message HornetQMessage[null]:PERSISTENT
09:57:25,850 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Client connection failed, clearing up resources for session
ddcf4afc-6116-11e1-be13-0010189e36c0
09:57:25,855 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Cleared up resources for session
ddcf4afc-6116-11e1-be13-0010189e36c0
09:57:25,857 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
(hornetq-failure-check-thread) Connection failure has been detected: Did not receive data
from invm:0. It is likely the client has exited or crashed without closing its connection,
or the network between the server and client has failed. You also might have configured
connection-ttl and client-failure-check-period incorrectly. Please check user manual for
more information. The connection will now be closed. [code=3]
09:57:25,863 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Client connection failed, clearing up resources for session
16a4a23e-610e-11e1-be13-0010189e36c0
09:57:25,865 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Cleared up resources for session
16a4a23e-610e-11e1-be13-0010189e36c0
09:57:25,867 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
(hornetq-failure-check-thread) Connection failure has been detected: Did not receive data
from invm:0. It is likely the client has exited or crashed without closing its connection,
or the network between the server and client has failed. You also might have configured
connection-ttl and client-failure-check-period incorrectly. Please check user manual for
more information. The connection will now be closed. [code=3]
09:57:25,884 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Client connection failed, clearing up resources for session
d1b6514e-6106-11e1-be13-0010189e36c0
09:57:25,886 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Cleared up resources for session
d1b6514e-6106-11e1-be13-0010189e36c0
09:57:25,888 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
(hornetq-failure-check-thread) Connection failure has been detected: Did not receive data
from invm:0. It is likely the client has exited or crashed without closing its connection,
or the network between the server and client has failed. You also might have configured
connection-ttl and client-failure-check-period incorrectly. Please check user manual for
more information. The connection will now be closed. [code=3]
09:57:25,891 INFO [com.parts.contract.bean.PerformanceContractInbox]
(ajp--127.0.0.1-8009-8) Contract Inbox loaded for The Beatles
09:57:25,894 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Client connection failed, clearing up resources for session
78a22446-60b3-11e1-be13-0010189e36c0
09:57:25,897 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Cleared up resources for session
78a22446-60b3-11e1-be13-0010189e36c0
09:57:25,899 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
(hornetq-failure-check-thread) Connection failure has been detected: Did not receive data
from invm:0. It is likely the client has exited or crashed without closing its connection,
or the network between the server and client has failed. You also might have configured
connection-ttl and client-failure-check-period incorrectly. Please check user manual for
more information. The connection will now be closed. [code=3]
09:57:25,905 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Client connection failed, clearing up resources for session
05fcbb0c-6112-11e1-be13-0010189e36c0
09:57:25,907 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Cleared up resources for session
05fcbb0c-6112-11e1-be13-0010189e36c0
09:57:25,909 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
(hornetq-failure-check-thread) Connection failure has been detected: Did not receive data
from invm:0. It is likely the client has exited or crashed without closing its connection,
or the network between the server and client has failed. You also might have configured
connection-ttl and client-failure-check-period incorrectly. Please check user manual for
more information. The connection will now be closed. [code=3]
09:57:25,981 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Client connection failed, clearing up resources for session
755c7c7e-606e-11e1-be13-0010189e36c0
09:57:25,983 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Cleared up resources for session
755c7c7e-606e-11e1-be13-0010189e36c0
09:57:25,857 WARN [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Thread-878
(HornetQ-client-global-threads-1151364488)) Notified of connection failure in xa recovery
connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl
[initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0],
discoveryGroupConfiguration=null],
connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0,
backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=2
message=Channel disconnected]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:380)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:203)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.11.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_24]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_24]
09:57:25,867 WARN [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Thread-877
(HornetQ-client-global-threads-1151364488)) Notified of connection failure in xa recovery
connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl
[initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0],
discoveryGroupConfiguration=null],
connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0,
backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=2
message=Channel disconnected]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:380)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:203)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.11.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_24]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_24]
09:57:25,888 WARN [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Thread-876
(HornetQ-client-global-threads-1151364488)) Notified of connection failure in xa recovery
connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl
[initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0],
discoveryGroupConfiguration=null],
connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0,
backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=2
message=Channel disconnected]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:380)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:203)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.11.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_24]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_24]
09:57:25,899 WARN [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Thread-875
(HornetQ-client-global-threads-1151364488)) Notified of connection failure in xa recovery
connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl
[initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0],
discoveryGroupConfiguration=null],
connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0,
backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=2
message=Channel disconnected]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:380)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:203)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.11.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_24]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_24]
09:57:25,822 WARN [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Thread-889
(HornetQ-client-global-threads-1151364488)) Notified of connection failure in xa recovery
connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl
[initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0],
discoveryGroupConfiguration=null],
connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0,
backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=2
message=Channel disconnected]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:380)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:203)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.11.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_24]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_24]
09:57:25,984 INFO [com.parts.mail.jms.MailMessageDrivenBean] (Thread-872
(HornetQ-client-global-threads-1151364488)) Recived async mail
09:57:25,909 WARN [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Thread-874
(HornetQ-client-global-threads-1151364488)) Notified of connection failure in xa recovery
connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl
[initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0],
discoveryGroupConfiguration=null],
connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0,
backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=2
message=Channel disconnected]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:380)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:203)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.11.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_24]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_24]
09:57:25,837 WARN [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Thread-885
(HornetQ-client-global-threads-1151364488)) Notified of connection failure in xa recovery
connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl
[initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0],
discoveryGroupConfiguration=null],
connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0,
backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=2
message=Channel disconnected]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:380)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:203)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.11.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_24]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_24]
09:57:25,822 WARN [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Thread-881
(HornetQ-client-global-threads-1151364488)) Notified of connection failure in xa recovery
connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl
[initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0],
discoveryGroupConfiguration=null],
connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0,
backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=2
message=Channel disconnected]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:380)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:203)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.11.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_24]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_24]
09:57:25,837 WARN [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Thread-883
(HornetQ-client-global-threads-1151364488)) Notified of connection failure in xa recovery
connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl
[initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0],
discoveryGroupConfiguration=null],
connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0,
backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=2
message=Channel disconnected]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:380)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:203)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.11.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_24]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_24]
09:57:25,837 WARN [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Thread-887
(HornetQ-client-global-threads-1151364488)) Notified of connection failure in xa recovery
connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl
[initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0],
discoveryGroupConfiguration=null],
connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0,
backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=2
message=Channel disconnected]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:380)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:203)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.11.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_24]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_24]
09:57:25,788 WARN [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Thread-888
(HornetQ-client-global-threads-1151364488)) Notified of connection failure in xa recovery
connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl
[initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0],
discoveryGroupConfiguration=null],
connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0,
backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=2
message=Channel disconnected]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:380)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:203)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.11.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_24]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_24]
09:57:25,837 WARN [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Thread-882
(HornetQ-client-global-threads-1151364488)) Notified of connection failure in xa recovery
connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl
[initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0],
discoveryGroupConfiguration=null],
connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0,
backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=2
message=Channel disconnected]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:380)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:203)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.11.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_24]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_24]
09:57:25,821 WARN [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Thread-880
(HornetQ-client-global-threads-1151364488)) Notified of connection failure in xa recovery
connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl
[initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0],
discoveryGroupConfiguration=null],
connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0,
backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=2
message=Channel disconnected]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:380)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:203)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.11.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_24]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_24]
09:57:25,788 WARN [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Thread-884
(HornetQ-client-global-threads-1151364488)) Notified of connection failure in xa recovery
connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl
[initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0],
discoveryGroupConfiguration=null],
connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0,
backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=2
message=Channel disconnected]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:380)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:203)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.11.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_24]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_24]
09:57:26,028 WARN [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Thread-873
(HornetQ-client-global-threads-1151364488)) Notified of connection failure in xa recovery
connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl
[initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0],
discoveryGroupConfiguration=null],
connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0,
backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=2
message=Channel disconnected]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:380)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:203)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.11.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_24]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_24]
09:57:26,026 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
(hornetq-failure-check-thread) Connection failure has been detected: Did not receive data
from invm:0. It is likely the client has exited or crashed without closing its connection,
or the network between the server and client has failed. You also might have configured
connection-ttl and client-failure-check-period incorrectly. Please check user manual for
more information. The connection will now be closed. [code=3]
09:57:25,845 WARN [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Thread-879
(HornetQ-client-global-threads-1151364488)) Notified of connection failure in xa recovery
connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl
[initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0],
discoveryGroupConfiguration=null],
connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0,
backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=2
message=Channel disconnected]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:380)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:203)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.11.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_24]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_24]
09:57:26,253 INFO [com.parts.contract.bean.PerformanceContractPrincipals]
(ajp--127.0.0.1-8009-5) Performance Contract Principals POST CONSTRUCT
09:57:26,248 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Client connection failed, clearing up resources for session
b87fd8da-6111-11e1-be13-0010189e36c0
09:57:26,090 INFO [com.parts.mail.jms.MailMessageDrivenBean] (Thread-872
(HornetQ-client-global-threads-1151364488)) EMailMessage:
com.parts.mail.model.EmailMessage@3b37e6b5
09:57:26,270 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Cleared up resources for session
b87fd8da-6111-11e1-be13-0010189e36c0
09:57:26,273 INFO [com.parts.mail.SendMailListener] (Thread-872
(HornetQ-client-global-threads-1151364488)) EmailMessage to send with id:
CONTRACT-MSG-TOKEN:a4d6cca1-d1e4-494c-a2d4-c61c0cfb9d06
09:57:26,275 WARN [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Thread-884
(HornetQ-client-global-threads-1151364488)) Notified of connection failure in xa recovery
connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl
[initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0],
discoveryGroupConfiguration=null],
connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0,
backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=2
message=Channel disconnected]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:380)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:203)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.11.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_24]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_24]
09:57:26,275 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
(hornetq-failure-check-thread) Connection failure has been detected: Did not receive data
from invm:0. It is likely the client has exited or crashed without closing its connection,
or the network between the server and client has failed. You also might have configured
connection-ttl and client-failure-check-period incorrectly. Please check user manual for
more information. The connection will now be closed. [code=3]
09:57:26,279 INFO [com.parts.mail.SendMailListener] (Thread-872
(HornetQ-client-global-threads-1151364488)) Send message for id
CONTRACT-MSG-TOKEN:a4d6cca1-d1e4-494c-a2d4-c61c0cfb9d06 FROM 'Musific Team' with
email 'team(a)musific.com'
09:57:26,278 INFO [com.parts.contract.bean.PerformanceContractInbox]
(ajp--127.0.0.1-8009-6) Contract Inbox loaded for Moelv Musikkforum
09:57:26,302 INFO [com.parts.mail.SendMailListener] (Thread-872
(HornetQ-client-global-threads-1151364488)) Send message for id
CONTRACT-MSG-TOKEN:a4d6cca1-d1e4-494c-a2d4-c61c0cfb9d06 TO 'Kjell-Ove Torkildsen'
with email 'krodum(a)musific.com'
09:57:26,299 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Client connection failed, clearing up resources for session
fc39ef8a-6118-11e1-be13-0010189e36c0
09:57:26,308 INFO [com.parts.mail.SendMailListener] (Thread-872
(HornetQ-client-global-threads-1151364488)) Produce mail session test mode: false
09:57:26,310 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Cleared up resources for session
fc39ef8a-6118-11e1-be13-0010189e36c0
09:57:26,314 WARN [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Thread-879
(HornetQ-client-global-threads-1151364488)) Notified of connection failure in xa recovery
connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl
[initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0],
discoveryGroupConfiguration=null],
connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0,
backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=2
message=Channel disconnected]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:380)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:203)
[hornetq-core-2.2.11.Final.jar:]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.11.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_24]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_24]
{noformat}
HornetQ causing deadlock when max-size-bytes reaches its limit
--------------------------------------------------------------
Key: AS7-3919
URL: https://issues.jboss.org/browse/AS7-3919
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.0.Final
Reporter: Ove Ranheim
Assignee: Andy Taylor
Priority: Critical
Fix For: 7.1.1.Final
I have journal-file-size set to 10 Mb and max-size-bytes set 10 Mb, with a address policy
of BLOCK. When my sendMailQueue reaches its limit a deadlock occurs. This is further
emphasized in the JBossAS Forum reference.
Here's a snippet from jstack after HornetQ crashes.
{noformat}
Found one Java-level deadlock:
=============================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x0000000041df8640 (object 0x00000000ea17d6e0, a
java.lang.Object),
which is held by "Thread-873 (HornetQ-client-global-threads-1151364488)"
"Thread-873 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x000000004107c8b8 (object 0x00000000ea17d6d0, a
java.lang.Object),
which is held by "Thread-878 (HornetQ-client-global-threads-1151364488)"
Java stack information for the threads listed above:
===================================================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:461)
- waiting to lock <0x00000000ea17d6e0> (a java.lang.Object)
- locked <0x00000000ea17d6d0> (a java.lang.Object)
at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1294)
at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1238)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:392)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000f0222060> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"Thread-873 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:458)
- waiting to lock <0x00000000ea17d6d0> (a java.lang.Object)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:391)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000ea17d6e0> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Found 1 deadlock.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:48 a.m.
[
https://issues.jboss.org/browse/AS7-3919?page=com.atlassian.jira.plugin.s...
]
Ove Ranheim updated AS7-3919:
-----------------------------
Attachment: jstack-2012-02-27_08_49.txt
jstack-2012-02-27_09_58_crash.txt
JStack dump attached for before and after the crash.
HornetQ causing deadlock when max-size-bytes reaches its limit
--------------------------------------------------------------
Key: AS7-3919
URL: https://issues.jboss.org/browse/AS7-3919
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.0.Final
Reporter: Ove Ranheim
Assignee: Andy Taylor
Priority: Critical
Fix For: 7.1.1.Final
Attachments: jstack-2012-02-27_08_49.txt, jstack-2012-02-27_09_58_crash.txt
I have journal-file-size set to 10 Mb and max-size-bytes set 10 Mb, with a address policy
of BLOCK. When my sendMailQueue reaches its limit a deadlock occurs. This is further
emphasized in the JBossAS Forum reference.
Here's a snippet from jstack after HornetQ crashes.
{noformat}
Found one Java-level deadlock:
=============================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x0000000041df8640 (object 0x00000000ea17d6e0, a
java.lang.Object),
which is held by "Thread-873 (HornetQ-client-global-threads-1151364488)"
"Thread-873 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x000000004107c8b8 (object 0x00000000ea17d6d0, a
java.lang.Object),
which is held by "Thread-878 (HornetQ-client-global-threads-1151364488)"
Java stack information for the threads listed above:
===================================================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:461)
- waiting to lock <0x00000000ea17d6e0> (a java.lang.Object)
- locked <0x00000000ea17d6d0> (a java.lang.Object)
at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1294)
at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1238)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:392)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000f0222060> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"Thread-873 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:458)
- waiting to lock <0x00000000ea17d6d0> (a java.lang.Object)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:391)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000ea17d6e0> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Found 1 deadlock.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:45 p.m.
[
https://issues.jboss.org/browse/AS7-3919?page=com.atlassian.jira.plugin.s...
]
Clebert Suconic commented on AS7-3919:
--------------------------------------
There's some configuration error on your test. You're setting pageSize=10M, and
maxSize=10M and the system shouldn't be started with that.
I agree the system should safe guard you from starting it with this wrong setup (which we
can address the proper message on startup).
But the configuration is broken. This is likely a configuration issue with an
inappropriate error message from hornetq
HornetQ causing deadlock when max-size-bytes reaches its limit
--------------------------------------------------------------
Key: AS7-3919
URL: https://issues.jboss.org/browse/AS7-3919
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.0.Final
Reporter: Ove Ranheim
Assignee: Andy Taylor
Priority: Critical
Fix For: 7.1.1.Final
Attachments: jstack-2012-02-27_08_49.txt, jstack-2012-02-27_09_58_crash.txt
I have journal-file-size set to 10 Mb and max-size-bytes set 10 Mb, with a address policy
of BLOCK. When my sendMailQueue reaches its limit a deadlock occurs. This is further
emphasized in the JBossAS Forum reference.
Here's a snippet from jstack after HornetQ crashes.
{noformat}
Found one Java-level deadlock:
=============================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x0000000041df8640 (object 0x00000000ea17d6e0, a
java.lang.Object),
which is held by "Thread-873 (HornetQ-client-global-threads-1151364488)"
"Thread-873 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x000000004107c8b8 (object 0x00000000ea17d6d0, a
java.lang.Object),
which is held by "Thread-878 (HornetQ-client-global-threads-1151364488)"
Java stack information for the threads listed above:
===================================================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:461)
- waiting to lock <0x00000000ea17d6e0> (a java.lang.Object)
- locked <0x00000000ea17d6d0> (a java.lang.Object)
at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1294)
at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1238)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:392)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000f0222060> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"Thread-873 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:458)
- waiting to lock <0x00000000ea17d6d0> (a java.lang.Object)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:391)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000ea17d6e0> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Found 1 deadlock.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:46 a.m.
[
https://issues.jboss.org/browse/AS7-3919?page=com.atlassian.jira.plugin.s...
]
Miroslav Novak commented on AS7-3919:
-------------------------------------
Hi Ove,
could you upload your standalone-...xml, please? Also if you know exactly the steps how to
reach the deadlock it would be great help for us.
I tried to reproduce the deadlock using description in community forum but without
success.
Thank you,
Mirek
HornetQ causing deadlock when max-size-bytes reaches its limit
--------------------------------------------------------------
Key: AS7-3919
URL: https://issues.jboss.org/browse/AS7-3919
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.0.Final
Reporter: Ove Ranheim
Assignee: Andy Taylor
Priority: Critical
Fix For: 7.1.1.Final
Attachments: jstack-2012-02-27_08_49.txt, jstack-2012-02-27_09_58_crash.txt
I have journal-file-size set to 10 Mb and max-size-bytes set 10 Mb, with a address policy
of BLOCK. When my sendMailQueue reaches its limit a deadlock occurs. This is further
emphasized in the JBossAS Forum reference.
Here's a snippet from jstack after HornetQ crashes.
{noformat}
Found one Java-level deadlock:
=============================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x0000000041df8640 (object 0x00000000ea17d6e0, a
java.lang.Object),
which is held by "Thread-873 (HornetQ-client-global-threads-1151364488)"
"Thread-873 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x000000004107c8b8 (object 0x00000000ea17d6d0, a
java.lang.Object),
which is held by "Thread-878 (HornetQ-client-global-threads-1151364488)"
Java stack information for the threads listed above:
===================================================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:461)
- waiting to lock <0x00000000ea17d6e0> (a java.lang.Object)
- locked <0x00000000ea17d6d0> (a java.lang.Object)
at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1294)
at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1238)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:392)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000f0222060> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"Thread-873 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:458)
- waiting to lock <0x00000000ea17d6d0> (a java.lang.Object)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:391)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000ea17d6e0> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Found 1 deadlock.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:12 a.m.
[
https://issues.jboss.org/browse/AS7-3919?page=com.atlassian.jira.plugin.s...
]
Ove Ranheim updated AS7-3919:
-----------------------------
Attachment: standalone-musific.xml
Adding Musific's standalone configuration. Have a look at the sendMailQueue configured
with address policy BLOCK.
HornetQ causing deadlock when max-size-bytes reaches its limit
--------------------------------------------------------------
Key: AS7-3919
URL: https://issues.jboss.org/browse/AS7-3919
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.0.Final
Reporter: Ove Ranheim
Assignee: Andy Taylor
Priority: Critical
Fix For: 7.1.1.Final
Attachments: jstack-2012-02-27_08_49.txt, jstack-2012-02-27_09_58_crash.txt,
standalone-musific.xml
I have journal-file-size set to 10 Mb and max-size-bytes set 10 Mb, with a address policy
of BLOCK. When my sendMailQueue reaches its limit a deadlock occurs. This is further
emphasized in the JBossAS Forum reference.
Here's a snippet from jstack after HornetQ crashes.
{noformat}
Found one Java-level deadlock:
=============================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x0000000041df8640 (object 0x00000000ea17d6e0, a
java.lang.Object),
which is held by "Thread-873 (HornetQ-client-global-threads-1151364488)"
"Thread-873 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x000000004107c8b8 (object 0x00000000ea17d6d0, a
java.lang.Object),
which is held by "Thread-878 (HornetQ-client-global-threads-1151364488)"
Java stack information for the threads listed above:
===================================================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:461)
- waiting to lock <0x00000000ea17d6e0> (a java.lang.Object)
- locked <0x00000000ea17d6d0> (a java.lang.Object)
at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1294)
at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1238)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:392)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000f0222060> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"Thread-873 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:458)
- waiting to lock <0x00000000ea17d6d0> (a java.lang.Object)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:391)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000ea17d6e0> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Found 1 deadlock.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:24 a.m.
[
https://issues.jboss.org/browse/AS7-3919?page=com.atlassian.jira.plugin.s...
]
Ove Ranheim commented on AS7-3919:
----------------------------------
We use Seam 3.1.0.Final together with JBoss AS 7.1.0.Final.
In Seam we bind the CDI Event to the JMS Queue by:
{noformat}
public interface JmsMailRoutingListener {
final static String MAIL_JMS_DESTINATION = "queue/sendMail";
@Routing(RouteType.EGRESS)
public void mapAsyncMailToQueue(
@Observes AsyncEmailMessage mailMessage,
@JmsDestination(jndiName = MAIL_JMS_DESTINATION) Queue q
);
}
{noformat}
The MDB is implemented as:
{noformat}
@MessageDriven(name = "MailProcessor", activationConfig = {
@ActivationConfigProperty(propertyName = "destinationType",
propertyValue = "javax.jms.Queue"),
@ActivationConfigProperty(propertyName = "destination", propertyValue =
JmsMailRoutingListener.MAIL_JMS_DESTINATION)})
public class MailMessageDrivenBean implements MessageListener {
@Override
public void onMessage(Message message) {
AsyncEmailMessage asyncEmailMessage =
objectMessageExtractor.extractObject(message, AsyncEmailMessage.class);
// handling code goes here
}
}
{noformat}
To put an object on the queue, we do:
{noformat}
@ApplicationScoped
public class ContractMessagingBean implements Serializable {
@Inject
private Event<AsyncEmailMessage> notificationEvent;
private String sendNotification(PerformanceContract contract, String templateName,
List<EmailContact> recipients, ContractPartyName recipient, boolean id) {
AsyncEmailMessage notification = notificationInstance.get();
notificationEvent.fire(notification);
}
}
{noformat}
So what's happening here are:
1. We bind with RoutType.EGRESS so all CDI Event objects of type AsyncEmailMessage will be
sent as a persisted object onto the sendMailQueue through JmsMailRoutingListener (this is
a great capability of Seam JMS)
2. We fire AsyncMailMessage onto the CDI Event observer from
ContractMessagingBean.sendNotification(..)
3. The message will be picked up when notified by HornetQ on
MailMessageDrivenBean.onMessage(..)
4. We extract the object and handle it. Finally we send a mail out through the smtp
gateway.
After sending approx 30 messages onto the queue, we get the exception described in this
issue.
HornetQ causing deadlock when max-size-bytes reaches its limit
--------------------------------------------------------------
Key: AS7-3919
URL: https://issues.jboss.org/browse/AS7-3919
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.0.Final
Reporter: Ove Ranheim
Assignee: Andy Taylor
Priority: Critical
Fix For: 7.1.1.Final
Attachments: jstack-2012-02-27_08_49.txt, jstack-2012-02-27_09_58_crash.txt,
standalone-musific.xml
I have journal-file-size set to 10 Mb and max-size-bytes set 10 Mb, with a address policy
of BLOCK. When my sendMailQueue reaches its limit a deadlock occurs. This is further
emphasized in the JBossAS Forum reference.
Here's a snippet from jstack after HornetQ crashes.
{noformat}
Found one Java-level deadlock:
=============================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x0000000041df8640 (object 0x00000000ea17d6e0, a
java.lang.Object),
which is held by "Thread-873 (HornetQ-client-global-threads-1151364488)"
"Thread-873 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x000000004107c8b8 (object 0x00000000ea17d6d0, a
java.lang.Object),
which is held by "Thread-878 (HornetQ-client-global-threads-1151364488)"
Java stack information for the threads listed above:
===================================================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:461)
- waiting to lock <0x00000000ea17d6e0> (a java.lang.Object)
- locked <0x00000000ea17d6d0> (a java.lang.Object)
at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1294)
at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1238)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:392)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000f0222060> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"Thread-873 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:458)
- waiting to lock <0x00000000ea17d6d0> (a java.lang.Object)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:391)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000ea17d6e0> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Found 1 deadlock.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:12 p.m.
[
https://issues.jboss.org/browse/AS7-3919?page=com.atlassian.jira.plugin.s...
]
Miroslav Novak commented on AS7-3919:
-------------------------------------
Thanks Ove for such a quick update.
I'm not sure what is the cause that hornetq resource adapter lost connection to in-vm
acceptor. This could be bad configuration but I don't understand fully the Seam thing
of sending message.
Still from threaddump it seems that it leads to multiple calls of
ClientSessionFactoryImpl.handleConnectionFailure which acquire varible 'sessions'
as lock and results to a state where one instance of ClientSessionFactoryImpl is trying to
close other instances ClientSessionFactoryImpl.
This fails on acquiring lock - variable 'sessions' in ClientSessionFactoryImpl
which were acquired at the beginning by ClientSessionFactoryImpl.handleConnectionFailure.
HornetQ causing deadlock when max-size-bytes reaches its limit
--------------------------------------------------------------
Key: AS7-3919
URL: https://issues.jboss.org/browse/AS7-3919
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.0.Final
Reporter: Ove Ranheim
Assignee: Andy Taylor
Priority: Critical
Fix For: 7.1.1.Final
Attachments: jstack-2012-02-27_08_49.txt, jstack-2012-02-27_09_58_crash.txt,
standalone-musific.xml
I have journal-file-size set to 10 Mb and max-size-bytes set 10 Mb, with a address policy
of BLOCK. When my sendMailQueue reaches its limit a deadlock occurs. This is further
emphasized in the JBossAS Forum reference.
Here's a snippet from jstack after HornetQ crashes.
{noformat}
Found one Java-level deadlock:
=============================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x0000000041df8640 (object 0x00000000ea17d6e0, a
java.lang.Object),
which is held by "Thread-873 (HornetQ-client-global-threads-1151364488)"
"Thread-873 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x000000004107c8b8 (object 0x00000000ea17d6d0, a
java.lang.Object),
which is held by "Thread-878 (HornetQ-client-global-threads-1151364488)"
Java stack information for the threads listed above:
===================================================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:461)
- waiting to lock <0x00000000ea17d6e0> (a java.lang.Object)
- locked <0x00000000ea17d6d0> (a java.lang.Object)
at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1294)
at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1238)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:392)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000f0222060> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"Thread-873 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:458)
- waiting to lock <0x00000000ea17d6d0> (a java.lang.Object)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:391)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000ea17d6e0> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Found 1 deadlock.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:36 p.m.
[
https://issues.jboss.org/browse/AS7-3919?page=com.atlassian.jira.plugin.s...
]
Clebert Suconic commented on AS7-3919:
--------------------------------------
@Miro: he had a bad configuration on page-size. I keep asking to someone to validate
page-size >= max-size and set it to block.
page-size should be ignored on block, but it's throwing an IllegalStateException what
could cause some collateral damage.
HornetQ causing deadlock when max-size-bytes reaches its limit
--------------------------------------------------------------
Key: AS7-3919
URL: https://issues.jboss.org/browse/AS7-3919
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.0.Final
Reporter: Ove Ranheim
Assignee: Andy Taylor
Priority: Critical
Fix For: 7.1.1.Final
Attachments: jstack-2012-02-27_08_49.txt, jstack-2012-02-27_09_58_crash.txt,
standalone-musific.xml
I have journal-file-size set to 10 Mb and max-size-bytes set 10 Mb, with a address policy
of BLOCK. When my sendMailQueue reaches its limit a deadlock occurs. This is further
emphasized in the JBossAS Forum reference.
Here's a snippet from jstack after HornetQ crashes.
{noformat}
Found one Java-level deadlock:
=============================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x0000000041df8640 (object 0x00000000ea17d6e0, a
java.lang.Object),
which is held by "Thread-873 (HornetQ-client-global-threads-1151364488)"
"Thread-873 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x000000004107c8b8 (object 0x00000000ea17d6d0, a
java.lang.Object),
which is held by "Thread-878 (HornetQ-client-global-threads-1151364488)"
Java stack information for the threads listed above:
===================================================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:461)
- waiting to lock <0x00000000ea17d6e0> (a java.lang.Object)
- locked <0x00000000ea17d6d0> (a java.lang.Object)
at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1294)
at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1238)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:392)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000f0222060> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"Thread-873 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:458)
- waiting to lock <0x00000000ea17d6d0> (a java.lang.Object)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:391)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000ea17d6e0> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Found 1 deadlock.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:26 p.m.
[
https://issues.jboss.org/browse/AS7-3919?page=com.atlassian.jira.plugin.s...
]
Ove Ranheim commented on AS7-3919:
----------------------------------
Please see my forum comment after testing.
Given that page-size is set for block policy it does not cause a dead-lock. That's
really good news. So, AS7.1.x should have a page-size set to 1/10 of max-size-bytes so
default configuration does make confusion. Conversely, the page-size must be set in
configuration to e.g. <page-size-bytes>1048576</page-size-bytes>.
HornetQ causing deadlock when max-size-bytes reaches its limit
--------------------------------------------------------------
Key: AS7-3919
URL: https://issues.jboss.org/browse/AS7-3919
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.0.Final
Reporter: Ove Ranheim
Assignee: Andy Taylor
Priority: Critical
Fix For: 7.1.1.Final
Attachments: jstack-2012-02-27_08_49.txt, jstack-2012-02-27_09_58_crash.txt,
standalone-musific.xml
I have journal-file-size set to 10 Mb and max-size-bytes set 10 Mb, with a address policy
of BLOCK. When my sendMailQueue reaches its limit a deadlock occurs. This is further
emphasized in the JBossAS Forum reference.
Here's a snippet from jstack after HornetQ crashes.
{noformat}
Found one Java-level deadlock:
=============================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x0000000041df8640 (object 0x00000000ea17d6e0, a
java.lang.Object),
which is held by "Thread-873 (HornetQ-client-global-threads-1151364488)"
"Thread-873 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x000000004107c8b8 (object 0x00000000ea17d6d0, a
java.lang.Object),
which is held by "Thread-878 (HornetQ-client-global-threads-1151364488)"
Java stack information for the threads listed above:
===================================================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:461)
- waiting to lock <0x00000000ea17d6e0> (a java.lang.Object)
- locked <0x00000000ea17d6d0> (a java.lang.Object)
at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1294)
at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1238)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:392)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000f0222060> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"Thread-873 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:458)
- waiting to lock <0x00000000ea17d6d0> (a java.lang.Object)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:391)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000ea17d6e0> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Found 1 deadlock.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:20 p.m.
New subject: [JBoss JIRA] (AS7-3919) Invalid PageSize and blocking could lead to collateral issues
[
https://issues.jboss.org/browse/AS7-3919?page=com.atlassian.jira.plugin.s...
]
Clebert Suconic updated AS7-3919:
---------------------------------
Summary: Invalid PageSize and blocking could lead to collateral issues
(was: HornetQ causing deadlock when max-size-bytes reaches its limit)
Description:
Page-size should be ignored on blocking. If the pageSize is configured beyond or the same
as max-size, the constructor will throw an exception what could cause collateral damage
since some state will be left unfilled.
The user has initially described this issue.
I have journal-file-size set to 10 Mb and max-size-bytes set 10 Mb, with a address policy
of BLOCK. When my sendMailQueue reaches its limit a deadlock occurs. This is further
emphasized in the JBossAS Forum reference.
Here's a snippet from jstack after HornetQ crashes.
{noformat}
Found one Java-level deadlock:
=============================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x0000000041df8640 (object 0x00000000ea17d6e0, a
java.lang.Object),
which is held by "Thread-873 (HornetQ-client-global-threads-1151364488)"
"Thread-873 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x000000004107c8b8 (object 0x00000000ea17d6d0, a
java.lang.Object),
which is held by "Thread-878 (HornetQ-client-global-threads-1151364488)"
Java stack information for the threads listed above:
===================================================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:461)
- waiting to lock <0x00000000ea17d6e0> (a java.lang.Object)
- locked <0x00000000ea17d6d0> (a java.lang.Object)
at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1294)
at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1238)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:392)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000f0222060> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"Thread-873 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:458)
- waiting to lock <0x00000000ea17d6d0> (a java.lang.Object)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:391)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000ea17d6e0> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Found 1 deadlock.
{noformat}
was:
I have journal-file-size set to 10 Mb and max-size-bytes set 10 Mb, with a address policy
of BLOCK. When my sendMailQueue reaches its limit a deadlock occurs. This is further
emphasized in the JBossAS Forum reference.
Here's a snippet from jstack after HornetQ crashes.
{noformat}
Found one Java-level deadlock:
=============================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x0000000041df8640 (object 0x00000000ea17d6e0, a
java.lang.Object),
which is held by "Thread-873 (HornetQ-client-global-threads-1151364488)"
"Thread-873 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x000000004107c8b8 (object 0x00000000ea17d6d0, a
java.lang.Object),
which is held by "Thread-878 (HornetQ-client-global-threads-1151364488)"
Java stack information for the threads listed above:
===================================================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:461)
- waiting to lock <0x00000000ea17d6e0> (a java.lang.Object)
- locked <0x00000000ea17d6d0> (a java.lang.Object)
at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1294)
at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1238)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:392)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000f0222060> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"Thread-873 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:458)
- waiting to lock <0x00000000ea17d6d0> (a java.lang.Object)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:391)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000ea17d6e0> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Found 1 deadlock.
{noformat}
Workaround Description: Configure the pagesize < max-size as it's been defined
on the user manual
Workaround: Workaround Exists
Forum Reference: https://community.jboss.org/thread/195901?tstart=0 (was:
https://community.jboss.org/thread/195901?tstart=0)
Invalid PageSize and blocking could lead to collateral issues
-------------------------------------------------------------
Key: AS7-3919
URL: https://issues.jboss.org/browse/AS7-3919
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.0.Final
Reporter: Ove Ranheim
Assignee: Andy Taylor
Priority: Critical
Fix For: 7.1.1.Final
Attachments: jstack-2012-02-27_08_49.txt, jstack-2012-02-27_09_58_crash.txt,
standalone-musific.xml
Page-size should be ignored on blocking. If the pageSize is configured beyond or the same
as max-size, the constructor will throw an exception what could cause collateral damage
since some state will be left unfilled.
The user has initially described this issue.
I have journal-file-size set to 10 Mb and max-size-bytes set 10 Mb, with a address policy
of BLOCK. When my sendMailQueue reaches its limit a deadlock occurs. This is further
emphasized in the JBossAS Forum reference.
Here's a snippet from jstack after HornetQ crashes.
{noformat}
Found one Java-level deadlock:
=============================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x0000000041df8640 (object 0x00000000ea17d6e0, a
java.lang.Object),
which is held by "Thread-873 (HornetQ-client-global-threads-1151364488)"
"Thread-873 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x000000004107c8b8 (object 0x00000000ea17d6d0, a
java.lang.Object),
which is held by "Thread-878 (HornetQ-client-global-threads-1151364488)"
Java stack information for the threads listed above:
===================================================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:461)
- waiting to lock <0x00000000ea17d6e0> (a java.lang.Object)
- locked <0x00000000ea17d6d0> (a java.lang.Object)
at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1294)
at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1238)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:392)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000f0222060> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"Thread-873 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:458)
- waiting to lock <0x00000000ea17d6d0> (a java.lang.Object)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:391)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000ea17d6e0> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Found 1 deadlock.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:22 p.m.
New subject: [JBoss JIRA] (AS7-3919) Invalid PageSize and blocking could lead to collateral issues
[
https://issues.jboss.org/browse/AS7-3919?page=com.atlassian.jira.plugin.s...
]
Clebert Suconic updated AS7-3919:
---------------------------------
Priority: Minor (was: Critical)
Forum Reference: https://community.jboss.org/thread/195901?tstart=0 (was:
https://community.jboss.org/thread/195901?tstart=0)
Invalid PageSize and blocking could lead to collateral issues
-------------------------------------------------------------
Key: AS7-3919
URL: https://issues.jboss.org/browse/AS7-3919
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.0.Final
Reporter: Ove Ranheim
Assignee: Andy Taylor
Priority: Minor
Fix For: 7.1.1.Final
Attachments: jstack-2012-02-27_08_49.txt, jstack-2012-02-27_09_58_crash.txt,
standalone-musific.xml
Page-size should be ignored on blocking. If the pageSize is configured beyond or the same
as max-size, the constructor will throw an exception what could cause collateral damage
since some state will be left unfilled.
The user has initially described this issue.
I have journal-file-size set to 10 Mb and max-size-bytes set 10 Mb, with a address policy
of BLOCK. When my sendMailQueue reaches its limit a deadlock occurs. This is further
emphasized in the JBossAS Forum reference.
Here's a snippet from jstack after HornetQ crashes.
{noformat}
Found one Java-level deadlock:
=============================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x0000000041df8640 (object 0x00000000ea17d6e0, a
java.lang.Object),
which is held by "Thread-873 (HornetQ-client-global-threads-1151364488)"
"Thread-873 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x000000004107c8b8 (object 0x00000000ea17d6d0, a
java.lang.Object),
which is held by "Thread-878 (HornetQ-client-global-threads-1151364488)"
Java stack information for the threads listed above:
===================================================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:461)
- waiting to lock <0x00000000ea17d6e0> (a java.lang.Object)
- locked <0x00000000ea17d6d0> (a java.lang.Object)
at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1294)
at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1238)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:392)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000f0222060> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"Thread-873 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:458)
- waiting to lock <0x00000000ea17d6d0> (a java.lang.Object)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:391)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000ea17d6e0> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Found 1 deadlock.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:41 a.m.
New subject: [JBoss JIRA] (AS7-3919) Invalid PageSize and blocking could lead to collateral issues
[
https://issues.jboss.org/browse/AS7-3919?page=com.atlassian.jira.plugin.s...
]
Miroslav Novak commented on AS7-3919:
-------------------------------------
@Clebert
From reading forum comments I think there could be improved two
things:
a) There are no warning messages in the log that page-size-bytes >=
max-size-bytes.
b) Ove correctly pointed out that max-size-bytes == page-size-bytes by default in
configuration. Here is default address settings from standalone-full.xml:
{code}
<address-setting match="#">
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<max-size-bytes>10485760</max-size-bytes>
<address-full-policy>BLOCK</address-full-policy>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
</address-setting>
{code}
Attribut <page-size-bytes> is not set and is by default 10485760 - taken from
org.hornetq.core.settings.impl.AddressSettings.java:
{code}
public static final long DEFAULT_MAX_SIZE_BYTES = -1;
public static final long DEFAULT_PAGE_SIZE = 10 * 1024 * 1024;
{code}
We could just delete <max-size-bytes>10485760</max-size-bytes> from
<address-settings>. What do you think?
Invalid PageSize and blocking could lead to collateral issues
-------------------------------------------------------------
Key: AS7-3919
URL: https://issues.jboss.org/browse/AS7-3919
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.0.Final
Reporter: Ove Ranheim
Assignee: Andy Taylor
Priority: Minor
Fix For: 7.1.1.Final
Attachments: jstack-2012-02-27_08_49.txt, jstack-2012-02-27_09_58_crash.txt,
standalone-musific.xml
Page-size should be ignored on blocking. If the pageSize is configured beyond or the same
as max-size, the constructor will throw an exception what could cause collateral damage
since some state will be left unfilled.
The user has initially described this issue.
I have journal-file-size set to 10 Mb and max-size-bytes set 10 Mb, with a address policy
of BLOCK. When my sendMailQueue reaches its limit a deadlock occurs. This is further
emphasized in the JBossAS Forum reference.
Here's a snippet from jstack after HornetQ crashes.
{noformat}
Found one Java-level deadlock:
=============================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x0000000041df8640 (object 0x00000000ea17d6e0, a
java.lang.Object),
which is held by "Thread-873 (HornetQ-client-global-threads-1151364488)"
"Thread-873 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x000000004107c8b8 (object 0x00000000ea17d6d0, a
java.lang.Object),
which is held by "Thread-878 (HornetQ-client-global-threads-1151364488)"
Java stack information for the threads listed above:
===================================================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:461)
- waiting to lock <0x00000000ea17d6e0> (a java.lang.Object)
- locked <0x00000000ea17d6d0> (a java.lang.Object)
at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1294)
at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1238)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:392)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000f0222060> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"Thread-873 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:458)
- waiting to lock <0x00000000ea17d6d0> (a java.lang.Object)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:391)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000ea17d6e0> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Found 1 deadlock.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:39 a.m.
New subject: [JBoss JIRA] (AS7-3919) Invalid PageSize and blocking could lead to collateral issues
[
https://issues.jboss.org/browse/AS7-3919?page=com.atlassian.jira.plugin.s...
]
Jason Greene updated AS7-3919:
------------------------------
Fix Version/s: 7.1.2.Final
(was: 7.1.1.Final)
Invalid PageSize and blocking could lead to collateral issues
-------------------------------------------------------------
Key: AS7-3919
URL: https://issues.jboss.org/browse/AS7-3919
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.0.Final
Reporter: Ove Ranheim
Assignee: Andy Taylor
Priority: Minor
Fix For: 7.1.2.Final
Attachments: jstack-2012-02-27_08_49.txt, jstack-2012-02-27_09_58_crash.txt,
standalone-musific.xml
Page-size should be ignored on blocking. If the pageSize is configured beyond or the same
as max-size, the constructor will throw an exception what could cause collateral damage
since some state will be left unfilled.
The user has initially described this issue.
I have journal-file-size set to 10 Mb and max-size-bytes set 10 Mb, with a address policy
of BLOCK. When my sendMailQueue reaches its limit a deadlock occurs. This is further
emphasized in the JBossAS Forum reference.
Here's a snippet from jstack after HornetQ crashes.
{noformat}
Found one Java-level deadlock:
=============================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x0000000041df8640 (object 0x00000000ea17d6e0, a
java.lang.Object),
which is held by "Thread-873 (HornetQ-client-global-threads-1151364488)"
"Thread-873 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x000000004107c8b8 (object 0x00000000ea17d6d0, a
java.lang.Object),
which is held by "Thread-878 (HornetQ-client-global-threads-1151364488)"
Java stack information for the threads listed above:
===================================================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:461)
- waiting to lock <0x00000000ea17d6e0> (a java.lang.Object)
- locked <0x00000000ea17d6d0> (a java.lang.Object)
at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1294)
at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1238)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:392)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000f0222060> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"Thread-873 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:458)
- waiting to lock <0x00000000ea17d6d0> (a java.lang.Object)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:391)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000ea17d6e0> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Found 1 deadlock.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:37 p.m.
New subject: [JBoss JIRA] (AS7-3919) Invalid PageSize and blocking could lead to collateral issues
[
https://issues.jboss.org/browse/AS7-3919?page=com.atlassian.jira.plugin.s...
]
Jeff Mesnil commented on AS7-3919:
----------------------------------
Is it still an issue?
By default, AS7 messaging subsystem is configured to BLOCK when an address reaches 10MiB
max-size-bytes.
If the user wants to PAGE instead, he should also change the max-size-bytes *and*
page-cache-bytes accordingly.
If we want to simplify switching from BLOCK to PAGE, I'd suggest to increased the
value of max-size-bytes in standalone-full[-ha].xml to 50MiB (5 times the value of
page-cache-bytes).
In any case, these 2 values are highly dependent on the number of queues managed by
HornetQ and the memory available on the servers. The user should specify values making
sense for his topology/hardware resources rather than rely on default values.
Invalid PageSize and blocking could lead to collateral issues
-------------------------------------------------------------
Key: AS7-3919
URL: https://issues.jboss.org/browse/AS7-3919
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.0.Final
Reporter: Ove Ranheim
Assignee: Andy Taylor
Priority: Minor
Fix For: 7.1.2.Final-redhat1
Attachments: jstack-2012-02-27_08_49.txt, jstack-2012-02-27_09_58_crash.txt,
standalone-musific.xml
Page-size should be ignored on blocking. If the pageSize is configured beyond or the same
as max-size, the constructor will throw an exception what could cause collateral damage
since some state will be left unfilled.
The user has initially described this issue.
I have journal-file-size set to 10 Mb and max-size-bytes set 10 Mb, with a address policy
of BLOCK. When my sendMailQueue reaches its limit a deadlock occurs. This is further
emphasized in the JBossAS Forum reference.
Here's a snippet from jstack after HornetQ crashes.
{noformat}
Found one Java-level deadlock:
=============================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x0000000041df8640 (object 0x00000000ea17d6e0, a
java.lang.Object),
which is held by "Thread-873 (HornetQ-client-global-threads-1151364488)"
"Thread-873 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x000000004107c8b8 (object 0x00000000ea17d6d0, a
java.lang.Object),
which is held by "Thread-878 (HornetQ-client-global-threads-1151364488)"
Java stack information for the threads listed above:
===================================================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:461)
- waiting to lock <0x00000000ea17d6e0> (a java.lang.Object)
- locked <0x00000000ea17d6d0> (a java.lang.Object)
at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1294)
at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1238)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:392)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000f0222060> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"Thread-873 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:458)
- waiting to lock <0x00000000ea17d6d0> (a java.lang.Object)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:391)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000ea17d6e0> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Found 1 deadlock.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:57 a.m.
New subject: [JBoss JIRA] (AS7-3919) Invalid PageSize and blocking could lead to collateral issues
[
https://issues.jboss.org/browse/AS7-3919?page=com.atlassian.jira.plugin.s...
]
Jason Greene updated AS7-3919:
------------------------------
Fix Version/s: 7.1.3.Final (EAP)
(was: 7.1.2.Final (EAP))
Invalid PageSize and blocking could lead to collateral issues
-------------------------------------------------------------
Key: AS7-3919
URL: https://issues.jboss.org/browse/AS7-3919
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.0.Final
Reporter: Ove Ranheim
Assignee: Andy Taylor
Priority: Minor
Fix For: 7.1.3.Final (EAP)
Attachments: jstack-2012-02-27_08_49.txt, jstack-2012-02-27_09_58_crash.txt,
standalone-musific.xml
Page-size should be ignored on blocking. If the pageSize is configured beyond or the same
as max-size, the constructor will throw an exception what could cause collateral damage
since some state will be left unfilled.
The user has initially described this issue.
I have journal-file-size set to 10 Mb and max-size-bytes set 10 Mb, with a address policy
of BLOCK. When my sendMailQueue reaches its limit a deadlock occurs. This is further
emphasized in the JBossAS Forum reference.
Here's a snippet from jstack after HornetQ crashes.
{noformat}
Found one Java-level deadlock:
=============================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x0000000041df8640 (object 0x00000000ea17d6e0, a
java.lang.Object),
which is held by "Thread-873 (HornetQ-client-global-threads-1151364488)"
"Thread-873 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x000000004107c8b8 (object 0x00000000ea17d6d0, a
java.lang.Object),
which is held by "Thread-878 (HornetQ-client-global-threads-1151364488)"
Java stack information for the threads listed above:
===================================================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:461)
- waiting to lock <0x00000000ea17d6e0> (a java.lang.Object)
- locked <0x00000000ea17d6d0> (a java.lang.Object)
at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1294)
at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1238)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:392)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000f0222060> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"Thread-873 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:458)
- waiting to lock <0x00000000ea17d6d0> (a java.lang.Object)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:391)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000ea17d6e0> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Found 1 deadlock.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:18 p.m.
New subject: [JBoss JIRA] (AS7-3919) Invalid PageSize and blocking could lead to collateral issues
[
https://issues.jboss.org/browse/AS7-3919?page=com.atlassian.jira.plugin.s...
]
Jeff Mesnil edited comment on AS7-3919 at 8/3/12 6:16 AM:
----------------------------------------------------------
I have updated the underlying HornetQ issue. I saw no deadlock with the user configuration
and I don't see any relation between the address settings and the deadlock that is
related to the XA wrapper.
was (Author: jmesnil):
I haved update the underlying HornetQ issue. I saw no deadlock with the user
configuration and I don't see any relation between the address settings and the
deadlock that is related to the XA wrapper.
Invalid PageSize and blocking could lead to collateral issues
-------------------------------------------------------------
Key: AS7-3919
URL: https://issues.jboss.org/browse/AS7-3919
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.0.Final
Reporter: Ove Ranheim
Assignee: Andy Taylor
Priority: Minor
Fix For: 7.1.3.Final (EAP)
Attachments: jstack-2012-02-27_08_49.txt, jstack-2012-02-27_09_58_crash.txt,
standalone-musific.xml
Page-size should be ignored on blocking. If the pageSize is configured beyond or the same
as max-size, the constructor will throw an exception what could cause collateral damage
since some state will be left unfilled.
The user has initially described this issue.
I have journal-file-size set to 10 Mb and max-size-bytes set 10 Mb, with a address policy
of BLOCK. When my sendMailQueue reaches its limit a deadlock occurs. This is further
emphasized in the JBossAS Forum reference.
Here's a snippet from jstack after HornetQ crashes.
{noformat}
Found one Java-level deadlock:
=============================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x0000000041df8640 (object 0x00000000ea17d6e0, a
java.lang.Object),
which is held by "Thread-873 (HornetQ-client-global-threads-1151364488)"
"Thread-873 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x000000004107c8b8 (object 0x00000000ea17d6d0, a
java.lang.Object),
which is held by "Thread-878 (HornetQ-client-global-threads-1151364488)"
Java stack information for the threads listed above:
===================================================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:461)
- waiting to lock <0x00000000ea17d6e0> (a java.lang.Object)
- locked <0x00000000ea17d6d0> (a java.lang.Object)
at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1294)
at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1238)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:392)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000f0222060> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"Thread-873 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:458)
- waiting to lock <0x00000000ea17d6d0> (a java.lang.Object)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:391)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000ea17d6e0> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Found 1 deadlock.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:18 p.m.
New subject: [JBoss JIRA] (AS7-3919) Invalid PageSize and blocking could lead to collateral issues
[
https://issues.jboss.org/browse/AS7-3919?page=com.atlassian.jira.plugin.s...
]
Jeff Mesnil commented on AS7-3919:
----------------------------------
I haved update the underlying HornetQ issue. I saw no deadlock with the user configuration
and I don't see any relation between the address settings and the deadlock that is
related to the XA wrapper.
Invalid PageSize and blocking could lead to collateral issues
-------------------------------------------------------------
Key: AS7-3919
URL: https://issues.jboss.org/browse/AS7-3919
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.0.Final
Reporter: Ove Ranheim
Assignee: Andy Taylor
Priority: Minor
Fix For: 7.1.3.Final (EAP)
Attachments: jstack-2012-02-27_08_49.txt, jstack-2012-02-27_09_58_crash.txt,
standalone-musific.xml
Page-size should be ignored on blocking. If the pageSize is configured beyond or the same
as max-size, the constructor will throw an exception what could cause collateral damage
since some state will be left unfilled.
The user has initially described this issue.
I have journal-file-size set to 10 Mb and max-size-bytes set 10 Mb, with a address policy
of BLOCK. When my sendMailQueue reaches its limit a deadlock occurs. This is further
emphasized in the JBossAS Forum reference.
Here's a snippet from jstack after HornetQ crashes.
{noformat}
Found one Java-level deadlock:
=============================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x0000000041df8640 (object 0x00000000ea17d6e0, a
java.lang.Object),
which is held by "Thread-873 (HornetQ-client-global-threads-1151364488)"
"Thread-873 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x000000004107c8b8 (object 0x00000000ea17d6d0, a
java.lang.Object),
which is held by "Thread-878 (HornetQ-client-global-threads-1151364488)"
Java stack information for the threads listed above:
===================================================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:461)
- waiting to lock <0x00000000ea17d6e0> (a java.lang.Object)
- locked <0x00000000ea17d6d0> (a java.lang.Object)
at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1294)
at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1238)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:392)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000f0222060> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"Thread-873 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:458)
- waiting to lock <0x00000000ea17d6d0> (a java.lang.Object)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:391)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000ea17d6e0> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Found 1 deadlock.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:43 p.m.
New subject: [JBoss JIRA] (AS7-3919) Invalid PageSize and blocking could lead to collateral issues
[
https://issues.jboss.org/browse/AS7-3919?page=com.atlassian.jira.plugin.s...
]
Jeff Mesnil resolved AS7-3919.
------------------------------
Assignee: Jeff Mesnil (was: Andy Taylor)
Resolution: Cannot Reproduce Bug
There is no relation betweend the paging configuration and the XAResource wrapper
deadlock.
The deadlock was fixed in https://issues.jboss.org/browse/JBPAPP-8038
Invalid PageSize and blocking could lead to collateral issues
-------------------------------------------------------------
Key: AS7-3919
URL: https://issues.jboss.org/browse/AS7-3919
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.0.Final
Reporter: Ove Ranheim
Assignee: Jeff Mesnil
Priority: Minor
Fix For: 7.1.3.Final (EAP)
Attachments: jstack-2012-02-27_08_49.txt, jstack-2012-02-27_09_58_crash.txt,
standalone-musific.xml
Page-size should be ignored on blocking. If the pageSize is configured beyond or the same
as max-size, the constructor will throw an exception what could cause collateral damage
since some state will be left unfilled.
The user has initially described this issue.
I have journal-file-size set to 10 Mb and max-size-bytes set 10 Mb, with a address policy
of BLOCK. When my sendMailQueue reaches its limit a deadlock occurs. This is further
emphasized in the JBossAS Forum reference.
Here's a snippet from jstack after HornetQ crashes.
{noformat}
Found one Java-level deadlock:
=============================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x0000000041df8640 (object 0x00000000ea17d6e0, a
java.lang.Object),
which is held by "Thread-873 (HornetQ-client-global-threads-1151364488)"
"Thread-873 (HornetQ-client-global-threads-1151364488)":
waiting to lock monitor 0x000000004107c8b8 (object 0x00000000ea17d6d0, a
java.lang.Object),
which is held by "Thread-878 (HornetQ-client-global-threads-1151364488)"
Java stack information for the threads listed above:
===================================================
"Thread-878 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:461)
- waiting to lock <0x00000000ea17d6e0> (a java.lang.Object)
- locked <0x00000000ea17d6d0> (a java.lang.Object)
at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1294)
at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1238)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:392)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000f0222060> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"Thread-873 (HornetQ-client-global-threads-1151364488)":
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:458)
- waiting to lock <0x00000000ea17d6d0> (a java.lang.Object)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:391)
at
org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
- locked <0x00000000ea17d6e0> (a java.lang.Object)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Found 1 deadlock.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
4344
days inactive
4505
days old
18 comments
5 participants
participants (5)
-
Clebert Suconic (JIRA) -
Jason Greene (JIRA)
-
Jeff Mesnil (JIRA)
-
Miroslav Novak (JIRA)
-
Ove Ranheim (JIRA)