[JBoss JIRA] (AS7-5394) MemoryLeak (StackTraceElement objects) in JMS Bridges when the connection times out
10:20 a.m.
Epi Vou created AS7-5394:
----------------------------
Summary: MemoryLeak (StackTraceElement objects) in JMS Bridges when the
connection times out
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP), 7.1.3.Final (EAP)
Reporter: Epi Vou
Assignee: Andy Taylor
A jboss server eventually will start throwing OutOfMemory exceptions when it is configured
to forward jms messages to another server using a bridge, and the network is unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:22 a.m.
[
https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.s...
]
Epi Vou updated AS7-5394:
-------------------------
Attachment: bug.png
jvisualvm on heap dump generated using the -XX:+HeapDumpOnOutOfMemoryError parameter on
the jvm
MemoryLeak (StackTraceElement objects) in JMS Bridges when the
connection times out
-----------------------------------------------------------------------------------
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP), 7.1.3.Final (EAP)
Reporter: Epi Vou
Assignee: Andy Taylor
Attachments: bug.png
A jboss server eventually will start throwing OutOfMemory exceptions when it is
configured to forward jms messages to another server using a bridge, and the network is
unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:28 a.m.
[
https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.s...
]
Jeff Mesnil commented on AS7-5394:
----------------------------------
are you talking about HornetQ core bridge or *JMS* bridge (that are not available in AS7
7.1.x)?
could you attach your messaging configuration please?
MemoryLeak (StackTraceElement objects) in JMS Bridges when the
connection times out
-----------------------------------------------------------------------------------
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP), 7.1.3.Final (EAP)
Reporter: Epi Vou
Assignee: Andy Taylor
Attachments: bug.png
A jboss server eventually will start throwing OutOfMemory exceptions when it is
configured to forward jms messages to another server using a bridge, and the network is
unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:28 a.m.
[
https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.s...
]
Jeff Mesnil commented on AS7-5394:
----------------------------------
do you have logs on either servers about a problem? In your dump, there are more than 5K
instances of HornetQException...
MemoryLeak (StackTraceElement objects) in JMS Bridges when the
connection times out
-----------------------------------------------------------------------------------
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP), 7.1.3.Final (EAP)
Reporter: Epi Vou
Assignee: Andy Taylor
Attachments: bug.png
A jboss server eventually will start throwing OutOfMemory exceptions when it is
configured to forward jms messages to another server using a bridge, and the network is
unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:30 a.m.
[
https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.s...
]
Jeff Mesnil reassigned AS7-5394:
--------------------------------
Assignee: Jeff Mesnil (was: Andy Taylor)
MemoryLeak (StackTraceElement objects) in JMS Bridges when the
connection times out
-----------------------------------------------------------------------------------
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP), 7.1.3.Final (EAP)
Reporter: Epi Vou
Assignee: Jeff Mesnil
Attachments: bug.png
A jboss server eventually will start throwing OutOfMemory exceptions when it is
configured to forward jms messages to another server using a bridge, and the network is
unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:36 a.m.
[
https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.s...
]
Epi Vou commented on AS7-5394:
------------------------------
Here's the messaging configuration :
<subsystem xmlns="urn:jboss:domain:messaging:1.2">
<hornetq-server>
<persistence-enabled>true</persistence-enabled>
<journal-file-size>102400</journal-file-size>
<journal-min-files>2</journal-min-files>
<connectors>
<netty-connector name="netty"
socket-binding="messaging"/>
<netty-connector name="netty-throughput"
socket-binding="messaging-throughput">
<param key="batch-delay" value="50"/>
</netty-connector>
<netty-connector name="jms-remote-connector"
socket-binding="jms-remote"/>
<in-vm-connector name="in-vm"
server-id="0"/>
</connectors>
<acceptors>
<netty-acceptor name="netty"
socket-binding="messaging"/>
<netty-acceptor name="netty-throughput"
socket-binding="messaging-throughput">
<param key="batch-delay" value="50"/>
<param key="direct-deliver"
value="false"/>
</netty-acceptor>
<in-vm-acceptor name="in-vm"
server-id="0"/>
</acceptors>
<diverts>
<divert name="proxy-divert">
<routing-name>proxy-divert</routing-name>
<address>jms.topic.livescoresTopic</address>
<forwarding-address>jms.queue.proxy-queue</forwarding-address>
<filter string="filter = 'match_update' OR filter
= 'odds_aagregate_update' OR filter = 'backend_push_event'"/>
<exclusive>false</exclusive>
</divert>
</diverts>
<bridges>
<bridge name="jms-proxy-bridge">
<queue-name>jms.queue.proxy-queue</queue-name>
<forwarding-address>jms.topic.livescoresTopic</forwarding-address>
<ha>false</ha>
<reconnect-attempts>9999999</reconnect-attempts>
<failover-on-server-shutdown>false</failover-on-server-shutdown>
<use-duplicate-detection>false</use-duplicate-detection>
<user>username</user>
<password>password</password>
<static-connectors>
<connector-ref>
jms-remote-connector
</connector-ref>
</static-connectors>
</bridge>
</bridges>
<security-settings>
<security-setting match="#">
<permission type="send" roles="guest"/>
<permission type="consume"
roles="guest"/>
<permission type="createNonDurableQueue"
roles="guest"/>
<permission type="deleteNonDurableQueue"
roles="guest"/>
</security-setting>
</security-settings>
<address-settings>
<address-setting match="#">
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<max-size-bytes>10485760</max-size-bytes>
<address-full-policy>BLOCK</address-full-policy>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
</address-setting>
</address-settings>
<jms-connection-factories>
<connection-factory name="InVmConnectionFactory">
<connectors>
<connector-ref connector-name="in-vm"/>
</connectors>
<entries>
<entry name="java:/ConnectionFactory"/>
</entries>
</connection-factory>
<connection-factory name="RemoteConnectionFactory">
<connectors>
<connector-ref connector-name="netty"/>
</connectors>
<entries>
<entry
name="java:jboss/exported/jms/RemoteConnectionFactory"/>
</entries>
</connection-factory>
<pooled-connection-factory name="hornetq-ra">
<transaction mode="xa"/>
<connectors>
<connector-ref connector-name="in-vm"/>
</connectors>
<entries>
<entry name="java:/JmsXA"/>
</entries>
</pooled-connection-factory>
</jms-connection-factories>
<jms-destinations>
<jms-queue name="fileQ">
<entry name="/queue/enetpulse/fileQ"/>
</jms-queue>
<jms-queue name="processingQ">
<entry name="/queue/enetpulse/processingQ"/>
</jms-queue>
<jms-queue name="inCompleteDataQ">
<entry name="/queue/enetpulse/inCompleteDataQ"/>
</jms-queue>
<jms-queue name="persistenceQ">
<entry name="/queue/enetpulse/persistenceQ"/>
</jms-queue>
<jms-queue name="DLQ">
<entry name="/queue/DLQ"/>
</jms-queue>
<jms-queue name="pushNotificationQ">
<entry name="/queue/pushNotificationQ"/>
</jms-queue>
<jms-queue name="proxy-queue">
<entry name="java:/queue/proxy"/>
<durable>true</durable>
</jms-queue>
<jms-topic name="livescoresTopic">
<entry name="topic/livescores"/>
</jms-topic>
</jms-destinations>
</hornetq-server>
</subsystem>
MemoryLeak (StackTraceElement objects) in JMS Bridges when the
connection times out
-----------------------------------------------------------------------------------
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP), 7.1.3.Final (EAP)
Reporter: Epi Vou
Assignee: Jeff Mesnil
Attachments: bug.png
A jboss server eventually will start throwing OutOfMemory exceptions when it is
configured to forward jms messages to another server using a bridge, and the network is
unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:44 a.m.
[
https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.s...
]
Jeff Mesnil commented on AS7-5394:
----------------------------------
your conf looks ok.
you divert (non-exclusively) messages sent to the jms.topic.livescoresTopic to the
jms.queue.proxy-queue.
this address is bound to the jms.queue.proxy-queue that is bridged to the
jms.topic.livescoresTopic on the *other* server using the jms-remote-connector (I assume
its socket-binding is correct and points to the other server).
Are you able to consume messages on the other server before blocking the JMS traffic?
Do you have errors in your logs?
MemoryLeak (StackTraceElement objects) in JMS Bridges when the
connection times out
-----------------------------------------------------------------------------------
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP), 7.1.3.Final (EAP)
Reporter: Epi Vou
Assignee: Jeff Mesnil
Attachments: bug.png
A jboss server eventually will start throwing OutOfMemory exceptions when it is
configured to forward jms messages to another server using a bridge, and the network is
unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:07 a.m.
[
https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.s...
]
Epi Vou commented on AS7-5394:
------------------------------
The bridge as I understand is a hornetQ bridge, which I have configured through the web
management interface of jboss. I use a queue, a divert which diverts messages going to a
topic to this queue, and a bridge that forwards messages from the queue on the other
jboss. Everything works as expected for some time. Messages are produced on the server
containing the bridge and are consumed by the other server. If the network drops, the
servers will output some warnings, and any messages produced during this time, will be
consumed by the other server as soon as the network works again.
at network disconnection the server that has the bridge configured outputs the following
:
{noformat}
18:54:51,211 ERROR [org.hornetq.core.remoting.impl.netty.NettyConnector] (Thread-7
(HornetQ-server-HornetQServerImpl::serverUUID=df86d314-e7b4-11e1-9714-895d978cba07-229910155))
Failed to create netty connection: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.6.0_33]
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) [rt.jar:1.6.0_33]
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
[rt.jar:1.6.0_33]
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) [rt.jar:1.6.0_33]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) [rt.jar:1.6.0_33]
at java.net.Socket.connect(Socket.java:529) [rt.jar:1.6.0_33]
at
org.jboss.netty.channel.socket.oio.OioClientSocketPipelineSink.connect(OioClientSocketPipelineSink.java:114)
[netty-3.2.6.Final.jar:]
at
org.jboss.netty.channel.socket.oio.OioClientSocketPipelineSink.eventSunk(OioClientSocketPipelineSink.java:74)
[netty-3.2.6.Final.jar:]
at org.jboss.netty.channel.Channels.connect(Channels.java:541)
[netty-3.2.6.Final.jar:]
at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:210)
[netty-3.2.6.Final.jar:]
at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:227)
[netty-3.2.6.Final.jar:]
at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:188)
[netty-3.2.6.Final.jar:]
at
org.hornetq.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:499)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1144)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:994)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:225)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:752)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.server.cluster.impl.BridgeImpl.createSessionFactory(BridgeImpl.java:789)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at org.hornetq.core.server.cluster.impl.BridgeImpl.connect(BridgeImpl.java:805)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.server.cluster.impl.BridgeImpl$ConnectRunnable.run(BridgeImpl.java:1089)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_33]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_33]
{noformat}
and the receiving jboss server outputs the following :
{noformat}
[19:02:04,284 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
(hornetq-failure-check-thread) Connection failure has been detected: Did not receive data
from /194.219.97.67:38329. It is likely the client has exited or crashed without closing
its connection, or the network between the server and client has failed. You also might
have configured connection-ttl and client-failure-check-period incorrectly. Please check
user manual for more information. The connection will now be closed. [code=3]
19:02:04,284 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Client connection failed, clearing up resources for session
76d40991-e884-11e1-8081-df4631c082f3
19:02:04,284 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Cleared up resources for session
76d40991-e884-11e1-8081-df4631c082f3
{noformat}
But after a few disconnects the log on the server that has the bridge configured get's
filled up with the following exceptions :
{noformat}
15:37:11,393 WARN [org.hornetq.core.server.cluster.impl.BridgeImpl] (Thread-1
(HornetQ-server-HornetQServerImpl::serverUUID=df86d314-e
7b4-11e1-9714-895d978cba07-1698482702)) Unable to send message
Reference[19327439667]:RELIABLE:ServerMessage[messageID=19327439667,priority=4,
bodySize=1500,expiration=0, durable=true,
address=null,properties=TypedProperties[{_HQ_ORIG_MESSAGE_ID=19327439666,
_HQ_ORIG_ADDRESS=jms.topic.livescoresTopic, filter=backend_push_event}]]@737247918, will
try again once bridge reconnects: HornetQException[errorCode=102 message=Producer is
closed]
at
org.hornetq.core.client.impl.ClientProducerImpl.checkClosed(ClientProducerImpl.java:317)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:140)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.server.cluster.impl.BridgeImpl.deliverStandardMessage(BridgeImpl.java:698)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at org.hornetq.core.server.cluster.impl.BridgeImpl.handle(BridgeImpl.java:584)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at org.hornetq.core.server.impl.QueueImpl.handle(QueueImpl.java:2208)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:1759)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at org.hornetq.core.server.impl.QueueImpl.access$1100(QueueImpl.java:77)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at org.hornetq.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2455)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_33]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_33]
{noformat}
MemoryLeak (StackTraceElement objects) in JMS Bridges when the
connection times out
-----------------------------------------------------------------------------------
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP), 7.1.3.Final (EAP)
Reporter: Epi Vou
Assignee: Jeff Mesnil
Attachments: bug.png
A jboss server eventually will start throwing OutOfMemory exceptions when it is
configured to forward jms messages to another server using a bridge, and the network is
unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:07 a.m.
[
https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.s...
]
Epi Vou edited comment on AS7-5394 at 8/17/12 12:06 PM:
--------------------------------------------------------
Here's the messaging configuration :
{noformat}
<subsystem xmlns="urn:jboss:domain:messaging:1.2">
<hornetq-server>
<persistence-enabled>true</persistence-enabled>
<journal-file-size>102400</journal-file-size>
<journal-min-files>2</journal-min-files>
<connectors>
<netty-connector name="netty"
socket-binding="messaging"/>
<netty-connector name="netty-throughput"
socket-binding="messaging-throughput">
<param key="batch-delay" value="50"/>
</netty-connector>
<netty-connector name="jms-remote-connector"
socket-binding="jms-remote"/>
<in-vm-connector name="in-vm"
server-id="0"/>
</connectors>
<acceptors>
<netty-acceptor name="netty"
socket-binding="messaging"/>
<netty-acceptor name="netty-throughput"
socket-binding="messaging-throughput">
<param key="batch-delay" value="50"/>
<param key="direct-deliver"
value="false"/>
</netty-acceptor>
<in-vm-acceptor name="in-vm"
server-id="0"/>
</acceptors>
<diverts>
<divert name="proxy-divert">
<routing-name>proxy-divert</routing-name>
<address>jms.topic.livescoresTopic</address>
<forwarding-address>jms.queue.proxy-queue</forwarding-address>
<filter string="filter = 'match_update' OR filter
= 'odds_aagregate_update' OR filter = 'backend_push_event'"/>
<exclusive>false</exclusive>
</divert>
</diverts>
<bridges>
<bridge name="jms-proxy-bridge">
<queue-name>jms.queue.proxy-queue</queue-name>
<forwarding-address>jms.topic.livescoresTopic</forwarding-address>
<ha>false</ha>
<reconnect-attempts>9999999</reconnect-attempts>
<failover-on-server-shutdown>false</failover-on-server-shutdown>
<use-duplicate-detection>false</use-duplicate-detection>
<user>username</user>
<password>password</password>
<static-connectors>
<connector-ref>
jms-remote-connector
</connector-ref>
</static-connectors>
</bridge>
</bridges>
<security-settings>
<security-setting match="#">
<permission type="send" roles="guest"/>
<permission type="consume"
roles="guest"/>
<permission type="createNonDurableQueue"
roles="guest"/>
<permission type="deleteNonDurableQueue"
roles="guest"/>
</security-setting>
</security-settings>
<address-settings>
<address-setting match="#">
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<max-size-bytes>10485760</max-size-bytes>
<address-full-policy>BLOCK</address-full-policy>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
</address-setting>
</address-settings>
<jms-connection-factories>
<connection-factory name="InVmConnectionFactory">
<connectors>
<connector-ref connector-name="in-vm"/>
</connectors>
<entries>
<entry name="java:/ConnectionFactory"/>
</entries>
</connection-factory>
<connection-factory name="RemoteConnectionFactory">
<connectors>
<connector-ref connector-name="netty"/>
</connectors>
<entries>
<entry
name="java:jboss/exported/jms/RemoteConnectionFactory"/>
</entries>
</connection-factory>
<pooled-connection-factory name="hornetq-ra">
<transaction mode="xa"/>
<connectors>
<connector-ref connector-name="in-vm"/>
</connectors>
<entries>
<entry name="java:/JmsXA"/>
</entries>
</pooled-connection-factory>
</jms-connection-factories>
<jms-destinations>
<jms-queue name="fileQ">
<entry name="/queue/enetpulse/fileQ"/>
</jms-queue>
<jms-queue name="processingQ">
<entry name="/queue/enetpulse/processingQ"/>
</jms-queue>
<jms-queue name="inCompleteDataQ">
<entry name="/queue/enetpulse/inCompleteDataQ"/>
</jms-queue>
<jms-queue name="persistenceQ">
<entry name="/queue/enetpulse/persistenceQ"/>
</jms-queue>
<jms-queue name="DLQ">
<entry name="/queue/DLQ"/>
</jms-queue>
<jms-queue name="pushNotificationQ">
<entry name="/queue/pushNotificationQ"/>
</jms-queue>
<jms-queue name="proxy-queue">
<entry name="java:/queue/proxy"/>
<durable>true</durable>
</jms-queue>
<jms-topic name="livescoresTopic">
<entry name="topic/livescores"/>
</jms-topic>
</jms-destinations>
</hornetq-server>
</subsystem>
{noformat}
was (Author: pimenas_black):
Here's the messaging configuration :
<subsystem xmlns="urn:jboss:domain:messaging:1.2">
<hornetq-server>
<persistence-enabled>true</persistence-enabled>
<journal-file-size>102400</journal-file-size>
<journal-min-files>2</journal-min-files>
<connectors>
<netty-connector name="netty"
socket-binding="messaging"/>
<netty-connector name="netty-throughput"
socket-binding="messaging-throughput">
<param key="batch-delay" value="50"/>
</netty-connector>
<netty-connector name="jms-remote-connector"
socket-binding="jms-remote"/>
<in-vm-connector name="in-vm"
server-id="0"/>
</connectors>
<acceptors>
<netty-acceptor name="netty"
socket-binding="messaging"/>
<netty-acceptor name="netty-throughput"
socket-binding="messaging-throughput">
<param key="batch-delay" value="50"/>
<param key="direct-deliver"
value="false"/>
</netty-acceptor>
<in-vm-acceptor name="in-vm"
server-id="0"/>
</acceptors>
<diverts>
<divert name="proxy-divert">
<routing-name>proxy-divert</routing-name>
<address>jms.topic.livescoresTopic</address>
<forwarding-address>jms.queue.proxy-queue</forwarding-address>
<filter string="filter = 'match_update' OR filter
= 'odds_aagregate_update' OR filter = 'backend_push_event'"/>
<exclusive>false</exclusive>
</divert>
</diverts>
<bridges>
<bridge name="jms-proxy-bridge">
<queue-name>jms.queue.proxy-queue</queue-name>
<forwarding-address>jms.topic.livescoresTopic</forwarding-address>
<ha>false</ha>
<reconnect-attempts>9999999</reconnect-attempts>
<failover-on-server-shutdown>false</failover-on-server-shutdown>
<use-duplicate-detection>false</use-duplicate-detection>
<user>username</user>
<password>password</password>
<static-connectors>
<connector-ref>
jms-remote-connector
</connector-ref>
</static-connectors>
</bridge>
</bridges>
<security-settings>
<security-setting match="#">
<permission type="send" roles="guest"/>
<permission type="consume"
roles="guest"/>
<permission type="createNonDurableQueue"
roles="guest"/>
<permission type="deleteNonDurableQueue"
roles="guest"/>
</security-setting>
</security-settings>
<address-settings>
<address-setting match="#">
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<max-size-bytes>10485760</max-size-bytes>
<address-full-policy>BLOCK</address-full-policy>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
</address-setting>
</address-settings>
<jms-connection-factories>
<connection-factory name="InVmConnectionFactory">
<connectors>
<connector-ref connector-name="in-vm"/>
</connectors>
<entries>
<entry name="java:/ConnectionFactory"/>
</entries>
</connection-factory>
<connection-factory name="RemoteConnectionFactory">
<connectors>
<connector-ref connector-name="netty"/>
</connectors>
<entries>
<entry
name="java:jboss/exported/jms/RemoteConnectionFactory"/>
</entries>
</connection-factory>
<pooled-connection-factory name="hornetq-ra">
<transaction mode="xa"/>
<connectors>
<connector-ref connector-name="in-vm"/>
</connectors>
<entries>
<entry name="java:/JmsXA"/>
</entries>
</pooled-connection-factory>
</jms-connection-factories>
<jms-destinations>
<jms-queue name="fileQ">
<entry name="/queue/enetpulse/fileQ"/>
</jms-queue>
<jms-queue name="processingQ">
<entry name="/queue/enetpulse/processingQ"/>
</jms-queue>
<jms-queue name="inCompleteDataQ">
<entry name="/queue/enetpulse/inCompleteDataQ"/>
</jms-queue>
<jms-queue name="persistenceQ">
<entry name="/queue/enetpulse/persistenceQ"/>
</jms-queue>
<jms-queue name="DLQ">
<entry name="/queue/DLQ"/>
</jms-queue>
<jms-queue name="pushNotificationQ">
<entry name="/queue/pushNotificationQ"/>
</jms-queue>
<jms-queue name="proxy-queue">
<entry name="java:/queue/proxy"/>
<durable>true</durable>
</jms-queue>
<jms-topic name="livescoresTopic">
<entry name="topic/livescores"/>
</jms-topic>
</jms-destinations>
</hornetq-server>
</subsystem>
MemoryLeak (StackTraceElement objects) in JMS Bridges when the
connection times out
-----------------------------------------------------------------------------------
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP), 7.1.3.Final (EAP)
Reporter: Epi Vou
Assignee: Jeff Mesnil
Attachments: bug.png
A jboss server eventually will start throwing OutOfMemory exceptions when it is
configured to forward jms messages to another server using a bridge, and the network is
unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:22 a.m.
[
https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.s...
]
Epi Vou edited comment on AS7-5394 at 8/17/12 12:21 PM:
--------------------------------------------------------
The bridge as I understand is a hornetQ bridge, which I have configured through the web
management interface of jboss. I use a queue, a divert which diverts messages going to a
topic to this queue, and a bridge that forwards messages from the queue on the other
jboss. Everything works as expected for some time. Messages are produced on the server
containing the bridge and are consumed by the other server. If the network drops, the
servers will output some warnings, and any messages produced during this time, will be
consumed by the other server as soon as the network works again.
at network disconnection the server that has the bridge configured outputs the following
:
{noformat}
18:54:51,211 ERROR [org.hornetq.core.remoting.impl.netty.NettyConnector] (Thread-7
(HornetQ-server-HornetQServerImpl::serverUUID=df86d314-e7b4-11e1-9714-895d978cba07-229910155))
Failed to create netty connection: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.6.0_33]
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) [rt.jar:1.6.0_33]
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
[rt.jar:1.6.0_33]
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) [rt.jar:1.6.0_33]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) [rt.jar:1.6.0_33]
at java.net.Socket.connect(Socket.java:529) [rt.jar:1.6.0_33]
at
org.jboss.netty.channel.socket.oio.OioClientSocketPipelineSink.connect(OioClientSocketPipelineSink.java:114)
[netty-3.2.6.Final.jar:]
at
org.jboss.netty.channel.socket.oio.OioClientSocketPipelineSink.eventSunk(OioClientSocketPipelineSink.java:74)
[netty-3.2.6.Final.jar:]
at org.jboss.netty.channel.Channels.connect(Channels.java:541)
[netty-3.2.6.Final.jar:]
at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:210)
[netty-3.2.6.Final.jar:]
at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:227)
[netty-3.2.6.Final.jar:]
at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:188)
[netty-3.2.6.Final.jar:]
at
org.hornetq.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:499)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1144)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:994)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:225)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:752)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.server.cluster.impl.BridgeImpl.createSessionFactory(BridgeImpl.java:789)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at org.hornetq.core.server.cluster.impl.BridgeImpl.connect(BridgeImpl.java:805)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.server.cluster.impl.BridgeImpl$ConnectRunnable.run(BridgeImpl.java:1089)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_33]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_33]
{noformat}
and the receiving jboss server outputs the following :
{noformat}
[19:02:04,284 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
(hornetq-failure-check-thread) Connection failure has been detected: Did not receive data
from /194.219.97.67:38329. It is likely the client has exited or crashed without closing
its connection, or the network between the server and client has failed. You also might
have configured connection-ttl and client-failure-check-period incorrectly. Please check
user manual for more information. The connection will now be closed. [code=3]
19:02:04,284 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Client connection failed, clearing up resources for session
76d40991-e884-11e1-8081-df4631c082f3
19:02:04,284 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Cleared up resources for session
76d40991-e884-11e1-8081-df4631c082f3
{noformat}
But after a few disconnects the log on the server that has the bridge configured get's
filled up with the following exceptions :
{noformat}
15:37:11,393 WARN [org.hornetq.core.server.cluster.impl.BridgeImpl] (Thread-1
(HornetQ-server-HornetQServerImpl::serverUUID=df86d314-e
7b4-11e1-9714-895d978cba07-1698482702)) Unable to send message
Reference[19327439667]:RELIABLE:ServerMessage[messageID=19327439667,priority=4,
bodySize=1500,expiration=0, durable=true,
address=null,properties=TypedProperties[{_HQ_ORIG_MESSAGE_ID=19327439666,
_HQ_ORIG_ADDRESS=jms.topic.livescoresTopic, filter=backend_push_event}]]@737247918, will
try again once bridge reconnects: HornetQException[errorCode=102 message=Producer is
closed]
at
org.hornetq.core.client.impl.ClientProducerImpl.checkClosed(ClientProducerImpl.java:317)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:140)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.server.cluster.impl.BridgeImpl.deliverStandardMessage(BridgeImpl.java:698)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at org.hornetq.core.server.cluster.impl.BridgeImpl.handle(BridgeImpl.java:584)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at org.hornetq.core.server.impl.QueueImpl.handle(QueueImpl.java:2208)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:1759)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at org.hornetq.core.server.impl.QueueImpl.access$1100(QueueImpl.java:77)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at org.hornetq.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2455)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_33]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_33]
{noformat}
And after a few minutes the server becomes unresponsive and out of memory...
was (Author: pimenas_black):
The bridge as I understand is a hornetQ bridge, which I have configured through the
web management interface of jboss. I use a queue, a divert which diverts messages going to
a topic to this queue, and a bridge that forwards messages from the queue on the other
jboss. Everything works as expected for some time. Messages are produced on the server
containing the bridge and are consumed by the other server. If the network drops, the
servers will output some warnings, and any messages produced during this time, will be
consumed by the other server as soon as the network works again.
at network disconnection the server that has the bridge configured outputs the following
:
{noformat}
18:54:51,211 ERROR [org.hornetq.core.remoting.impl.netty.NettyConnector] (Thread-7
(HornetQ-server-HornetQServerImpl::serverUUID=df86d314-e7b4-11e1-9714-895d978cba07-229910155))
Failed to create netty connection: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.6.0_33]
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) [rt.jar:1.6.0_33]
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
[rt.jar:1.6.0_33]
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) [rt.jar:1.6.0_33]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) [rt.jar:1.6.0_33]
at java.net.Socket.connect(Socket.java:529) [rt.jar:1.6.0_33]
at
org.jboss.netty.channel.socket.oio.OioClientSocketPipelineSink.connect(OioClientSocketPipelineSink.java:114)
[netty-3.2.6.Final.jar:]
at
org.jboss.netty.channel.socket.oio.OioClientSocketPipelineSink.eventSunk(OioClientSocketPipelineSink.java:74)
[netty-3.2.6.Final.jar:]
at org.jboss.netty.channel.Channels.connect(Channels.java:541)
[netty-3.2.6.Final.jar:]
at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:210)
[netty-3.2.6.Final.jar:]
at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:227)
[netty-3.2.6.Final.jar:]
at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:188)
[netty-3.2.6.Final.jar:]
at
org.hornetq.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:499)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1144)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:994)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:225)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:752)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.server.cluster.impl.BridgeImpl.createSessionFactory(BridgeImpl.java:789)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at org.hornetq.core.server.cluster.impl.BridgeImpl.connect(BridgeImpl.java:805)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.server.cluster.impl.BridgeImpl$ConnectRunnable.run(BridgeImpl.java:1089)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_33]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_33]
{noformat}
and the receiving jboss server outputs the following :
{noformat}
[19:02:04,284 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
(hornetq-failure-check-thread) Connection failure has been detected: Did not receive data
from /194.219.97.67:38329. It is likely the client has exited or crashed without closing
its connection, or the network between the server and client has failed. You also might
have configured connection-ttl and client-failure-check-period incorrectly. Please check
user manual for more information. The connection will now be closed. [code=3]
19:02:04,284 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Client connection failed, clearing up resources for session
76d40991-e884-11e1-8081-df4631c082f3
19:02:04,284 WARN [org.hornetq.core.server.impl.ServerSessionImpl]
(hornetq-failure-check-thread) Cleared up resources for session
76d40991-e884-11e1-8081-df4631c082f3
{noformat}
But after a few disconnects the log on the server that has the bridge configured get's
filled up with the following exceptions :
{noformat}
15:37:11,393 WARN [org.hornetq.core.server.cluster.impl.BridgeImpl] (Thread-1
(HornetQ-server-HornetQServerImpl::serverUUID=df86d314-e
7b4-11e1-9714-895d978cba07-1698482702)) Unable to send message
Reference[19327439667]:RELIABLE:ServerMessage[messageID=19327439667,priority=4,
bodySize=1500,expiration=0, durable=true,
address=null,properties=TypedProperties[{_HQ_ORIG_MESSAGE_ID=19327439666,
_HQ_ORIG_ADDRESS=jms.topic.livescoresTopic, filter=backend_push_event}]]@737247918, will
try again once bridge reconnects: HornetQException[errorCode=102 message=Producer is
closed]
at
org.hornetq.core.client.impl.ClientProducerImpl.checkClosed(ClientProducerImpl.java:317)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:140)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.core.server.cluster.impl.BridgeImpl.deliverStandardMessage(BridgeImpl.java:698)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at org.hornetq.core.server.cluster.impl.BridgeImpl.handle(BridgeImpl.java:584)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at org.hornetq.core.server.impl.QueueImpl.handle(QueueImpl.java:2208)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:1759)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at org.hornetq.core.server.impl.QueueImpl.access$1100(QueueImpl.java:77)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at org.hornetq.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2455)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
[hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_33]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_33]
{noformat}
MemoryLeak (StackTraceElement objects) in JMS Bridges when the
connection times out
-----------------------------------------------------------------------------------
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP), 7.1.3.Final (EAP)
Reporter: Epi Vou
Assignee: Jeff Mesnil
Attachments: bug.png
A jboss server eventually will start throwing OutOfMemory exceptions when it is
configured to forward jms messages to another server using a bridge, and the network is
unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:01 a.m.
New subject: [JBoss JIRA] (AS7-5394) MemoryLeak in core bridges when the connection times out
[
https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.s...
]
Jeff Mesnil updated AS7-5394:
-----------------------------
Summary: MemoryLeak in core bridges when the connection times out (was: MemoryLeak
(StackTraceElement objects) in JMS Bridges when the connection times out)
MemoryLeak in core bridges when the connection times out
--------------------------------------------------------
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP), 7.1.3.Final (EAP)
Reporter: Epi Vou
Assignee: Jeff Mesnil
Attachments: bug.png
A jboss server eventually will start throwing OutOfMemory exceptions when it is
configured to forward jms messages to another server using a bridge, and the network is
unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:03 a.m.
New subject: [JBoss JIRA] (AS7-5394) MemoryLeak in core bridges when the connection times out
[
https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.s...
]
Jeff Mesnil updated AS7-5394:
-----------------------------
Affects Version/s: (was: 7.1.3.Final (EAP))
MemoryLeak in core bridges when the connection times out
--------------------------------------------------------
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP)
Reporter: Epi Vou
Assignee: Jeff Mesnil
Attachments: bug.png
A jboss server eventually will start throwing OutOfMemory exceptions when it is
configured to forward jms messages to another server using a bridge, and the network is
unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:41 a.m.
New subject: [JBoss JIRA] (AS7-5394) MemoryLeak in core bridges when the connection times out
[
https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.s...
]
Jeff Mesnil commented on AS7-5394:
----------------------------------
could you attach the logs of the 2 servers so that I can have a better idea of what's
going on and leads to the OOME?
MemoryLeak in core bridges when the connection times out
--------------------------------------------------------
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP)
Reporter: Epi Vou
Assignee: Jeff Mesnil
Attachments: bug.png
A jboss server eventually will start throwing OutOfMemory exceptions when it is
configured to forward jms messages to another server using a bridge, and the network is
unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:35 a.m.
New subject: [JBoss JIRA] (AS7-5394) MemoryLeak in core bridges when the connection times out
[
https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.s...
]
Epi Vou updated AS7-5394:
-------------------------
Attachment: jboss1-logs.tar.gz
this is the server that contains the bridge and produces the messages. The logs are from
boot to outofmemory exception where it get's killed. The test.log contains log
messages from a test MDB which displays the messages that should be sent through the
bridge
MemoryLeak in core bridges when the connection times out
--------------------------------------------------------
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP)
Reporter: Epi Vou
Assignee: Jeff Mesnil
Attachments: bug.png, jboss1-logs.tar.gz
A jboss server eventually will start throwing OutOfMemory exceptions when it is
configured to forward jms messages to another server using a bridge, and the network is
unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:37 a.m.
New subject: [JBoss JIRA] (AS7-5394) MemoryLeak in core bridges when the connection times out
[
https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.s...
]
Epi Vou updated AS7-5394:
-------------------------
Attachment: jboss2-logs.tar.gz
this is the second server which consumes the messages that are being sent over the
bridge.
The test log contains the logs of the messages that were consumed.
MemoryLeak in core bridges when the connection times out
--------------------------------------------------------
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP)
Reporter: Epi Vou
Assignee: Jeff Mesnil
Attachments: bug.png, jboss1-logs.tar.gz, jboss2-logs.tar.gz
A jboss server eventually will start throwing OutOfMemory exceptions when it is
configured to forward jms messages to another server using a bridge, and the network is
unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:39 a.m.
New subject: [JBoss JIRA] (AS7-5394) MemoryLeak in core bridges when the connection times out
[
https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.s...
]
Epi Vou commented on AS7-5394:
------------------------------
I just uploaded a set of logs I produced yesterday. I also have the corresponding heap
dump, if you need to take a look. I can upload it somewhere, since it's 107MB zipped.
MemoryLeak in core bridges when the connection times out
--------------------------------------------------------
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP)
Reporter: Epi Vou
Assignee: Jeff Mesnil
Attachments: bug.png, jboss1-logs.tar.gz, jboss2-logs.tar.gz
A jboss server eventually will start throwing OutOfMemory exceptions when it is
configured to forward jms messages to another server using a bridge, and the network is
unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:02 p.m.
New subject: [JBoss JIRA] (AS7-5394) MemoryLeak in core bridges when the connection times out
[
https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.s...
]
Clebert Suconic commented on AS7-5394:
--------------------------------------
It's memory leaking or the messages are building up as nobody is consuming them? (i.e.
the bridge is inactive?)
MemoryLeak in core bridges when the connection times out
--------------------------------------------------------
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP)
Reporter: Epi Vou
Assignee: Jeff Mesnil
Attachments: bug.png, jboss1-logs.tar.gz, jboss2-logs.tar.gz
A jboss server eventually will start throwing OutOfMemory exceptions when it is
configured to forward jms messages to another server using a bridge, and the network is
unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:15 a.m.
New subject: [JBoss JIRA] (AS7-5394) MemoryLeak in core bridges when the connection times out
[
https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.s...
]
Epi Vou commented on AS7-5394:
------------------------------
Hi Clebert,
It's not the messages building up, since it's not more than 200 small objects that
are being produced through the whole test. Also if you tak a look at the heap dump, the
leak seams to be coming from BridgeImpl objects, or StackTraceElement[] objects...
MemoryLeak in core bridges when the connection times out
--------------------------------------------------------
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP)
Reporter: Epi Vou
Assignee: Jeff Mesnil
Attachments: bug.png, jboss1-logs.tar.gz, jboss2-logs.tar.gz
A jboss server eventually will start throwing OutOfMemory exceptions when it is
configured to forward jms messages to another server using a bridge, and the network is
unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:20 a.m.
New subject: [JBoss JIRA] (AS7-5394) MemoryLeak in core bridges when the connection times out
[
https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.s...
]
Epi Vou commented on AS7-5394:
------------------------------
From the stacktrace printed on the logs, it seems that the bug has
something to do with this code :
{code}
try
{
producer.send(dest, message);
}
catch (final HornetQException e)
{
log.warn("Unable to send message " + ref + ", will try again once
bridge reconnects", e);
// We remove this reference as we are returning busy which means the reference
will never leave the Queue.
// because of this we have to remove the reference here
refs.remove(ref);
executor.execute(new Runnable()
{
public void run()
{
connectionFailed(e, false);
}
});
return HandleStatus.BUSY;
}
{code}
It seems to me like the executor never runs the code, and this must be the reason why all
these StackTraceElements pile up in the heap.
So this must be clearly a hornetq issue, maybe it should be better if we reproduce it on
standalone hornetq server and post it on the hornetq project issues....
MemoryLeak in core bridges when the connection times out
--------------------------------------------------------
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP)
Reporter: Epi Vou
Assignee: Jeff Mesnil
Attachments: bug.png, jboss1-logs.tar.gz, jboss2-logs.tar.gz
A jboss server eventually will start throwing OutOfMemory exceptions when it is
configured to forward jms messages to another server using a bridge, and the network is
unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:27 p.m.
New subject: [JBoss JIRA] (AS7-5394) MemoryLeak in core bridges when the connection times out
[
https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.s...
]
Brian Stansberry commented on AS7-5394:
---------------------------------------
I'm not sure what JIRA permissions are required to move an AS7 issue to the HORNETQ
project, but if you all want it moved and don't have the needed permissions, ping me
and I'll move it.
That sounds better than opening a new issue since this one already has so much data.
MemoryLeak in core bridges when the connection times out
--------------------------------------------------------
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP)
Reporter: Epi Vou
Assignee: Jeff Mesnil
Attachments: bug.png, jboss1-logs.tar.gz, jboss2-logs.tar.gz
A jboss server eventually will start throwing OutOfMemory exceptions when it is
configured to forward jms messages to another server using a bridge, and the network is
unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:09 a.m.
New subject: [JBoss JIRA] (AS7-5394) MemoryLeak in core bridges when the connection times out
[
https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.s...
]
Epi Vou commented on AS7-5394:
------------------------------
Hello again,
Yesterday I made a change on BridgeImpl.java at the part of the code that I have
referenced above, so that the connectionFailed() routine is not called by the executor,
but in the same thread handling the exception. I run the test the whole day and the issue
seems resolved. However I believe someone with deeper knowledge of HornetQ should have a
look at this, I guess there is a reason why the connectionFailed() routine is called
through an executor...
MemoryLeak in core bridges when the connection times out
--------------------------------------------------------
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP)
Reporter: Epi Vou
Assignee: Jeff Mesnil
Attachments: bug.png, jboss1-logs.tar.gz, jboss2-logs.tar.gz
A jboss server eventually will start throwing OutOfMemory exceptions when it is
configured to forward jms messages to another server using a bridge, and the network is
unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:13 a.m.
New subject: [JBoss JIRA] (AS7-5394) MemoryLeak in core bridges when the connection times out
[
https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.s...
]
Epi Vou updated AS7-5394:
-------------------------
Attachment: BridgeImpl.java
This is the "fixed" BridgeImpl.java which I used to workaround this issue.
HornetQ version used is 2.2.19.Final which I checked out from the corresponding tag.
MemoryLeak in core bridges when the connection times out
--------------------------------------------------------
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP)
Reporter: Epi Vou
Assignee: Jeff Mesnil
Attachments: BridgeImpl.java, bug.png, jboss1-logs.tar.gz, jboss2-logs.tar.gz
A jboss server eventually will start throwing OutOfMemory exceptions when it is
configured to forward jms messages to another server using a bridge, and the network is
unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:17 a.m.
New subject: [JBoss JIRA] (AS7-5394) MemoryLeak in core bridges when the connection times out
[
https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.s...
]
Epi Vou commented on AS7-5394:
------------------------------
Brian, I believe this should be moved to hornetq project, as you suggested. I used the
edit function to try to do it, but I didn't see any option to move it to another
project. So please move it when you can.
MemoryLeak in core bridges when the connection times out
--------------------------------------------------------
Key: AS7-5394
URL: https://issues.jboss.org/browse/AS7-5394
Project: Application Server 7
Issue Type: Bug
Components: JMS
Affects Versions: 7.1.2.Final (EAP)
Reporter: Epi Vou
Assignee: Jeff Mesnil
Attachments: BridgeImpl.java, bug.png, jboss1-logs.tar.gz, jboss2-logs.tar.gz
A jboss server eventually will start throwing OutOfMemory exceptions when it is
configured to forward jms messages to another server using a bridge, and the network is
unstable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
4505
days inactive
4516
days old
22 comments
4 participants
participants (4)
-
Brian Stansberry (JIRA) -
Clebert Suconic (JIRA)
-
Epi Vou (JIRA)
-
Jeff Mesnil (JIRA)