[jboss-jira] [JBoss JIRA] (AS7-5394) MemoryLeak (StackTraceElement objects) in JMS Bridges when the connection times out

Epi Vou (JIRA) jira-events at lists.jboss.org
Fri Aug 17 12:22:15 EDT 2012 

    [ https://issues.jboss.org/browse/AS7-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712279#comment-12712279 ] 

Epi Vou edited comment on AS7-5394 at 8/17/12 12:21 PM:
--------------------------------------------------------

The bridge as I understand is a hornetQ bridge, which I have configured through the web management interface of jboss. I use a queue, a divert which diverts messages going to a topic to this queue, and a bridge that forwards messages from the queue on the other jboss. Everything works as expected for some time. Messages are produced on the server containing the bridge and are consumed by the other server. If the network drops, the servers will output some warnings, and any messages produced during this time, will be consumed by the other server as soon as the network works again.

at network disconnection the server that has the bridge configured outputs the following :
{noformat}
18:54:51,211 ERROR [org.hornetq.core.remoting.impl.netty.NettyConnector] (Thread-7 (HornetQ-server-HornetQServerImpl::serverUUID=df86d314-e7b4-11e1-9714-895d978cba07-229910155)) Failed to create netty connection: java.net.SocketTimeoutException: connect timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.6.0_33]
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) [rt.jar:1.6.0_33]
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) [rt.jar:1.6.0_33]
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) [rt.jar:1.6.0_33]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) [rt.jar:1.6.0_33]
        at java.net.Socket.connect(Socket.java:529) [rt.jar:1.6.0_33]
        at org.jboss.netty.channel.socket.oio.OioClientSocketPipelineSink.connect(OioClientSocketPipelineSink.java:114) [netty-3.2.6.Final.jar:]
        at org.jboss.netty.channel.socket.oio.OioClientSocketPipelineSink.eventSunk(OioClientSocketPipelineSink.java:74) [netty-3.2.6.Final.jar:]
        at org.jboss.netty.channel.Channels.connect(Channels.java:541) [netty-3.2.6.Final.jar:]
        at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:210) [netty-3.2.6.Final.jar:]
        at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:227) [netty-3.2.6.Final.jar:]
        at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:188) [netty-3.2.6.Final.jar:]
        at org.hornetq.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:499) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1144) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:994) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:225) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:752) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.server.cluster.impl.BridgeImpl.createSessionFactory(BridgeImpl.java:789) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.server.cluster.impl.BridgeImpl.connect(BridgeImpl.java:805) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.server.cluster.impl.BridgeImpl$ConnectRunnable.run(BridgeImpl.java:1089) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_33]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_33]
        at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_33]
{noformat}

and the receiving jboss server outputs the following :
{noformat}
[19:02:04,284 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from /194.219.97.67:38329. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]
19:02:04,284 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Client connection failed, clearing up resources for session 76d40991-e884-11e1-8081-df4631c082f3
19:02:04,284 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Cleared up resources for session 76d40991-e884-11e1-8081-df4631c082f3
{noformat}

But after a few disconnects the log on the server that has the bridge configured get's filled up with the following exceptions :

{noformat}
15:37:11,393 WARN  [org.hornetq.core.server.cluster.impl.BridgeImpl] (Thread-1 (HornetQ-server-HornetQServerImpl::serverUUID=df86d314-e
7b4-11e1-9714-895d978cba07-1698482702)) Unable to send message Reference[19327439667]:RELIABLE:ServerMessage[messageID=19327439667,priority=4, bodySize=1500,expiration=0, durable=true, address=null,properties=TypedProperties[{_HQ_ORIG_MESSAGE_ID=19327439666, _HQ_ORIG_ADDRESS=jms.topic.livescoresTopic, filter=backend_push_event}]]@737247918, will try again once bridge reconnects: HornetQException[errorCode=102 message=Producer is closed]
        at org.hornetq.core.client.impl.ClientProducerImpl.checkClosed(ClientProducerImpl.java:317) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:140) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.server.cluster.impl.BridgeImpl.deliverStandardMessage(BridgeImpl.java:698) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.server.cluster.impl.BridgeImpl.handle(BridgeImpl.java:584) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.server.impl.QueueImpl.handle(QueueImpl.java:2208) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:1759) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.server.impl.QueueImpl.access$1100(QueueImpl.java:77) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2455) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_33]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_33]
        at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_33]
{noformat}

And after a few minutes the server becomes unresponsive and out of memory...
                
      was (Author: pimenas_black):
    The bridge as I understand is a hornetQ bridge, which I have configured through the web management interface of jboss. I use a queue, a divert which diverts messages going to a topic to this queue, and a bridge that forwards messages from the queue on the other jboss. Everything works as expected for some time. Messages are produced on the server containing the bridge and are consumed by the other server. If the network drops, the servers will output some warnings, and any messages produced during this time, will be consumed by the other server as soon as the network works again.

at network disconnection the server that has the bridge configured outputs the following :
{noformat}
18:54:51,211 ERROR [org.hornetq.core.remoting.impl.netty.NettyConnector] (Thread-7 (HornetQ-server-HornetQServerImpl::serverUUID=df86d314-e7b4-11e1-9714-895d978cba07-229910155)) Failed to create netty connection: java.net.SocketTimeoutException: connect timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.6.0_33]
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) [rt.jar:1.6.0_33]
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) [rt.jar:1.6.0_33]
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) [rt.jar:1.6.0_33]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) [rt.jar:1.6.0_33]
        at java.net.Socket.connect(Socket.java:529) [rt.jar:1.6.0_33]
        at org.jboss.netty.channel.socket.oio.OioClientSocketPipelineSink.connect(OioClientSocketPipelineSink.java:114) [netty-3.2.6.Final.jar:]
        at org.jboss.netty.channel.socket.oio.OioClientSocketPipelineSink.eventSunk(OioClientSocketPipelineSink.java:74) [netty-3.2.6.Final.jar:]
        at org.jboss.netty.channel.Channels.connect(Channels.java:541) [netty-3.2.6.Final.jar:]
        at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:210) [netty-3.2.6.Final.jar:]
        at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:227) [netty-3.2.6.Final.jar:]
        at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:188) [netty-3.2.6.Final.jar:]
        at org.hornetq.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:499) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1144) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:994) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:225) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:752) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.server.cluster.impl.BridgeImpl.createSessionFactory(BridgeImpl.java:789) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.server.cluster.impl.BridgeImpl.connect(BridgeImpl.java:805) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.server.cluster.impl.BridgeImpl$ConnectRunnable.run(BridgeImpl.java:1089) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_33]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_33]
        at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_33]
{noformat}

and the receiving jboss server outputs the following :
{noformat}
[19:02:04,284 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from /194.219.97.67:38329. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]
19:02:04,284 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Client connection failed, clearing up resources for session 76d40991-e884-11e1-8081-df4631c082f3
19:02:04,284 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Cleared up resources for session 76d40991-e884-11e1-8081-df4631c082f3
{noformat}

But after a few disconnects the log on the server that has the bridge configured get's filled up with the following exceptions :

{noformat}
15:37:11,393 WARN  [org.hornetq.core.server.cluster.impl.BridgeImpl] (Thread-1 (HornetQ-server-HornetQServerImpl::serverUUID=df86d314-e
7b4-11e1-9714-895d978cba07-1698482702)) Unable to send message Reference[19327439667]:RELIABLE:ServerMessage[messageID=19327439667,priority=4, bodySize=1500,expiration=0, durable=true, address=null,properties=TypedProperties[{_HQ_ORIG_MESSAGE_ID=19327439666, _HQ_ORIG_ADDRESS=jms.topic.livescoresTopic, filter=backend_push_event}]]@737247918, will try again once bridge reconnects: HornetQException[errorCode=102 message=Producer is closed]
        at org.hornetq.core.client.impl.ClientProducerImpl.checkClosed(ClientProducerImpl.java:317) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:140) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.server.cluster.impl.BridgeImpl.deliverStandardMessage(BridgeImpl.java:698) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.server.cluster.impl.BridgeImpl.handle(BridgeImpl.java:584) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.server.impl.QueueImpl.handle(QueueImpl.java:2208) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:1759) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.server.impl.QueueImpl.access$1100(QueueImpl.java:77) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2455) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [hornetq-core-2.2.19.Final.jar:2.2.19.SNAPSHOT (HQ_2_2_19_SNAPSHOT, 122)]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_33]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_33]
        at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_33]
{noformat}
                  
> MemoryLeak (StackTraceElement objects) in JMS Bridges when the connection times out
> -----------------------------------------------------------------------------------
>
>                 Key: AS7-5394
>                 URL: https://issues.jboss.org/browse/AS7-5394
>             Project: Application Server 7
>          Issue Type: Bug
>          Components: JMS
>    Affects Versions: 7.1.2.Final (EAP), 7.1.3.Final (EAP)
>            Reporter: Epi Vou
>            Assignee: Jeff Mesnil
>         Attachments: bug.png
>
>
> A jboss server eventually will start throwing OutOfMemory exceptions when it is configured to forward jms messages to another server using a bridge, and the network is unstable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

More information about the jboss-jira mailing list