[jboss-jira] [JBoss JIRA] (JGRP-1618) missingMessageReceived() never called in NAKACK resulting in memory leak

[Akhtar Momin (Jira)]

    [ https://issues.redhat.com/browse/JGRP-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13939330#comment-13939330 ] 

Akhtar Momin commented on JGRP-1618:
------------------------------------

We are using 2.6.15 JGroups jar for our application and we are facing memory leak issue and heap dump revealed that the problem was in the xmit_table ConcurrentHashMap of Org.jgroups.protocols.pbcast.NAKACK. 

Heap dump generation in production is quiet frequent i.e. something around every 14-15 days .We have to always restart the application in production in order to make it work which is not good.

1.       Could you please help us to understand the issue and why it is retaining so much of heap?
2.       Which version of jgroup jar would be best in order to avoid this issue?
3.       Is there any fix in particular java file of jgroup collection which will help in reducing the retained heap memory?

Please let us know if you need any information in order to understand the problem. 

Thanks in advance


> missingMessageReceived() never called in NAKACK resulting in memory leak
> ------------------------------------------------------------------------
>
>                 Key: JGRP-1618
>                 URL: https://issues.redhat.com/browse/JGRP-1618
>             Project: JGroups
>          Issue Type: Bug
>    Affects Versions: 2.8.1, 2.12.2
>         Environment: Java 6
>            Reporter: Harry Mark
>            Assignee: Bela Ban
>            Priority: Major
>             Fix For: 3.2.9, 3.3
>
>         Attachments: 1618-jgrp-patch.jar, NakReceiverWindow.java, screenshot-1.png
>
>
> We are using JGroups 2.8.1 and encountered a memory leak where it eventually ran out of CMS Old Gen memory.  The heap dump revealed that the problem was in the xmit_stats ConcurrentHashMap of org.jgroups.protocols.pbcast.NAKACK.
> After much analysis here's what we found:  when the system is under load, messages can start arriving out of order. When the receiver receives a higher sequence number than expected, it requests the sender retransmit the missing messages with the lower sequence numbers. The sender sends the missing message, however the bug in the NakReceiverWindow meant that the missing message was never purged from the Map that tracks missing messages (xmit_stats) because missingMessageReceived() was never invoked. Over time this Map grows and starts using up CMS Old Gen; the only way it would get reduced was when a server left the cluster and the missing messages were purged for that server.
> In JMX, the MissingMsgsReceived attribute of jgroups:cluster=*,protocol=NAKACK,type=protocol was always zero, confirming that it never purged any received "missing messages".
> When I looked at the most recent GA version 3.2.8 of NakReceiverWindow.java , it has the corrected logic that ensures that missingMessageReceived() is called.  I checked the the most recent 2.x, which is 2.12.2, and found it also has the same bug in the logic as 2.8.1.  This bug may apply to other 2.x but I did not check.
> Attached is the fixed NakReceiverWindow.java for 2.8.1.
> After applying the patch, the memory leak went away.



--
This message was sent by Atlassian Jira
(v7.13.8#713008)