[JBoss JIRA] (JGRP-1618) missingMessageReceived() never called in NAKACK resulting in memory leak
by Akhtar Momin (Jira)
[ https://issues.jboss.org/browse/JGRP-1618?page=com.atlassian.jira.plugin.... ]
Akhtar Momin commented on JGRP-1618:
------------------------------------
We are using 2.6.15 JGroups jar for our application and we are facing same memory leak issue and heap dump revealed that the problem was in the xmit_stats ConcurrentHashMap of org.jgroups.protocols.pbcast.NAKACK. We found the above mentioned issue in customer production environment and are trying to reproduce the same. Can you give some suggestions on how could we do so??
Note:
We are planning to update the jar as suggested by Harry but before that we wanted to reproduce the issue in our local environment.
Thanks in advance.
> missingMessageReceived() never called in NAKACK resulting in memory leak
> ------------------------------------------------------------------------
>
> Key: JGRP-1618
> URL: https://issues.jboss.org/browse/JGRP-1618
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 2.8.1, 2.12.2
> Environment: Java 6
> Reporter: Harry Mark
> Assignee: Bela Ban
> Priority: Major
> Fix For: 3.2.9, 3.3
>
> Attachments: 1618-jgrp-patch.jar, NakReceiverWindow.java
>
>
> We are using JGroups 2.8.1 and encountered a memory leak where it eventually ran out of CMS Old Gen memory. The heap dump revealed that the problem was in the xmit_stats ConcurrentHashMap of org.jgroups.protocols.pbcast.NAKACK.
> After much analysis here's what we found: when the system is under load, messages can start arriving out of order. When the receiver receives a higher sequence number than expected, it requests the sender retransmit the missing messages with the lower sequence numbers. The sender sends the missing message, however the bug in the NakReceiverWindow meant that the missing message was never purged from the Map that tracks missing messages (xmit_stats) because missingMessageReceived() was never invoked. Over time this Map grows and starts using up CMS Old Gen; the only way it would get reduced was when a server left the cluster and the missing messages were purged for that server.
> In JMX, the MissingMsgsReceived attribute of jgroups:cluster=*,protocol=NAKACK,type=protocol was always zero, confirming that it never purged any received "missing messages".
> When I looked at the most recent GA version 3.2.8 of NakReceiverWindow.java , it has the corrected logic that ensures that missingMessageReceived() is called. I checked the the most recent 2.x, which is 2.12.2, and found it also has the same bug in the logic as 2.8.1. This bug may apply to other 2.x but I did not check.
> Attached is the fixed NakReceiverWindow.java for 2.8.1.
> After applying the patch, the memory leak went away.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 5 months
[JBoss JIRA] (JGRP-2394) Provide an overloaded JmxConfigurator.registerChannel that takes an ObjectName to be used as prefix instead of the domain
by Bela Ban (Jira)
[ https://issues.jboss.org/browse/JGRP-2394?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-2394:
--------------------------------
Excellent! Feel free to reopen if you run into an issue when using this change.
> Provide an overloaded JmxConfigurator.registerChannel that takes an ObjectName to be used as prefix instead of the domain
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: JGRP-2394
> URL: https://issues.jboss.org/browse/JGRP-2394
> Project: JGroups
> Issue Type: Enhancement
> Affects Versions: 4.1.6
> Reporter: Nistor Adrian
> Assignee: Bela Ban
> Priority: Major
> Fix For: 4.1.8
>
>
> The JmxConfigurator.registerChannel variant that takes a domain is not enough for use cases where multiple apps running in a jvm join the same JGroups cluster and also share the same jmx domain. In this case the channel MBean object names will collide. The de-duplication mechanism using random number suffixes does not provide predictable results. Avoiding jmx domain sharing is also not possible due to the nature of the use case.
> So we propose adding someting like this:
> JmxConfigurator.registerChannel(JChannel channel, MBeanServer server, ObjectName namePrefix, String cluster_name, boolean register_protocols)
> which would allow the application using JGroups to have greater control of the generated object names for channel and protocols. The namePrefix will provide the domain and also a key-value list that will serve as the prefix for all ObjectNames internally generated by JGroups. It will be the caller's task to ensure this leads to generation of unique names.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 5 months
[JBoss JIRA] (JGRP-2394) Provide an overloaded JmxConfigurator.registerChannel that takes an ObjectName to be used as prefix instead of the domain
by Nistor Adrian (Jira)
[ https://issues.jboss.org/browse/JGRP-2394?page=com.atlassian.jira.plugin.... ]
Nistor Adrian commented on JGRP-2394:
-------------------------------------
As I said in the description, we cannot use domain for uniqueness because we have to use a single domain in that particular use case and we want all our MBeans in it, including the JGroups stuff of course. I did tentatively use clusterName successfully already by adding a prefix that was able to differentiate (cache manager name in that particular case) , but [~dan.berindei] felt we are breaking compatibility with existing clients if we mangle the MBean names. I would not loose sleep on that, but it's not just my call. I've seen the commit in github and I think it fits our needs. Thanks [~belaban]!
> Provide an overloaded JmxConfigurator.registerChannel that takes an ObjectName to be used as prefix instead of the domain
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: JGRP-2394
> URL: https://issues.jboss.org/browse/JGRP-2394
> Project: JGroups
> Issue Type: Enhancement
> Affects Versions: 4.1.6
> Reporter: Nistor Adrian
> Assignee: Bela Ban
> Priority: Major
> Fix For: 4.1.8
>
>
> The JmxConfigurator.registerChannel variant that takes a domain is not enough for use cases where multiple apps running in a jvm join the same JGroups cluster and also share the same jmx domain. In this case the channel MBean object names will collide. The de-duplication mechanism using random number suffixes does not provide predictable results. Avoiding jmx domain sharing is also not possible due to the nature of the use case.
> So we propose adding someting like this:
> JmxConfigurator.registerChannel(JChannel channel, MBeanServer server, ObjectName namePrefix, String cluster_name, boolean register_protocols)
> which would allow the application using JGroups to have greater control of the generated object names for channel and protocols. The namePrefix will provide the domain and also a key-value list that will serve as the prefix for all ObjectNames internally generated by JGroups. It will be the caller's task to ensure this leads to generation of unique names.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 5 months
[JBoss JIRA] (JGRP-2394) Provide an overloaded JmxConfigurator.registerChannel that takes an ObjectName to be used as prefix instead of the domain
by Bela Ban (Jira)
[ https://issues.jboss.org/browse/JGRP-2394?page=com.atlassian.jira.plugin.... ]
Bela Ban resolved JGRP-2394.
----------------------------
Resolution: Done
> Provide an overloaded JmxConfigurator.registerChannel that takes an ObjectName to be used as prefix instead of the domain
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: JGRP-2394
> URL: https://issues.jboss.org/browse/JGRP-2394
> Project: JGroups
> Issue Type: Enhancement
> Affects Versions: 4.1.6
> Reporter: Nistor Adrian
> Assignee: Bela Ban
> Priority: Major
> Fix For: 4.1.8
>
>
> The JmxConfigurator.registerChannel variant that takes a domain is not enough for use cases where multiple apps running in a jvm join the same JGroups cluster and also share the same jmx domain. In this case the channel MBean object names will collide. The de-duplication mechanism using random number suffixes does not provide predictable results. Avoiding jmx domain sharing is also not possible due to the nature of the use case.
> So we propose adding someting like this:
> JmxConfigurator.registerChannel(JChannel channel, MBeanServer server, ObjectName namePrefix, String cluster_name, boolean register_protocols)
> which would allow the application using JGroups to have greater control of the generated object names for channel and protocols. The namePrefix will provide the domain and also a key-value list that will serve as the prefix for all ObjectNames internally generated by JGroups. It will be the caller's task to ensure this leads to generation of unique names.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 5 months
[JBoss JIRA] (JGRP-2394) Provide an overloaded JmxConfigurator.registerChannel that takes an ObjectName to be used as prefix instead of the domain
by Bela Ban (Jira)
[ https://issues.jboss.org/browse/JGRP-2394?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-2394:
--------------------------------
Anyway: I created {{public static void registerChannel(JChannel ch, MBeanServer server, ObjectName prefix, cluster_name, boolean register_protocols) throws Exception}}. This is supposed to be used as follows:
{{JmxConfigurator.registerChannel(ch, server, new ObjectName("domain:type=server"), "cluster", true);}}
The ObjectName contains the domain and a list of key/value pairs. If 2 identical ObjectNames are given, then a {{,duplicate=12345}} key/value will be added.
> Provide an overloaded JmxConfigurator.registerChannel that takes an ObjectName to be used as prefix instead of the domain
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: JGRP-2394
> URL: https://issues.jboss.org/browse/JGRP-2394
> Project: JGroups
> Issue Type: Enhancement
> Affects Versions: 4.1.6
> Reporter: Nistor Adrian
> Assignee: Bela Ban
> Priority: Major
> Fix For: 4.1.8
>
>
> The JmxConfigurator.registerChannel variant that takes a domain is not enough for use cases where multiple apps running in a jvm join the same JGroups cluster and also share the same jmx domain. In this case the channel MBean object names will collide. The de-duplication mechanism using random number suffixes does not provide predictable results. Avoiding jmx domain sharing is also not possible due to the nature of the use case.
> So we propose adding someting like this:
> JmxConfigurator.registerChannel(JChannel channel, MBeanServer server, ObjectName namePrefix, String cluster_name, boolean register_protocols)
> which would allow the application using JGroups to have greater control of the generated object names for channel and protocols. The namePrefix will provide the domain and also a key-value list that will serve as the prefix for all ObjectNames internally generated by JGroups. It will be the caller's task to ensure this leads to generation of unique names.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 5 months
[JBoss JIRA] (JGRP-2394) Provide an overloaded JmxConfigurator.registerChannel that takes an ObjectName to be used as prefix instead of the domain
by Bela Ban (Jira)
[ https://issues.jboss.org/browse/JGRP-2394?page=com.atlassian.jira.plugin.... ]
Bela Ban updated JGRP-2394:
---------------------------
Fix Version/s: 4.1.8
> Provide an overloaded JmxConfigurator.registerChannel that takes an ObjectName to be used as prefix instead of the domain
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: JGRP-2394
> URL: https://issues.jboss.org/browse/JGRP-2394
> Project: JGroups
> Issue Type: Enhancement
> Affects Versions: 4.1.6
> Reporter: Nistor Adrian
> Assignee: Bela Ban
> Priority: Major
> Fix For: 4.1.8
>
>
> The JmxConfigurator.registerChannel variant that takes a domain is not enough for use cases where multiple apps running in a jvm join the same JGroups cluster and also share the same jmx domain. In this case the channel MBean object names will collide. The de-duplication mechanism using random number suffixes does not provide predictable results. Avoiding jmx domain sharing is also not possible due to the nature of the use case.
> So we propose adding someting like this:
> JmxConfigurator.registerChannel(JChannel channel, MBeanServer server, ObjectName namePrefix, String cluster_name, boolean register_protocols)
> which would allow the application using JGroups to have greater control of the generated object names for channel and protocols. The namePrefix will provide the domain and also a key-value list that will serve as the prefix for all ObjectNames internally generated by JGroups. It will be the caller's task to ensure this leads to generation of unique names.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 5 months
[JBoss JIRA] (JGRP-2394) Provide an overloaded JmxConfigurator.registerChannel that takes an ObjectName to be used as prefix instead of the domain
by Bela Ban (Jira)
[ https://issues.jboss.org/browse/JGRP-2394?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-2394:
--------------------------------
Why can't you use {{domain}} and/or {{clusterName}} to create unique registrations? These don't have to correspond to the real domain or cluster name.
> Provide an overloaded JmxConfigurator.registerChannel that takes an ObjectName to be used as prefix instead of the domain
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: JGRP-2394
> URL: https://issues.jboss.org/browse/JGRP-2394
> Project: JGroups
> Issue Type: Enhancement
> Affects Versions: 4.1.6
> Reporter: Nistor Adrian
> Assignee: Bela Ban
> Priority: Major
>
> The JmxConfigurator.registerChannel variant that takes a domain is not enough for use cases where multiple apps running in a jvm join the same JGroups cluster and also share the same jmx domain. In this case the channel MBean object names will collide. The de-duplication mechanism using random number suffixes does not provide predictable results. Avoiding jmx domain sharing is also not possible due to the nature of the use case.
> So we propose adding someting like this:
> JmxConfigurator.registerChannel(JChannel channel, MBeanServer server, ObjectName namePrefix, String cluster_name, boolean register_protocols)
> which would allow the application using JGroups to have greater control of the generated object names for channel and protocols. The namePrefix will provide the domain and also a key-value list that will serve as the prefix for all ObjectNames internally generated by JGroups. It will be the caller's task to ensure this leads to generation of unique names.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 5 months
[JBoss JIRA] (WFCORE-4740) (7.3.0) preferIPv6Addresses and preferIPv4Stack System Properties are Mishandled in the Config
by Ivo Studensky (Jira)
[ https://issues.jboss.org/browse/WFCORE-4740?page=com.atlassian.jira.plugi... ]
Ivo Studensky reassigned WFCORE-4740:
-------------------------------------
Assignee: Ivo Studensky (was: Teresa Miyar Gil)
> (7.3.0) preferIPv6Addresses and preferIPv4Stack System Properties are Mishandled in the Config
> ----------------------------------------------------------------------------------------------
>
> Key: WFCORE-4740
> URL: https://issues.jboss.org/browse/WFCORE-4740
> Project: WildFly Core
> Issue Type: Bug
> Components: Server
> Affects Versions: 10.0.0.Final
> Environment: * JBoss EAP 7.1/7.2
> * Interface attached to port 0.0.0.0
> * Red Hat Enterprise Linux 7
> * IPv6 disabled in the kernel
> sysctl -w net.ipv6.conf.all.disable_ipv6=1
> sysctl -w net.ipv6.conf.default.disable_ipv6=1
> * System properties -Djava.net.preferIPv4Stack=false -Djava.net.preferIPv6Addresses=true
> Reporter: Ivo Studensky
> Assignee: Ivo Studensky
> Priority: Major
> Labels: ipv6, startup
>
> Error is thrown on startup.
> Caused by: java.net.SocketException: Protocol family unavailable
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.xnio.nio.NioXnioWorker.createTcpConnectionServer(NioXnioWorker.java:179)
> at org.xnio.XnioWorker.createStreamConnectionServer(XnioWorker.java:310)
> at io.undertow.protocols.ssl.UndertowXnioSsl.createSslConnectionServer(UndertowXnioSsl.java:301)
> at org.wildfly.extension.undertow.HttpsListenerService.startListening(HttpsListenerService.java:127)
> The customer ships a turn key solution that has the two system properties set: -Djava.net.preferIPv4Stack=false -Djava.net.preferIPv6Addresses=true because they are needed for some cases for customers running IPv6, but others want to harden their systems by disabling IPv4.
> This works on JBoss EAP 6, but it throws the error on JBoss EAP 7 on the same version of Java. Furthermore, adding just Djava.net.preferIPv4Stack=false has the same issue, even though it the default value, while leaving it off starts.
> This appears to be related to controller/src/main/java/org/jboss/as/controller/interfaces/OverallInterfaceCriteria.java#pruneIPTypes where if both properties are null, it leaves the set of candidate addresses alone, but it either are set, it strips out all IPv4 addresses.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 5 months
[JBoss JIRA] (WFCORE-4740) (7.3.0) preferIPv6Addresses and preferIPv4Stack System Properties are Mishandled in the Config
by Ivo Studensky (Jira)
[ https://issues.jboss.org/browse/WFCORE-4740?page=com.atlassian.jira.plugi... ]
Ivo Studensky updated WFCORE-4740:
----------------------------------
Fix Version/s: (was: 11.0.0.Beta1)
> (7.3.0) preferIPv6Addresses and preferIPv4Stack System Properties are Mishandled in the Config
> ----------------------------------------------------------------------------------------------
>
> Key: WFCORE-4740
> URL: https://issues.jboss.org/browse/WFCORE-4740
> Project: WildFly Core
> Issue Type: Bug
> Components: Server
> Affects Versions: 10.0.0.Final
> Environment: * JBoss EAP 7.1/7.2
> * Interface attached to port 0.0.0.0
> * Red Hat Enterprise Linux 7
> * IPv6 disabled in the kernel
> sysctl -w net.ipv6.conf.all.disable_ipv6=1
> sysctl -w net.ipv6.conf.default.disable_ipv6=1
> * System properties -Djava.net.preferIPv4Stack=false -Djava.net.preferIPv6Addresses=true
> Reporter: Ivo Studensky
> Assignee: Ivo Studensky
> Priority: Major
> Labels: ipv6, startup
>
> Error is thrown on startup.
> Caused by: java.net.SocketException: Protocol family unavailable
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.xnio.nio.NioXnioWorker.createTcpConnectionServer(NioXnioWorker.java:179)
> at org.xnio.XnioWorker.createStreamConnectionServer(XnioWorker.java:310)
> at io.undertow.protocols.ssl.UndertowXnioSsl.createSslConnectionServer(UndertowXnioSsl.java:301)
> at org.wildfly.extension.undertow.HttpsListenerService.startListening(HttpsListenerService.java:127)
> The customer ships a turn key solution that has the two system properties set: -Djava.net.preferIPv4Stack=false -Djava.net.preferIPv6Addresses=true because they are needed for some cases for customers running IPv6, but others want to harden their systems by disabling IPv4.
> This works on JBoss EAP 6, but it throws the error on JBoss EAP 7 on the same version of Java. Furthermore, adding just Djava.net.preferIPv4Stack=false has the same issue, even though it the default value, while leaving it off starts.
> This appears to be related to controller/src/main/java/org/jboss/as/controller/interfaces/OverallInterfaceCriteria.java#pruneIPTypes where if both properties are null, it leaves the set of candidate addresses alone, but it either are set, it strips out all IPv4 addresses.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 5 months