[JBoss JIRA] (JGRP-1618) missingMessageReceived() never called in NAKACK resulting in memory leak
by Bela Ban (Jira)
[ https://issues.redhat.com/browse/JGRP-1618?page=com.atlassian.jira.plugin... ]
Bela Ban commented on JGRP-1618:
--------------------------------
Sorry, but I don't support such an old version, 2.6.x is almost 10 years old!
> missingMessageReceived() never called in NAKACK resulting in memory leak
> ------------------------------------------------------------------------
>
> Key: JGRP-1618
> URL: https://issues.redhat.com/browse/JGRP-1618
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 2.8.1, 2.12.2
> Environment: Java 6
> Reporter: Harry Mark
> Assignee: Bela Ban
> Priority: Major
> Fix For: 3.2.9, 3.3
>
> Attachments: 1618-jgrp-patch.jar, NakReceiverWindow.java, screenshot-1.png
>
>
> We are using JGroups 2.8.1 and encountered a memory leak where it eventually ran out of CMS Old Gen memory. The heap dump revealed that the problem was in the xmit_stats ConcurrentHashMap of org.jgroups.protocols.pbcast.NAKACK.
> After much analysis here's what we found: when the system is under load, messages can start arriving out of order. When the receiver receives a higher sequence number than expected, it requests the sender retransmit the missing messages with the lower sequence numbers. The sender sends the missing message, however the bug in the NakReceiverWindow meant that the missing message was never purged from the Map that tracks missing messages (xmit_stats) because missingMessageReceived() was never invoked. Over time this Map grows and starts using up CMS Old Gen; the only way it would get reduced was when a server left the cluster and the missing messages were purged for that server.
> In JMX, the MissingMsgsReceived attribute of jgroups:cluster=*,protocol=NAKACK,type=protocol was always zero, confirming that it never purged any received "missing messages".
> When I looked at the most recent GA version 3.2.8 of NakReceiverWindow.java , it has the corrected logic that ensures that missingMessageReceived() is called. I checked the the most recent 2.x, which is 2.12.2, and found it also has the same bug in the logic as 2.8.1. This bug may apply to other 2.x but I did not check.
> Attached is the fixed NakReceiverWindow.java for 2.8.1.
> After applying the patch, the memory leak went away.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years
[JBoss JIRA] (DROOLS-4888) spring boot and drools integration occur path error and cannot load rule
by 韩 济蓬 (Jira)
[ https://issues.redhat.com/browse/DROOLS-4888?page=com.atlassian.jira.plug... ]
韩 济蓬 commented on DROOLS-4888:
------------------------------
hello, I package the code as a jar, execute it, and get the output as follows:
{code:java}
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.2.2.RELEASE)
2019-12-23 09:28:33.301 INFO 12968 --- [ main] c.b.test.spring.drools.SpringDroolsApp : Starting SpringDroolsApp on LAPTOP-6B2HCHVA with PID 12968 (C:\Users\user\Desktop\springdrools\target\spring-drools-1.0.jar started by user in C:\Users\user\Desktop\springdrools\target)
2019-12-23 09:28:33.306 INFO 12968 --- [ main] c.b.test.spring.drools.SpringDroolsApp : No active profile set, falling back to default profiles: default
2019-12-23 09:28:34.206 INFO 12968 --- [ main] o.k.s.KModuleBeanFactoryPostProcessor : :: BeanFactoryPostProcessor::postProcessBeanFactory called ::
2019-12-23 09:28:34.215 WARN 12968 --- [ main] o.d.c.k.b.impl.ClasspathKieProject : Unable to find pom.properties in /C:/Users/user/Desktop/springdrools/target/spring-drools-1.0.jar!/BOOT-INF/classes
2019-12-23 09:28:34.223 INFO 12968 --- [ main] o.d.c.k.b.impl.ClasspathKieProject : Recursed up folders, found and used pom.xml C:\Users\user\Desktop\springdrools\pom.xml
2019-12-23 09:28:34.223 INFO 12968 --- [ main] o.k.s.KModuleBeanFactoryPostProcessor : Found project with releaseId: cn.beausoft.test:spring-drools:1.0
2019-12-23 09:28:34.384 INFO 12968 --- [ main] o.k.s.KModuleBeanFactoryPostProcessor : Adding KieModule from /C:/Users/user/Desktop/springdrools/target/spring-drools-1.0.jar!/BOOT-INF/classes to repository.
2019-12-23 09:28:35.138 INFO 12968 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 1235 (http)
2019-12-23 09:28:35.151 INFO 12968 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2019-12-23 09:28:35.151 INFO 12968 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.29]
2019-12-23 09:28:35.499 INFO 12968 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2019-12-23 09:28:35.500 INFO 12968 --- [ main] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 2139 ms
2019-12-23 09:28:35.737 WARN 12968 --- [ main] o.d.c.kie.builder.impl.KieProject : No files found for KieBase testRules
2019-12-23 09:28:36.075 INFO 12968 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor'
2019-12-23 09:28:36.302 INFO 12968 --- [ main] o.s.s.c.ThreadPoolTaskScheduler : Initializing ExecutorService 'taskScheduler'
2019-12-23 09:28:36.390 INFO 12968 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 1235 (http) with context path ''
2019-12-23 09:28:36.395 INFO 12968 --- [ main] c.b.test.spring.drools.SpringDroolsApp : Started SpringDroolsApp in 3.543 seconds (JVM running for 4.269)
{code}
There was a warning:
!screenshot-1.png|thumbnail!
Visit http://localhost:1235/testUser interface, the results are as follows:
{code:json}
{"username":"han","name":"aaa","age":18}
{code}
The correct result should be the following:
{code:json}
{"username":"han","name":"bbb","age":18}
{code}
> spring boot and drools integration occur path error and cannot load rule
> ------------------------------------------------------------------------
>
> Key: DROOLS-4888
> URL: https://issues.redhat.com/browse/DROOLS-4888
> Project: Drools
> Issue Type: Bug
> Components: docs, integration
> Affects Versions: 7.30.0.Final
> Reporter: 韩 济蓬
> Assignee: Mario Fusco
> Priority: Major
> Attachments: screenshot-1.png, springdrools.zip
>
>
> I'm trying to run drools 7.30.0.Final with Spring Boot 2.2.2.RELEASE.I configured drools as the documentation says. My drools-config.xml configuration file is where I keep the drools beans and looks like this:
> {code:xml}
> <?xml version="1.0" encoding="UTF-8"?>
> <beans xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns:kie="http://drools.org/schema/kie-spring"
> xsi:schemaLocation="http://www.springframework.org/schema/beans https://www.springframework.org/schema/beans/spring-beans.xsd http://drools.org/schema/kie-spring http://drools.org/schema/kie-spring.xsd">
> <kie:kmodule id="sample_module">
> <kie:kbase name="testRules" packages="rules" declarativeAgenda="true">
> <kie:ksession name="ksession1" type="stateless"/>
> <kie:ksession name="ksession2" scope="prototype">
> <kie:ruleRuntimeEventListener ref="mockRuleRuntimeEventListener" />
> <kie:consoleLogger />
> </kie:ksession>
> </kie:kbase>
> </kie:kmodule>
> <!-- https://docs.jboss.org/drools/release/7.30.0.Final/drools-docs/html_singl... -->
> <!-- <bean id="kiePostProcessor" class="org.kie.spring.KModuleBeanFactoryPostProcessor"/> -->
> <bean class="org.kie.spring.annotations.KModuleAnnotationPostProcessor"/>
> </beans>
> {code}
> occur a warn looks:
> 2019-12-19 14:44:25.231 WARN 9916 --- [ main] o.d.c.kie.builder.impl.KieProject : No files found for KieBase testRules
> when i try debug org.kie.spring.KModuleBeanFactoryPostProcessor. Trace to ZipKieModule at line 133. cannot uncompress,get variable urlPath value is "/BOOT-INF/classes" jarFile values is "\Users\user\Desktop\springdrools\target\spring-drools-1.0.jar!\BOOT-INF\classes". lead to cannot load resource.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years
[JBoss JIRA] (JGRP-1618) missingMessageReceived() never called in NAKACK resulting in memory leak
by Akhtar Momin (Jira)
[ https://issues.redhat.com/browse/JGRP-1618?page=com.atlassian.jira.plugin... ]
Akhtar Momin commented on JGRP-1618:
------------------------------------
We are using 2.6.15 JGroups jar for our application and we are facing memory leak issue and heap dump revealed that the problem was in the xmit_table ConcurrentHashMap of Org.jgroups.protocols.pbcast.NAKACK.
Heap dump generation in production is quiet frequent i.e. something around every 14-15 days .We have to always restart the application in production in order to make it work which is not good.
1. Could you please help us to understand the issue and why it is retaining so much of heap?
2. Which version of jgroup jar would be best in order to avoid this issue?
3. Is there any fix in particular java file of jgroup collection which will help in reducing the retained heap memory?
Please let us know if you need any information in order to understand the problem.
Thanks in advance
> missingMessageReceived() never called in NAKACK resulting in memory leak
> ------------------------------------------------------------------------
>
> Key: JGRP-1618
> URL: https://issues.redhat.com/browse/JGRP-1618
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 2.8.1, 2.12.2
> Environment: Java 6
> Reporter: Harry Mark
> Assignee: Bela Ban
> Priority: Major
> Fix For: 3.2.9, 3.3
>
> Attachments: 1618-jgrp-patch.jar, NakReceiverWindow.java, screenshot-1.png
>
>
> We are using JGroups 2.8.1 and encountered a memory leak where it eventually ran out of CMS Old Gen memory. The heap dump revealed that the problem was in the xmit_stats ConcurrentHashMap of org.jgroups.protocols.pbcast.NAKACK.
> After much analysis here's what we found: when the system is under load, messages can start arriving out of order. When the receiver receives a higher sequence number than expected, it requests the sender retransmit the missing messages with the lower sequence numbers. The sender sends the missing message, however the bug in the NakReceiverWindow meant that the missing message was never purged from the Map that tracks missing messages (xmit_stats) because missingMessageReceived() was never invoked. Over time this Map grows and starts using up CMS Old Gen; the only way it would get reduced was when a server left the cluster and the missing messages were purged for that server.
> In JMX, the MissingMsgsReceived attribute of jgroups:cluster=*,protocol=NAKACK,type=protocol was always zero, confirming that it never purged any received "missing messages".
> When I looked at the most recent GA version 3.2.8 of NakReceiverWindow.java , it has the corrected logic that ensures that missingMessageReceived() is called. I checked the the most recent 2.x, which is 2.12.2, and found it also has the same bug in the logic as 2.8.1. This bug may apply to other 2.x but I did not check.
> Attached is the fixed NakReceiverWindow.java for 2.8.1.
> After applying the patch, the memory leak went away.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years
[JBoss JIRA] (JGRP-1618) missingMessageReceived() never called in NAKACK resulting in memory leak
by Akhtar Momin (Jira)
[ https://issues.redhat.com/browse/JGRP-1618?page=com.atlassian.jira.plugin... ]
Akhtar Momin updated JGRP-1618:
-------------------------------
Attachment: screenshot-1.png
> missingMessageReceived() never called in NAKACK resulting in memory leak
> ------------------------------------------------------------------------
>
> Key: JGRP-1618
> URL: https://issues.redhat.com/browse/JGRP-1618
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 2.8.1, 2.12.2
> Environment: Java 6
> Reporter: Harry Mark
> Assignee: Bela Ban
> Priority: Major
> Fix For: 3.2.9, 3.3
>
> Attachments: 1618-jgrp-patch.jar, NakReceiverWindow.java, screenshot-1.png
>
>
> We are using JGroups 2.8.1 and encountered a memory leak where it eventually ran out of CMS Old Gen memory. The heap dump revealed that the problem was in the xmit_stats ConcurrentHashMap of org.jgroups.protocols.pbcast.NAKACK.
> After much analysis here's what we found: when the system is under load, messages can start arriving out of order. When the receiver receives a higher sequence number than expected, it requests the sender retransmit the missing messages with the lower sequence numbers. The sender sends the missing message, however the bug in the NakReceiverWindow meant that the missing message was never purged from the Map that tracks missing messages (xmit_stats) because missingMessageReceived() was never invoked. Over time this Map grows and starts using up CMS Old Gen; the only way it would get reduced was when a server left the cluster and the missing messages were purged for that server.
> In JMX, the MissingMsgsReceived attribute of jgroups:cluster=*,protocol=NAKACK,type=protocol was always zero, confirming that it never purged any received "missing messages".
> When I looked at the most recent GA version 3.2.8 of NakReceiverWindow.java , it has the corrected logic that ensures that missingMessageReceived() is called. I checked the the most recent 2.x, which is 2.12.2, and found it also has the same bug in the logic as 2.8.1. This bug may apply to other 2.x but I did not check.
> Attached is the fixed NakReceiverWindow.java for 2.8.1.
> After applying the patch, the memory leak went away.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years