[JBoss JIRA] (DROOLS-5615) Deadlock in ProjectClassLoader
by Michael Osganian (Jira)
Michael Osganian created DROOLS-5615:
----------------------------------------
Summary: Deadlock in ProjectClassLoader
Key: DROOLS-5615
URL: https://issues.redhat.com/browse/DROOLS-5615
Project: Drools
Issue Type: Bug
Components: core engine
Affects Versions: 6.5.0.Final
Reporter: Michael Osganian
Assignee: Mario Fusco
Hi I am using 6.5.0.Final of drools and I have a deadlock situation that is hard to reproduce. We are running about 4 threads each running its own rule engine with the same set of rules but different data. Below is the deadlock on 3 of the threads and looks like the ProjectClassLoader is at the root of the problem:
Found one Java-level deadlock:Found one Java-level deadlock:============================="InferenceThread-4": waiting to lock monitor 0x00007fd5cc044a38 (object 0x0000000081b0ee18, a org.drools.core.rule.JavaDialectRuntimeData$PackageClassLoader), which is held by "InferenceThread-2""InferenceThread-2": waiting to lock monitor 0x00007fd5cc044ae8 (object 0x0000000081a2a4a8, a org.drools.core.common.ProjectClassLoader$DefaultInternalTypesClassLoader), which is held by "InferenceThread-1""InferenceThread-1": waiting to lock monitor 0x00007fd5bc01fd28 (object 0x0000000081acc610, a org.drools.core.common.ProjectClassLoader), which is held by "InferenceThread-2"
Java stack information for the threads listed above:==================================================="InferenceThread-4": at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.mvel2.util.ParseTools.forNameWithInner(ParseTools.java:2187) at org.mvel2.ParserConfiguration.checkForDynamicImport(ParserConfiguration.java:169) at org.mvel2.ParserConfiguration.hasImport(ParserConfiguration.java:197) at org.mvel2.ParserContext.hasImport(ParserContext.java:373) at org.mvel2.compiler.AbstractParser.createPropertyToken(AbstractParser.java:1399) at org.mvel2.compiler.AbstractParser.nextToken(AbstractParser.java:903) at org.mvel2.compiler.ExpressionCompiler._compile(ExpressionCompiler.java:111) at org.mvel2.compiler.ExpressionCompiler.compile(ExpressionCompiler.java:52) at org.mvel2.MVEL.compileExpression(MVEL.java:810) at org.drools.core.base.mvel.MVELCompilationUnit.compile(MVELCompilationUnit.java:451) at org.drools.core.base.mvel.MVELCompilationUnit.getCompiledExpression(MVELCompilationUnit.java:273) at org.drools.core.rule.constraint.MvelConstraint.createMvelConditionEvaluator(MvelConstraint.java:269) at org.drools.core.rule.constraint.MvelConstraint.evaluate(MvelConstraint.java:249) at org.drools.core.rule.constraint.MvelConstraint.isAllowedCachedLeft(MvelConstraint.java:227) at org.drools.core.common.TripleBetaConstraints.isAllowedCachedLeft(TripleBetaConstraints.java:118) at org.drools.core.phreak.PhreakJoinNode.doLeftInserts(PhreakJoinNode.java:113) at org.drools.core.phreak.PhreakJoinNode.doNode(PhreakJoinNode.java:76) at org.drools.core.phreak.RuleNetworkEvaluator.switchOnDoBetaNode(RuleNetworkEvaluator.java:519) at org.drools.core.phreak.RuleNetworkEvaluator.evalBetaNode(RuleNetworkEvaluator.java:505) at org.drools.core.phreak.RuleNetworkEvaluator.evalNode(RuleNetworkEvaluator.java:341) at org.drools.core.phreak.RuleNetworkEvaluator.innerEval(RuleNetworkEvaluator.java:301) at org.drools.core.phreak.RuleNetworkEvaluator.outerEval(RuleNetworkEvaluator.java:136) at org.drools.core.phreak.RuleNetworkEvaluator.evaluateNetwork(RuleNetworkEvaluator.java:94) at org.drools.core.phreak.RuleExecutor.reEvaluateNetwork(RuleExecutor.java:194) at org.drools.core.phreak.RuleExecutor.evaluateNetworkAndFire(RuleExecutor.java:73) at org.drools.core.common.DefaultAgenda.fireNextItem(DefaultAgenda.java:970) at org.drools.core.common.DefaultAgenda.fireLoop(DefaultAgenda.java:1312) at org.drools.core.common.DefaultAgenda.fireAllRules(DefaultAgenda.java:1251) at org.drools.core.impl.StatefulKnowledgeSessionImpl.internalFireAllRules(StatefulKnowledgeSessionImpl.java:1359) at org.drools.core.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:1350) at org.drools.core.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:1331) at ActivityInferenceSession.fireRules(ActivityInferenceSession.java:261) at InferenceRunnable.doTryInference(InferenceRunnable.java:226) at InferenceRunnable.doInference(InferenceRunnable.java:129) at InferenceRunnable.run(InferenceRunnable.java:100) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)"InferenceThread-2": at java.lang.ClassLoader.loadClass(ClassLoader.java:404) - waiting to lock <0x0000000081a2a4a8> (a org.drools.core.common.ProjectClassLoader$DefaultInternalTypesClassLoader) at org.drools.core.common.ProjectClassLoader$DefaultInternalTypesClassLoader.loadType(ProjectClassLoader.java:394) at org.drools.core.common.ProjectClassLoader.loadType(ProjectClassLoader.java:172) at org.drools.core.common.ProjectClassLoader.loadClass(ProjectClassLoader.java:143) - locked <0x0000000081acc610> (a org.drools.core.common.ProjectClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.drools.core.rule.JavaDialectRuntimeData$PackageClassLoader.loadClass(JavaDialectRuntimeData.java:646) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.mvel2.util.ParseTools.forNameWithInner(ParseTools.java:2187) at org.mvel2.ParserConfiguration.checkForDynamicImport(ParserConfiguration.java:169) at org.mvel2.ParserConfiguration.hasImport(ParserConfiguration.java:197) at org.mvel2.ParserContext.hasImport(ParserContext.java:373) at org.mvel2.compiler.AbstractParser.createPropertyToken(AbstractParser.java:1399) at org.mvel2.compiler.AbstractParser.nextToken(AbstractParser.java:903) at org.mvel2.compiler.ExpressionCompiler._compile(ExpressionCompiler.java:111) at org.mvel2.compiler.ExpressionCompiler.compile(ExpressionCompiler.java:52) at org.mvel2.MVEL.compileExpression(MVEL.java:810) at org.drools.core.base.mvel.MVELCompilationUnit.compile(MVELCompilationUnit.java:451) at org.drools.core.base.mvel.MVELCompilationUnit.getCompiledExpression(MVELCompilationUnit.java:273) at org.drools.core.rule.constraint.MvelConstraint.createMvelConditionEvaluator(MvelConstraint.java:269) at org.drools.core.rule.constraint.MvelConstraint.evaluate(MvelConstraint.java:249) at org.drools.core.rule.constraint.MvelConstraint.isAllowedCachedLeft(MvelConstraint.java:227) at org.drools.core.common.TripleBetaConstraints.isAllowedCachedLeft(TripleBetaConstraints.java:118) at org.drools.core.phreak.PhreakJoinNode.doLeftInserts(PhreakJoinNode.java:113) at org.drools.core.phreak.PhreakJoinNode.doNode(PhreakJoinNode.java:76) at org.drools.core.phreak.RuleNetworkEvaluator.switchOnDoBetaNode(RuleNetworkEvaluator.java:519) at org.drools.core.phreak.RuleNetworkEvaluator.evalBetaNode(RuleNetworkEvaluator.java:505) at org.drools.core.phreak.RuleNetworkEvaluator.evalNode(RuleNetworkEvaluator.java:341) at org.drools.core.phreak.RuleNetworkEvaluator.innerEval(RuleNetworkEvaluator.java:301) at org.drools.core.phreak.RuleNetworkEvaluator.outerEval(RuleNetworkEvaluator.java:136) at org.drools.core.phreak.RuleNetworkEvaluator.evaluateNetwork(RuleNetworkEvaluator.java:94) at org.drools.core.phreak.RuleExecutor.reEvaluateNetwork(RuleExecutor.java:194) at org.drools.core.phreak.RuleExecutor.evaluateNetworkAndFire(RuleExecutor.java:73) at org.drools.core.common.DefaultAgenda.fireNextItem(DefaultAgenda.java:970) at org.drools.core.common.DefaultAgenda.fireLoop(DefaultAgenda.java:1312) at org.drools.core.common.DefaultAgenda.fireAllRules(DefaultAgenda.java:1251) at org.drools.core.impl.StatefulKnowledgeSessionImpl.internalFireAllRules(StatefulKnowledgeSessionImpl.java:1359) at org.drools.core.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:1350) at org.drools.core.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:1331) at ActivityInferenceSession.fireRules(ActivityInferenceSession.java:261) at InferenceRunnable.doTryInference(InferenceRunnable.java:226) at InferenceRunnable.doInference(InferenceRunnable.java:129) at InferenceRunnable.run(InferenceRunnable.java:100) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)"InferenceThread-1": at org.drools.core.common.ProjectClassLoader$DefaultInternalTypesClassLoader.loadClass(ProjectClassLoader.java:385) - waiting to lock <0x0000000081acc610> (a org.drools.core.common.ProjectClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at com.sun.beans.finder.ClassFinder.findClass(ClassFinder.java:103) at java.beans.Introspector.findCustomizerClass(Introspector.java:1301) at java.beans.Introspector.getTargetBeanDescriptor(Introspector.java:1295) at java.beans.Introspector.getBeanInfo(Introspector.java:425) at java.beans.Introspector.getBeanInfo(Introspector.java:173) at org.apache.commons.beanutils.PropertyUtilsBean.getPropertyDescriptors(PropertyUtilsBean.java:980) at org.apache.commons.beanutils.PropertyUtilsBean.getPropertyDescriptors(PropertyUtilsBean.java:1084) at org.apache.commons.beanutils.PropertyUtilsBean.getPropertyDescriptor(PropertyUtilsBean.java:912) at org.apache.commons.beanutils.PropertyUtilsBean.getSimpleProperty(PropertyUtilsBean.java:1319) at org.apache.commons.beanutils.PropertyUtilsBean.getNestedProperty(PropertyUtilsBean.java:770) at org.apache.commons.beanutils.PropertyUtilsBean.getProperty(PropertyUtilsBean.java:846) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.getMethod(ReflectiveAccessorOptimizer.java:1120) at org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.getMethod(ReflectiveAccessorOptimizer.java:1003) at org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.compileGetChain(ReflectiveAccessorOptimizer.java:396) at org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.optimizeAccessor(ReflectiveAccessorOptimizer.java:163) at org.mvel2.ast.ASTNode.optimize(ASTNode.java:159) at org.mvel2.ast.ASTNode.getReducedValueAccelerated(ASTNode.java:115) at org.mvel2.ast.BinaryOperation.getReducedValueAccelerated(BinaryOperation.java:117) at org.mvel2.compiler.ExecutableAccessor.getValue(ExecutableAccessor.java:38) at org.mvel2.ast.Substatement.getReducedValueAccelerated(Substatement.java:44) at org.mvel2.ast.Or.getReducedValueAccelerated(Or.java:34) at org.mvel2.compiler.ExecutableAccessor.getValue(ExecutableAccessor.java:38) at org.mvel2.ast.Substatement.getReducedValueAccelerated(Substatement.java:44) at org.mvel2.ast.And.getReducedValueAccelerated(And.java:34) at org.mvel2.MVELRuntime.execute(MVELRuntime.java:85) at org.mvel2.compiler.CompiledExpression.getDirectValue(CompiledExpression.java:123) at org.mvel2.compiler.CompiledExpression.getValue(CompiledExpression.java:119) at org.mvel2.compiler.CompiledExpression.getValue(CompiledExpression.java:113) at org.mvel2.MVEL.executeExpression(MVEL.java:929) at org.drools.core.base.mvel.MVELEvalExpression.evaluate(MVELEvalExpression.java:104) at org.drools.core.rule.EvalCondition.isAllowed(EvalCondition.java:118) at org.drools.core.phreak.PhreakEvalNode.doLeftInserts(PhreakEvalNode.java:72) at org.drools.core.phreak.PhreakEvalNode.doNode(PhreakEvalNode.java:56) at org.drools.core.phreak.RuleNetworkEvaluator.evalNode(RuleNetworkEvaluator.java:349) at org.drools.core.phreak.RuleNetworkEvaluator.innerEval(RuleNetworkEvaluator.java:301) at org.drools.core.phreak.RuleNetworkEvaluator.outerEval(RuleNetworkEvaluator.java:136) at org.drools.core.phreak.RuleNetworkEvaluator.evaluateNetwork(RuleNetworkEvaluator.java:94) at org.drools.core.phreak.RuleExecutor.reEvaluateNetwork(RuleExecutor.java:194) at org.drools.core.phreak.RuleExecutor.evaluateNetworkAndFire(RuleExecutor.java:73) at org.drools.core.common.DefaultAgenda.fireNextItem(DefaultAgenda.java:970) at org.drools.core.common.DefaultAgenda.fireLoop(DefaultAgenda.java:1312) at org.drools.core.common.DefaultAgenda.fireAllRules(DefaultAgenda.java:1251) at org.drools.core.impl.StatefulKnowledgeSessionImpl.internalFireAllRules(StatefulKnowledgeSessionImpl.java:1359) at org.drools.core.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:1350) at org.drools.core.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:1331) at ActivityInferenceSession.fireRules(ActivityInferenceSession.java:261) at InferenceRunnable.doTryInference(InferenceRunnable.java:226) at InferenceRunnable.doInference(InferenceRunnable.java:129) at InferenceRunnable.run(InferenceRunnable.java:100) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Found 1 deadlock.
I know there has been other issues entered on deadlock with ProjectClassLoader and they have either been fixed in an earlier version or closed as not reproducible. Its hard to reproduce this as it has only happened in production a few times. In some of the rules we are running we are using PropertyUtilsBean as you can see in one of the deadlock threads so not sure if that is a cause for concern here or not but in our situation each thread has its own PropertyUtilsBean instance.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
3 years, 8 months
[JBoss JIRA] (WFWIP-343) Bootable JAR - FeaturePack element exposes misleading Galleon concepts
by Jean Francois Denise (Jira)
[ https://issues.redhat.com/browse/WFWIP-343?page=com.atlassian.jira.plugin... ]
Jean Francois Denise resolved WFWIP-343.
----------------------------------------
Resolution: Done
> Bootable JAR - FeaturePack element exposes misleading Galleon concepts
> ----------------------------------------------------------------------
>
> Key: WFWIP-343
> URL: https://issues.redhat.com/browse/WFWIP-343
> Project: WildFly WIP
> Issue Type: Bug
> Reporter: Jean Francois Denise
> Assignee: Jean Francois Denise
> Priority: Major
>
> When providing a list of feature-packs, each feature-pack item can be configured in a way that is not applicable to Bootable JAR. Furthermore some default values are not correct. This comes from the fack that the featurePack item is shared with the galleon generic maven plugin that offers more capabilities than the bootable JAR.
> In order to simplify configuration of feature-pack elements the following changes are required in the wildfly-jar-maven-plugin:
> * Remove inherit-configs item. It is implicitly false.
> * Make inherit-packages false by default.
> * Remove included-configs and excluded-configs elements. In a bootable JAR context a single configuration can be included.
> * Introduce "included-default-config"=[name of included config, eg:standalone-microprofile.xml]. Used to identify the default configuration to include in the bootable JAR.
> * Remove transitive attribute.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
3 years, 8 months
[JBoss JIRA] (JGRP-2369) Remove message indicating that UFC is not needed when using a TCP transport
by Bela Ban (Jira)
[ https://issues.redhat.com/browse/JGRP-2369?page=com.atlassian.jira.plugin... ]
Bela Ban closed JGRP-2369.
--------------------------
Resolution: Won't Fix
> Remove message indicating that UFC is not needed when using a TCP transport
> ----------------------------------------------------------------------------
>
> Key: JGRP-2369
> URL: https://issues.redhat.com/browse/JGRP-2369
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 4.0.11
> Environment: Red Hat JBoss Data Grid
> * 7.2 (which uses JGroups 4.0.11)
> Reporter: Jonathan Mason
> Assignee: Bela Ban
> Priority: Minor
>
> By default, protocol UFC is removed from the default TCP stack (in JDG). However, depending on the type of cache replication used (i.e., ASYNC), the UFC protocol may be of some benefit. Adding the UFC protocol into the TCP stack results in a message [1] indicating that it is not needed. This message should be removed.
> [1]
> INFO [org.jgroups.protocols.UFC] (MSC service thread 1-7) UFC is not needed (and can be removed) as we're running on a TCP transport
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
3 years, 8 months
[JBoss JIRA] (JGRP-2256) Connection.Receiver - Failed handling incoming message
by Bela Ban (Jira)
[ https://issues.redhat.com/browse/JGRP-2256?page=com.atlassian.jira.plugin... ]
Bela Ban closed JGRP-2256.
--------------------------
Resolution: Cannot Reproduce
> Connection.Receiver - Failed handling incoming message
> ------------------------------------------------------
>
> Key: JGRP-2256
> URL: https://issues.redhat.com/browse/JGRP-2256
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 4.0.10
> Reporter: Sibin Karnavar
> Assignee: Bela Ban
> Priority: Minor
>
> In AWS environment,
> I have not defined the port {color:red}41493{color} . I have configured TCP bind_port as 7803.
> But I can see from the below stack-trace that,
> Connection.Receiver [10.91.133.210:7803 - 10.91.135.64:{color:red}41493{color}]
> 1) Do I need to open/configure any other port ? I am wondering How 41493 is used when i have configured my bind_port as 7803.
> 2) What is the significance of client_bind_port for TCP. What happens if I dont configure it?
> 2018-03-05 19:07:28.833 ERROR 23012 --- [TQ-Bundler-9,ABC-SKS-stage_SJX_040320181613_XSJ,ip-10-91-133-210-46045] org.jgroups.protocols.TCP : - - JGRP000034: ip-10-91-133-210-46045: failure sending message to ip-10-91-135-64-18021: java.net.SocketTimeoutException: connect timed out
> 2018-03-05 19:07:29.834 WARN 23012 --- [localhost-startStop-1] org.jgroups.protocols.pbcast.GMS : - - ip-10-91-133-210-46045: JOIN(ip-10-91-133-210-46045) sent to ip-10-91-135-64-18021 timed out (after 3000 ms), on try 1
> 2018-03-05 19:07:32.842 ERROR 23012 --- [TQ-Bundler-9,ABC-SKS-stage_SJX_040320181613_XSJ,ip-10-91-133-210-46045] org.jgroups.protocols.TCP : - - JGRP000034: ip-10-91-133-210-46045: failure sending message to ip-10-91-135-64-18021: java.net.SocketTimeoutException: connect timed out
> 2018-03-05 19:07:32.889 WARN 23012 --- [localhost-startStop-1] org.jgroups.protocols.pbcast.GMS : - - ip-10-91-133-210-46045: JOIN(ip-10-91-133-210-46045) sent to ip-10-91-135-64-18021 timed out (after 3000 ms), on try 2
> 2018-03-05 19:07:35.944 WARN 23012 --- [localhost-startStop-1] org.jgroups.protocols.pbcast.GMS : - - ip-10-91-133-210-46045: JOIN(ip-10-91-133-210-46045) sent to ip-10-91-135-64-18021 timed out (after 3000 ms), on try 3
> 2018-03-05 19:07:36.848 ERROR 23012 --- [TQ-Bundler-9,ABC-SKS-stage_SJX_040320181613_XSJ,ip-10-91-133-210-46045] org.jgroups.protocols.TCP : - - JGRP000034: ip-10-91-133-210-46045: failure sending message to ip-10-91-135-64-18021: java.net.SocketTimeoutException: connect timed out
> 2018-03-05 19:07:38.999 WARN 23012 --- [localhost-startStop-1] org.jgroups.protocols.pbcast.GMS : - - ip-10-91-133-210-46045: JOIN(ip-10-91-133-210-46045) sent to ip-10-91-135-64-18021 timed out (after 3000 ms), on try 4
> 2018-03-05 19:07:40.854 ERROR 23012 --- [TQ-Bundler-9,ABC-SKS-stage_SJX_040320181613_XSJ,ip-10-91-133-210-46045] org.jgroups.protocols.TCP : - - JGRP000034: ip-10-91-133-210-46045: failure sending message to ip-10-91-135-64-18021: java.net.SocketTimeoutException: connect timed out
> 2018-03-05 19:07:42.053 WARN 23012 --- [localhost-startStop-1] org.jgroups.protocols.pbcast.GMS : - - ip-10-91-133-210-46045: JOIN(ip-10-91-133-210-46045) sent to ip-10-91-135-64-18021 timed out (after 3000 ms), on try 5
> 2018-03-05 19:07:44.860 ERROR 23012 --- [TQ-Bundler-9,ABC-SKS-stage_SJX_040320181613_XSJ,ip-10-91-133-210-46045] org.jgroups.protocols.TCP : - - JGRP000034: ip-10-91-133-210-46045: failure sending message to ip-10-91-135-64-18021: java.net.SocketTimeoutException: connect timed out
> 2018-03-05 19:07:45.108 WARN 23012 --- [localhost-startStop-1] org.jgroups.protocols.pbcast.GMS : - - ip-10-91-133-210-46045: JOIN(ip-10-91-133-210-46045) sent to ip-10-91-135-64-18021 timed out (after 3000 ms), on try 6
> 2018-03-05 19:07:48.163 WARN 23012 --- [localhost-startStop-1] org.jgroups.protocols.pbcast.GMS : - - ip-10-91-133-210-46045: JOIN(ip-10-91-133-210-46045) sent to ip-10-91-135-64-18021 timed out (after 3000 ms), on try 7
> 2018-03-05 19:07:48.866 ERROR 23012 --- [TQ-Bundler-9,ABC-SKS-stage_SJX_040320181613_XSJ,ip-10-91-133-210-46045] org.jgroups.protocols.TCP : - - JGRP000034: ip-10-91-133-210-46045: failure sending message to ip-10-91-135-64-18021: java.net.SocketTimeoutException: connect timed out
> 2018-03-05 19:07:51.218 WARN 23012 --- [localhost-startStop-1] org.jgroups.protocols.pbcast.GMS : - - ip-10-91-133-210-46045: JOIN(ip-10-91-133-210-46045) sent to ip-10-91-135-64-18021 timed out (after 3000 ms), on try 8
> 2018-03-05 19:07:51.218 WARN 23012 --- [localhost-startStop-1] org.jgroups.protocols.pbcast.GMS : - - ip-10-91-133-210-46045: too many JOIN attempts (8): becoming singleton
> 2018-03-05 19:07:52.041 ERROR 23012 --- [Connection.Receiver [10.91.133.210:7803 - 10.91.135.64:41493]-12,ABC-SKS-stage_SJX_040320181613_XSJ,ip-10-91-133-210-46045] org.jgroups.protocols.TCP : - - JGRP000030: ip-10-91-133-210-46045: failed handling incoming message
> java.io.IOException: Stream closed
> at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:269)
> at java.io.DataInputStream.readByte(DataInputStream.java:265)
> at org.jgroups.Message.readFrom(Message.java:724)
> at org.jgroups.util.Util.readMessageBatch(Util.java:1193)
> at org.jgroups.protocols.TP.handleMessageBatch(TP.java:1329)
> at org.jgroups.protocols.TP.receive(TP.java:1321)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
3 years, 8 months
[JBoss JIRA] (JGRP-2485) UDP is not working after upgarde to 3_6_19 from jgroups-3.4.0.Alpha2
by Bela Ban (Jira)
[ https://issues.redhat.com/browse/JGRP-2485?page=com.atlassian.jira.plugin... ]
Bela Ban closed JGRP-2485.
--------------------------
Resolution: Cannot Reproduce
> UDP is not working after upgarde to 3_6_19 from jgroups-3.4.0.Alpha2
> --------------------------------------------------------------------
>
> Key: JGRP-2485
> URL: https://issues.redhat.com/browse/JGRP-2485
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.6.19
> Reporter: Janardhan Naidu
> Assignee: Bela Ban
> Priority: Major
> Attachments: noapp.log.D20200619.T053202_NODE1, udp.xml
>
>
> Hi Team,
>
> we just upgraded from jgroups-3.4.0.Alpha2 to 3_6_19. post the UDP cluster communication is not working.
> after upgrade, we were hitting some warning as below and we solved those by changing property_string of UDP.
> *Warnings:*
> *WARNING: JGRP000014: Discovery.timeout has been deprecated: GMS.join_timeout should be used instead*
> [2020-06-17 14:05:49.271] ALL 000000000000 GLOBAL_SCOPE Jun 17, 2020 2:05:49 PM org.jgroups.stack.Configurator resolveAndAssignField
> *WARNING: JGRP000014: Discovery.num_initial_members has been deprecated: will be ignored*
> [2020-06-17 14:05:49.396] ALL 000000000000 GLOBAL_SCOPE Jun 17, 2020 2:05:49 PM org.jgroups.protocols.pbcast.NAKACK init
> *WARNING: use_mcast_xmit should not be used because the transport (TCP) does not support IP multicasting; setting use_mcast_xmit to false*
>
> To solve above warnings, we went through tcp.xml which was shipped using in 3_6_19 jar:
> and now our properties looks like:
> *property_string=UDP*(bind_addr=*hostIP*;bind_port=39061;mcast_addr=239.255.166.17;mcast_port=39060;ip_ttl=32;mcast_send_buf_size=150000;mcast_recv_buf_size=80000):PING:MERGE2(min_interval=5000;max_interval=10000):FD_ALL(interval=5000;timeout=20000):VERIFY_SUSPECT(timeout=1500):pbcast.NAKACK(use_mcast_xmit=true;retransmit_timeout=300,600,1200,2400,4800;discard_delivered_msgs=true):UNICAST:pbcast.STABLE(desired_avg_gossip=20000):FRAG(frag_size=8096):pbcast.GMS(join_timeout=5000;print_local_addr=true)
>
> *distribution_property_string=TCP*(bind_port=39060;thread_pool_rejection_policy=run):TCPPING(async_discovery=true;initial_hosts=*hostIP*[39060];port_range=0;):MERGE2(min_interval=3000;max_interval=5000):FD_SOCK:FD(timeout=5000;max_tries=48):VERIFY_SUSPECT(timeout=1500):pbcast.NAKACK(use_mcast_xmit=false;discard_delivered_msgs=true;):pbcast.STABLE(stability_delay=1000;desired_avg_gossip=20000;max_bytes=0):pbcast.GMS(join_timeout=5000;print_local_addr=true)
>
> *lock.protocolStack=UDP*(bind_addr=*hostIP*;bind_port=39062;mcast_addr=239.255.166.17;mcast_port=39069;ip_ttl=32;mcast_send_buf_size=150000;mcast_recv_buf_size=80000):PING:MERGE2(min_interval=5000;max_interval=10000):FD_ALL(interval=5000;timeout=20000):VERIFY_SUSPECT(timeout=1500):pbcast.NAKACK(use_mcast_xmit=true;retransmit_timeout=300,600,1200,2400,4800):UNICAST:pbcast.STABLE(desired_avg_gossip=20000):FRAG(frag_size=8096):pbcast.GMS(join_timeout=5000;print_local_addr=true):CENTRAL_LOCK(num_backups=2)
>
>
> With above properties, we are not seeing any warning or error. But still cluster communication is not working with UDP.
> Please help me in resolving the same
>
> Thank
> Janardhan
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
3 years, 8 months
[JBoss JIRA] (JGRP-2461) Clustering can fail when re-adding an existing node using TCP_NIO2
by Bela Ban (Jira)
[ https://issues.redhat.com/browse/JGRP-2461?page=com.atlassian.jira.plugin... ]
Bela Ban closed JGRP-2461.
--------------------------
Resolution: Won't Fix
> Clustering can fail when re-adding an existing node using TCP_NIO2
> ------------------------------------------------------------------
>
> Key: JGRP-2461
> URL: https://issues.redhat.com/browse/JGRP-2461
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 4.1.8
> Reporter: Robert Mitchell
> Assignee: Bela Ban
> Priority: Major
>
> When a node leaves a cluster and then later attempts to re-enter, a race condition can occur where the clustering fails to occur. Here is the sequence of events that seems to allow this to occur:
> # The rejoining node must have a "higher" IP address than the current cluster coordinator.
> # On the rejoin attempt, the coordinator sends a message to the rejoining node before the rejoining node sends to the coordinator using its prior address. I have seen this happen for two reasons:
> ## UNICAST3 is resending messages (which often happens with the final LEAVE_RSP from the prior cluster membership because it apparently does not get acked before the connection closes)
> ## TCPPING is sending a ping request to the cached prior address.
> # The connection gets established. It will then be used by the rejoining node whenever communicating with the cluster coordinator.
> # However, the cluster coordinator has this as the connection for the prior address. So the following happens whenever it wants to send a message to the rejoining node:
> ## It will attempt to create a new connection.
> ## The rejoining node will reject the connection as a redundant connection with its current connection taking precedence since it is coming from the same logical address as the "bad" connection.
> Since the messages needed to find and join the cluster or merge the two clusters are all unicast messages, the rejoining node will never get them and not be able to join until something happens that causes the initial connection to get closed.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
3 years, 8 months
[JBoss JIRA] (JGRP-2361) Error related to Jgroup and Database connection is getting reset
by Bela Ban (Jira)
[ https://issues.redhat.com/browse/JGRP-2361?page=com.atlassian.jira.plugin... ]
Bela Ban closed JGRP-2361.
--------------------------
Resolution: Cannot Reproduce
This is not JGroups itself, but Hybris. I suggest contact Hybris for help (I know they've done some work on upgrading JGroups in their product)
> Error related to Jgroup and Database connection is getting reset
> ----------------------------------------------------------------
>
> Key: JGRP-2361
> URL: https://issues.redhat.com/browse/JGRP-2361
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.6.11
> Environment: Hybris running on tomcat - Centos 7
> Reporter: karthikeyan Aruljothi
> Assignee: Bela Ban
> Priority: Major
> Attachments: Jgroup error in preprod-000.txt, Jgroup node configuration.txt, Jgroups blocking and terminating connection.txt, Jgroups error in console.txt, error Jgroups.txt, jgroups-tcp.xml
>
>
> Hi ,
> we are facing an issue with our cluster configuration and due to this JVM responding time also takes more time, after clearing the cache / restarting all nodes application works as expected.
> When issue arises one of the core occupies 100% cpu utilization then it confirms to restart the server otherwise it never process any request. Below is our configuration in local.properties. Also providing error logs as attachment. could see error in logs related to Jgroups blocking and connection getting terminated between nodes.
> Let us know your valuable inputs, on what exactly the issue i.e causing the slowness then blocking the whole server.
> Attached cluster configuration for each nodes and error logs
> Adding to this we are getting below error while doing deployment/restarting of servers
> WARN [localhost-startStop-1] [GMS] hybrisnode-0: JOIN(hybrisnode-0) sent to hybrisnode-2 timed out (after 3000 ms), on try 3
> WARN [pool-3-thread-1] [GMS] hybrisnode-3: JOIN(hybrisnode-3) sent to hybrisnode-1 timed out (after 3000 ms), on try 4
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
3 years, 8 months
[JBoss JIRA] (JGRP-2399) Problem with loading resources from the bundle root (/) in OSGI environment
by Bela Ban (Jira)
[ https://issues.redhat.com/browse/JGRP-2399?page=com.atlassian.jira.plugin... ]
Bela Ban resolved JGRP-2399.
----------------------------
Resolution: Done
> Problem with loading resources from the bundle root (/) in OSGI environment
> ----------------------------------------------------------------------------
>
> Key: JGRP-2399
> URL: https://issues.redhat.com/browse/JGRP-2399
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.6.19
> Environment: Reproducible both on linux(centos 7.3) and windows 10, my project is using org.eclipse.equinox 3.6.0.v20100503, oracle jdk 1.8.0.212
> Reporter: Orlin Stalyanov
> Assignee: Bela Ban
> Priority: Major
>
> {color:#0747A6}Util.java:108
> resource_bundle=ResourceBundle.getBundle("jg-messages",Locale.getDefault(),Util.class.getClassLoader());{color}
> is crashing with:
> ...
> {color:#0747A6}Caused by: java.util.MissingResourceException: Can't find bundle for base name jg-messages, locale en_US
> at java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:1581)
> at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1396)
> at java.util.ResourceBundle.getBundle(ResourceBundle.java:1091)
> at org.jgroups.util.Util.<clinit>(Util.java:108)
> ... 162 more{color}
> found some relevant info here : https://stackoverflow.com/questions/7564370/importing-resources-from-osgi...
> seems like OSGI doesn't really like resources residing in the bundle root ...
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
3 years, 8 months