[JBoss JIRA] (JGRP-1980) Improve throughput under contention for FlowControl.Credit
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1980?page=com.atlassian.jira.plugin.... ]
Bela Ban edited comment on JGRP-1980 at 1/21/16 12:15 PM:
----------------------------------------------------------
Possible improvements in {{UFC}} ({{MFC}} is not really used by Infinispan, therefore we'll look at it later):
* {{FlowControl$Credit.needsToSendCreditRequest()}} uses the same lock as {{decrementIfEnoughCredits()}} and {{increment()}}. Since this method is called *for every sent message*, using a separate lock will remove roughly 50% of the lock access:
** Every send calls {{decrementIfEnoughCredits()}}: ca. 50%
** Every send calls {{needsToSendCreditRequest()}}: ca. 50%
** A replenishment calls {{increment()}}: ca. 2%. This is not called frequently and depends on the max number of credits, the size of the messages and the number of sender threads
Before implementing any of the above suggestions, a perf test needs to be written, so that we can judge whether a change really improves performance.
was (Author: belaban):
Possible improvements in {{UFC}} ({{MFC}} is not really used by Infinispan, therefore we'll look at it later):
* {{FlowControl$Credit.needsToSendCreditRequest()}} uses the same lock as {{decrementIfEnoughCredits()}} and {{increment()}}. Since this method is called *for every sent message*, using a separate lock will remove roughly 50% of the lock access:
** Every send calls {{decrementIfEnoughCredits()}}: ca. 50%
** Every send calls {{needsToSendCreditRequest()}}: ca. 50%
** A replenishment calls {{increment()}}: ca. 2%. This is not called frequently and depends on the max number of credits, the size of the messages and the number of sender threads
> Improve throughput under contention for FlowControl.Credit
> ----------------------------------------------------------
>
> Key: JGRP-1980
> URL: https://issues.jboss.org/browse/JGRP-1980
> Project: JGroups
> Issue Type: Enhancement
> Affects Versions: 3.6.6
> Reporter: Sanne Grinovero
> Assignee: Bela Ban
> Fix For: 3.6.8
>
> Attachments: bla.java
>
>
> The methods {{org.jgroups.protocols.FlowControl.Credit.decrementIfEnoughCredits(long, long)}} and {{org.jgroups.protocols.FlowControl.Credit.decrementAndGet(long)}} are contending the locks on class synchronization when stress testing JGroups.
> Wondering if we can think of polishing the implementation, although it looks like challenging.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 3 months
[JBoss JIRA] (JGRP-1980) Improve throughput under contention for FlowControl.Credit
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1980?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-1980:
--------------------------------
Possible improvements in {{UFC}} ({{MFC}} is not really used by Infinispan, therefore we'll look at it later):
* {{FlowControl$Credit.needsToSendCreditRequest()}} uses the same lock as {{decrementIfEnoughCredits()}} and {{increment()}}. Since this method is called *for every sent message*, using a separate lock will remove roughly 50% of the lock access:
** Every send calls {{decrementIfEnoughCredits()}}: ca. 50%
** Every send calls {{needsToSendCreditRequest()}}: ca. 50%
** A replenishment calls {{increment()}}: ca. 2%. This is not called frequently and depends on the max number of credits, the size of the messages and the number of sender threads
> Improve throughput under contention for FlowControl.Credit
> ----------------------------------------------------------
>
> Key: JGRP-1980
> URL: https://issues.jboss.org/browse/JGRP-1980
> Project: JGroups
> Issue Type: Enhancement
> Affects Versions: 3.6.6
> Reporter: Sanne Grinovero
> Assignee: Bela Ban
> Fix For: 3.6.8
>
> Attachments: bla.java
>
>
> The methods {{org.jgroups.protocols.FlowControl.Credit.decrementIfEnoughCredits(long, long)}} and {{org.jgroups.protocols.FlowControl.Credit.decrementAndGet(long)}} are contending the locks on class synchronization when stress testing JGroups.
> Wondering if we can think of polishing the implementation, although it looks like challenging.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 3 months
[JBoss JIRA] (WFLY-6044) Relative path names in jsp includes do not resolve properly
by Tomaz Cerar (JIRA)
[ https://issues.jboss.org/browse/WFLY-6044?page=com.atlassian.jira.plugin.... ]
Tomaz Cerar moved JBEAP-2957 to WFLY-6044:
------------------------------------------
Project: WildFly (was: JBoss Enterprise Application Platform)
Key: WFLY-6044 (was: JBEAP-2957)
Workflow: GIT Pull Request workflow (was: CDW with loose statuses v1)
Target Release: (was: 7.0.0.GA)
Affects Version/s: 10.0.0.CR5
(was: 7.0.0.ER2 (Beta))
> Relative path names in jsp includes do not resolve properly
> -----------------------------------------------------------
>
> Key: WFLY-6044
> URL: https://issues.jboss.org/browse/WFLY-6044
> Project: WildFly
> Issue Type: Bug
> Affects Versions: 10.0.0.CR5
> Reporter: Tomaz Cerar
> Assignee: Tomaz Cerar
> Priority: Blocker
>
> Hi, I have noticed that jsp:include behaves differently in EAP 7.0.0.ER2 than in WildFly 10.0.0.CR4.
>
> To reproduce this, use kitchensink-jsp quickstart (1) and in index.jsp replace
> {code}
> <%@ include file="registrationForm.jsp"%>
> {code}
> with
> {code}
> <jsp:include page="./registrationForm.jsp"/>
> {code}
> With this modification the quickstart no longer works on EAP 7.0.0.ER2 but it still works on WildFly 10.0.0.CR4 and EAP 7.0.0.Alpha.
> (1) https://github.com/trepel/jboss-eap-quickstarts/tree/7.0.x-develop/kitche...
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 3 months
[JBoss JIRA] (DROOLS-1030) Nullpointer in JpaTimerJobInstance.call on startup
by Artur Kronenberg (JIRA)
[ https://issues.jboss.org/browse/DROOLS-1030?page=com.atlassian.jira.plugi... ]
Artur Kronenberg edited comment on DROOLS-1030 at 1/21/16 12:00 PM:
--------------------------------------------------------------------
Ok,
I think I have the issue (or some issue) reproduced. I will attach this in a second. Here is what I do:
1. The standalone has 2 methods; create and load. Load takes the id from the create run to load the thing. I made the code as close as I could to where I found the bug/feature in the first place. The exception is not 100% the same, but I think I may have been hunting a red herring. I think those exceptions are all related, but they might happen at a slightly different point in time - so different symptoms with maybe the same root.
2. First run: create
Just run the main, wait until the thing is executed and then shut down the run. This is important as I think it has something to do with persistence + expiry which means the run needs to terminate so that drools can't update the session anymore (timerjobs and friends I believe).
3. Wait for 1-2 minutes. I recommend 9gag, cracks me up :D
4. Change load to take the ID of the run in step 2. Comment out the create method, and run the same thing again.
This has worked for me 3 out of 3 times with an exception talking about transaction and stuff: http://pastebin.com/RqKr3FcE
Observations:
1. Load under 1 minute wait time. (so immediately)
Everything works fine. It appears no timer jobs are started and no exceptions are thrown.
2. Load after 1 minute wait
The 1k models that event 1 loads are set to expire after 1 minute.
When loading the session, it appears that for each fact that ought to be expired, 1 JpaTimerJob is started. The logs show heeeaaaapps of hibernate operations going on. I suspect that all the jobs try to do their thing and there may or may not be a bunch of retrying happening in the background.
Conclusion is that I am still not sure what is happening exactly. But the signs stand in a way that I suspect loading a session which has facts/events that are supposed to be expired seems to be what the problem is. Please let me know if there's anything else I can do. If I manage to reproduce the NPE properly I will do that. Sadly the exception of my actual system vs my standalone example are slightly wrong.
Note: I believe it is essential for this exception that:
* Facts/events are inserted in a transaction - see TransactionHelper class
* fireAllRules is called in a separate thread
Update:
Maybe this is off some value. I don't think this is correct behaviour. After re-running (loading) the same session 3 or 4 times, the update count (OPTLOCK) on that session skyrocketed:
MariaDB [drools]> select id, OPTLOCK from SEssioninfo;
+----+---------+
| id | OPTLOCK |
+----+---------+
| 89 | 12923 |
+----+---------+
1 rows in set (0.00 sec)
was (Author: pandaadb):
Ok,
I think I have the issue (or some issue) reproduced. I will attach this in a second. Here is what I do:
1. The standalone has 2 methods; create and load. Load takes the id from the create run to load the thing. I made the code as close as I could to where I found the bug/feature in the first place. The exception is not 100% the same, but I think I may have been hunting a red herring. I think those exceptions are all related, but they might happen at a slightly different point in time - so different symptoms with maybe the same root.
2. First run: create
Just run the main, wait until the thing is executed and then shut down the run. This is important as I think it has something to do with persistence + expiry which means the run needs to terminate so that drools can't update the session anymore (timerjobs and friends I believe).
3. Wait for 1-2 minutes. I recommend 9gag, cracks me up :D
4. Change load to take the ID of the run in step 2. Comment out the create method, and run the same thing again.
This has worked for me 3 out of 3 times with an exception talking about transaction and stuff: http://pastebin.com/RqKr3FcE
Observations:
1. Load under 1 minute wait time. (so immediately)
Everything works fine. It appears no timer jobs are started and no exceptions are thrown.
2. Load after 1 minute wait
The 1k models that event 1 loads are set to expire after 1 minute.
When loading the session, it appears that for each fact that ought to be expired, 1 JpaTimerJob is started. The logs show heeeaaaapps of hibernate operations going on. I suspect that all the jobs try to do their thing and there may or may not be a bunch of retrying happening in the background.
Conclusion is that I am still not sure what is happening exactly. But the signs stand in a way that I suspect loading a session which has facts/events that are supposed to be expired seems to be what the problem is. Please let me know if there's anything else I can do. If I manage to reproduce the NPE properly I will do that. Sadly the exception of my actual system vs my standalone example are slightly wrong.
Note: I believe it is essential for this exception that:
* Facts/events are inserted in a transaction - see TransactionHelper class
* fireAllRules is called in a separate thread
> Nullpointer in JpaTimerJobInstance.call on startup
> ---------------------------------------------------
>
> Key: DROOLS-1030
> URL: https://issues.jboss.org/browse/DROOLS-1030
> Project: Drools
> Issue Type: Feature Request
> Components: core engine
> Environment: Mac OS 10.10.5, Eclipse Mars Release, Java 1.8, Drools 6.3.0.Final
> Reporter: Artur Kronenberg
> Assignee: Mario Fusco
> Attachments: test-standalone.zip
>
>
> Hi,
> I have noticed Nullpointer exceptions when I try and reload a persisted session on startup. It is a bit hard to recreate (I am actually not managing it now since I deleted all old sessions to attempt recreation) but I figured maybe someone has an idea. Essentially I am getting this NPE twice:
> java.lang.NullPointerException: null
> at org.drools.persistence.jpa.JpaTimerJobInstance.call(JpaTimerJobInstance.java:50) [drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.persistence.jpa.JpaTimerJobInstance.call(JpaTimerJobInstance.java:30) [drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_51]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_51]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_51]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_51]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_51]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> When debugging I noticed the following behaviour which points to a race condition:
> If I start my server and try to recreate the session, those NPEs happen 2x.
> If I start my server and put a breakpoint BEFORE creating the CommandService for execution, I can wait for a few seconds and the service can be found.
> It appears that the scheduler's timerJobFactoryManager is not fully there at the time the sessions is being loaded? Or something else is racing with the service creation.
> Is there a way for me to make sure everything is fully instantiated before I use drools? Does a workaround exist? Is this even a bug?
> Thanks and let me know your thoughts. If I run into this again I will attempt to try and reproduce it. Let me know if more info is needed!
> UPDATE:
> I noticed that the reason I can't reproduce it at the moment is that the JpaTimerJobInstance is never called with the newly persisted session. It appears that that timer job depends on something else to happen so that it thinks it needs to start the timerJob
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 3 months
[JBoss JIRA] (DROOLS-1030) Nullpointer in JpaTimerJobInstance.call on startup
by Artur Kronenberg (JIRA)
[ https://issues.jboss.org/browse/DROOLS-1030?page=com.atlassian.jira.plugi... ]
Artur Kronenberg updated DROOLS-1030:
-------------------------------------
Attachment: test-standalone.zip
Standalone example for hopefully this issue.
> Nullpointer in JpaTimerJobInstance.call on startup
> ---------------------------------------------------
>
> Key: DROOLS-1030
> URL: https://issues.jboss.org/browse/DROOLS-1030
> Project: Drools
> Issue Type: Feature Request
> Components: core engine
> Environment: Mac OS 10.10.5, Eclipse Mars Release, Java 1.8, Drools 6.3.0.Final
> Reporter: Artur Kronenberg
> Assignee: Mario Fusco
> Attachments: test-standalone.zip
>
>
> Hi,
> I have noticed Nullpointer exceptions when I try and reload a persisted session on startup. It is a bit hard to recreate (I am actually not managing it now since I deleted all old sessions to attempt recreation) but I figured maybe someone has an idea. Essentially I am getting this NPE twice:
> java.lang.NullPointerException: null
> at org.drools.persistence.jpa.JpaTimerJobInstance.call(JpaTimerJobInstance.java:50) [drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.persistence.jpa.JpaTimerJobInstance.call(JpaTimerJobInstance.java:30) [drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_51]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_51]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_51]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_51]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_51]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> When debugging I noticed the following behaviour which points to a race condition:
> If I start my server and try to recreate the session, those NPEs happen 2x.
> If I start my server and put a breakpoint BEFORE creating the CommandService for execution, I can wait for a few seconds and the service can be found.
> It appears that the scheduler's timerJobFactoryManager is not fully there at the time the sessions is being loaded? Or something else is racing with the service creation.
> Is there a way for me to make sure everything is fully instantiated before I use drools? Does a workaround exist? Is this even a bug?
> Thanks and let me know your thoughts. If I run into this again I will attempt to try and reproduce it. Let me know if more info is needed!
> UPDATE:
> I noticed that the reason I can't reproduce it at the moment is that the JpaTimerJobInstance is never called with the newly persisted session. It appears that that timer job depends on something else to happen so that it thinks it needs to start the timerJob
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 3 months
[JBoss JIRA] (DROOLS-1030) Nullpointer in JpaTimerJobInstance.call on startup
by Artur Kronenberg (JIRA)
[ https://issues.jboss.org/browse/DROOLS-1030?page=com.atlassian.jira.plugi... ]
Artur Kronenberg commented on DROOLS-1030:
------------------------------------------
Ok,
I think I have the issue (or some issue) reproduced. I will attach this in a second. Here is what I do:
1. The standalone has 2 methods; create and load. Load takes the id from the create run to load the thing. I made the code as close as I could to where I found the bug/feature in the first place. The exception is not 100% the same, but I think I may have been hunting a red herring. I think those exceptions are all related, but they might happen at a slightly different point in time - so different symptoms with maybe the same root.
2. First run: create
Just run the main, wait until the thing is executed and then shut down the run. This is important as I think it has something to do with persistence + expiry which means the run needs to terminate so that drools can't update the session anymore (timerjobs and friends I believe).
3. Wait for 1-2 minutes. I recommend 9gag, cracks me up :D
4. Change load to take the ID of the run in step 2. Comment out the create method, and run the same thing again.
This has worked for me 3 out of 3 times with an exception talking about transaction and stuff: http://pastebin.com/RqKr3FcE
Observations:
1. Load under 1 minute wait time. (so immediately)
Everything works fine. It appears no timer jobs are started and no exceptions are thrown.
2. Load after 1 minute wait
The 1k models that event 1 loads are set to expire after 1 minute.
When loading the session, it appears that for each fact that ought to be expired, 1 JpaTimerJob is started. The logs show heeeaaaapps of hibernate operations going on. I suspect that all the jobs try to do their thing and there may or may not be a bunch of retrying happening in the background.
Conclusion is that I am still not sure what is happening exactly. But the signs stand in a way that I suspect loading a session which has facts/events that are supposed to be expired seems to be what the problem is. Please let me know if there's anything else I can do. If I manage to reproduce the NPE properly I will do that. Sadly the exception of my actual system vs my standalone example are slightly wrong.
Note: I believe it is essential for this exception that:
* Facts/events are inserted in a transaction - see TransactionHelper class
* fireAllRules is called in a separate thread
> Nullpointer in JpaTimerJobInstance.call on startup
> ---------------------------------------------------
>
> Key: DROOLS-1030
> URL: https://issues.jboss.org/browse/DROOLS-1030
> Project: Drools
> Issue Type: Feature Request
> Components: core engine
> Environment: Mac OS 10.10.5, Eclipse Mars Release, Java 1.8, Drools 6.3.0.Final
> Reporter: Artur Kronenberg
> Assignee: Mario Fusco
>
> Hi,
> I have noticed Nullpointer exceptions when I try and reload a persisted session on startup. It is a bit hard to recreate (I am actually not managing it now since I deleted all old sessions to attempt recreation) but I figured maybe someone has an idea. Essentially I am getting this NPE twice:
> java.lang.NullPointerException: null
> at org.drools.persistence.jpa.JpaTimerJobInstance.call(JpaTimerJobInstance.java:50) [drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.persistence.jpa.JpaTimerJobInstance.call(JpaTimerJobInstance.java:30) [drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_51]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_51]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_51]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_51]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_51]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> When debugging I noticed the following behaviour which points to a race condition:
> If I start my server and try to recreate the session, those NPEs happen 2x.
> If I start my server and put a breakpoint BEFORE creating the CommandService for execution, I can wait for a few seconds and the service can be found.
> It appears that the scheduler's timerJobFactoryManager is not fully there at the time the sessions is being loaded? Or something else is racing with the service creation.
> Is there a way for me to make sure everything is fully instantiated before I use drools? Does a workaround exist? Is this even a bug?
> Thanks and let me know your thoughts. If I run into this again I will attempt to try and reproduce it. Let me know if more info is needed!
> UPDATE:
> I noticed that the reason I can't reproduce it at the moment is that the JpaTimerJobInstance is never called with the newly persisted session. It appears that that timer job depends on something else to happen so that it thinks it needs to start the timerJob
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 3 months
[JBoss JIRA] (JGRP-1980) Improve throughput under contention for FlowControl.Credit
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1980?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-1980:
--------------------------------
There's very little contention: mainly on the sender side, between sender threads decrementing (and possibly blocking) credits and credits getting refilled. Sender threads also contend with other sender threads.
If this turns out to be a major bottleneck, we can always revisit it.
> Improve throughput under contention for FlowControl.Credit
> ----------------------------------------------------------
>
> Key: JGRP-1980
> URL: https://issues.jboss.org/browse/JGRP-1980
> Project: JGroups
> Issue Type: Enhancement
> Affects Versions: 3.6.6
> Reporter: Sanne Grinovero
> Assignee: Bela Ban
> Fix For: 3.6.8
>
> Attachments: bla.java
>
>
> The methods {{org.jgroups.protocols.FlowControl.Credit.decrementIfEnoughCredits(long, long)}} and {{org.jgroups.protocols.FlowControl.Credit.decrementAndGet(long)}} are contending the locks on class synchronization when stress testing JGroups.
> Wondering if we can think of polishing the implementation, although it looks like challenging.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 3 months
[JBoss JIRA] (JGRP-1980) Improve throughput under contention for FlowControl.Credit
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1980?page=com.atlassian.jira.plugin.... ]
Bela Ban updated JGRP-1980:
---------------------------
Attachment: bla.java
Simple implementation of lockless credits scheme for the receiver. Investigate whether LongAdder (in JDK 8) improves this.
> Improve throughput under contention for FlowControl.Credit
> ----------------------------------------------------------
>
> Key: JGRP-1980
> URL: https://issues.jboss.org/browse/JGRP-1980
> Project: JGroups
> Issue Type: Enhancement
> Affects Versions: 3.6.6
> Reporter: Sanne Grinovero
> Assignee: Bela Ban
> Fix For: 3.6.8
>
> Attachments: bla.java
>
>
> The methods {{org.jgroups.protocols.FlowControl.Credit.decrementIfEnoughCredits(long, long)}} and {{org.jgroups.protocols.FlowControl.Credit.decrementAndGet(long)}} are contending the locks on class synchronization when stress testing JGroups.
> Wondering if we can think of polishing the implementation, although it looks like challenging.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 3 months