[jboss-jira] [JBoss JIRA] (DROOLS-2409) Memory Leak in sliding window
Mario Fusco (JIRA)
issues at jboss.org
Thu Mar 22 05:57:03 EDT 2018
[ https://issues.jboss.org/browse/DROOLS-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13549601#comment-13549601 ]
Mario Fusco edited comment on DROOLS-2409 at 3/22/18 5:57 AM:
--------------------------------------------------------------
I gave a run at your reproducer and I see your point. The problem there is that inserting events in a so tight loop has the effect of overwhelming the engine not giving it a chance of getting any job done. In fact in this case not only the expiration mechanism is not working but also the accumulate rule is not firing at all and your reproducer generates an output as the following:
{code}
3500000 inserted into list of 2702077
3510000 inserted into list of 2712077
3520000 inserted into list of 2722077
3530000 inserted into list of 2732077
3540000 inserted into list of 2742077
3550000 inserted into list of 2752077
3560000 inserted into list of 2762077
3570000 inserted into list of 2772077
3580000 inserted into list of 2782077
3590000 inserted into list of 2792077
3600000 inserted into list of 2802077
3610000 inserted into list of 2812077
{code}
However I believe that your use case is quite unrealistic. I cannot imagine any real-world application that keeps inserting events forever in a tight all in memory for-loop as in your example. More commonly those events will come for an event bus or a similar data source which at best will be an order of magnitude slower of your for-loop. I expect that Drools will work without any problem in this situation. Indeed just adding a 1ms sleep every 20 inserted events:
{code}
if (i % 20 == 0) {
Thread.sleep( 1L );
}
{code}
gives the engine enough time to do its job producing the following output:
{code}
3500000 inserted into list of 20584
zzZZzz: 49.04286510590858
zzZZzz: 49.040446927374305
zzZZzz: 49.04486607142857
3510000 inserted into list of 15737
zzZZzz: 49.578387458006716
zzZZzz: 49.5359217877095
zzZZzz: 49.540670391061454
3520000 inserted into list of 11409
3530000 inserted into list of 21409
zzZZzz: 49.26767676767677
zzZZzz: 49.34880410022779
zzZZzz: 49.35626423690205
3540000 inserted into list of 16670
zzZZzz: 49.279316338354576
zzZZzz: 49.29466357308585
zzZZzz: 49.29895591647332
3550000 inserted into list of 12099
3560000 inserted into list of 22099
zzZZzz: 49.537780269058295
zzZZzz: 49.60458612975391
zzZZzz: 49.60173184357542
3570000 inserted into list of 18351
zzZZzz: 49.64232062780269
zzZZzz: 49.62682885811985
zzZZzz: 49.62618287698079
3580000 inserted into list of 15067
zzZZzz: 49.538758389261744
zzZZzz: 49.51251396648045
zzZZzz: 49.503407821229054
3590000 inserted into list of 12190
3600000 inserted into list of 22190
zzZZzz: 49.426733622975796
zzZZzz: 49.446553672316384
zzZZzz: 49.44441309255079
3610000 inserted into list of 19332
{code]
As you can see in this case the engine is now having time to fire the rule and by doing so it is also evicting the expired objects from the working memory as expected.
I'm not saying that we don't have a problem here. I'll keep investigating your test case to check if I can make this work in a more reliable way even in this extreme scenario. Any further feedback is welcome.
was (Author: mfusco):
I gave a run at your reproducer and I see your point. The problem there is that inserting events in a so tight loop has the effect of overwhelming the engine not giving it a chance of getting any job done. In fact in this case not only the expiration mechanism is not working but also the accumulate rule is not firing at all and your reproducer generates an output as the following:
{code}
3500000 inserted into list of 2702077
3510000 inserted into list of 2712077
3520000 inserted into list of 2722077
3530000 inserted into list of 2732077
3540000 inserted into list of 2742077
3550000 inserted into list of 2752077
3560000 inserted into list of 2762077
3570000 inserted into list of 2772077
3580000 inserted into list of 2782077
3590000 inserted into list of 2792077
3600000 inserted into list of 2802077
3610000 inserted into list of 2812077
{code}
However I believe that your use case is quite unrealistic. I cannot imagine any real-world application that keeps inserting events forever in a tight all in memory for-loop as in your example. More commonly those events will come for an event bus or a similar data source which at best will be an order of magnitude slower of your for-loop. I expect that Drools will work without any problem in this situation. Indeed just adding a 1ms sleep every 20 inserted events:
{code}
if (i % 20 == 0) {
Thread.sleep( 1L );
}
{code}
gives the engine enough time to do its job producing the following output:
{code}
3500000 inserted into list of 20584
zzZZzz: 49.04286510590858
zzZZzz: 49.040446927374305
zzZZzz: 49.04486607142857
3510000 inserted into list of 15737
zzZZzz: 49.578387458006716
zzZZzz: 49.5359217877095
zzZZzz: 49.540670391061454
3520000 inserted into list of 11409
3530000 inserted into list of 21409
zzZZzz: 49.26767676767677
zzZZzz: 49.34880410022779
zzZZzz: 49.35626423690205
3540000 inserted into list of 16670
zzZZzz: 49.279316338354576
zzZZzz: 49.29466357308585
zzZZzz: 49.29895591647332
3550000 inserted into list of 12099
3560000 inserted into list of 22099
zzZZzz: 49.537780269058295
zzZZzz: 49.60458612975391
zzZZzz: 49.60173184357542
3570000 inserted into list of 18351
zzZZzz: 49.64232062780269
zzZZzz: 49.62682885811985
zzZZzz: 49.62618287698079
3580000 inserted into list of 15067
zzZZzz: 49.538758389261744
zzZZzz: 49.51251396648045
zzZZzz: 49.503407821229054
3590000 inserted into list of 12190
3600000 inserted into list of 22190
zzZZzz: 49.426733622975796
zzZZzz: 49.446553672316384
zzZZzz: 49.44441309255079
3610000 inserted into list of 19332
{code]
As you can see in this case the engine is now having time to fire the rule and by doing so it is also evicting the expired objects from the working memory as expected.
I'm not saying that we don't have a problem here. I'll keep investigating your test case to check if I can make this work in a more reliable way even in this extreme scenario. Any further feedback is welcome.
> Memory Leak in sliding window
> -----------------------------
>
> Key: DROOLS-2409
> URL: https://issues.jboss.org/browse/DROOLS-2409
> Project: Drools
> Issue Type: Feature Request
> Components: core engine
> Affects Versions: 7.6.0.Final
> Environment: Mac with Java 1.8 using Spring Boot (see attached demo project)
> Reporter: Mike Baranski
> Assignee: Mario Fusco
> Priority: Critical
> Attachments: complete.tar.gz
>
>
> See attached project.
> `gradle test` causes an out of memory exception on my machine after ~2.6 million events. Events should be released for GC after 1 second, but that does not appear to be happening.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
More information about the jboss-jira
mailing list