[jboss-jira] [JBoss JIRA] (JGRP-2396) increasing networkdata, cpu and heap

[Rob van der Boom (Jira)]

    [ https://issues.jboss.org/browse/JGRP-2396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13810074#comment-13810074 ] 

Rob van der Boom edited comment on JGRP-2396 at 11/8/19 8:31 AM:
-----------------------------------------------------------------

True many components involved.
I will deliver more details next week, heap dump etc.
Its more tracing down what it can and can not be, ofcourse i am not sure, only things i no so far:
- there are no know keycloak related issues so far and it looks like relating to infinispan cache replication between the nodes. The issue increases/ show up only when having many cached sessions (>300.000) but it is NOT related to activity since it keeps increasing with same amount even at night when almost no traffic on the site.
- Its is only the TQ bundler taken up more and more cpu not other tasks (unless heap grows towards max. but since we doubled heap GC doesnt grow above 1% cpu when tq bundler is already into the 30% and quickly after higher. Network data transfer (also when almost no trafic on site is growing as fast as the cpu to levels we can not explain since). 
- zero errors occure anywhere that can explain issues.

So will try to hand over more details,  


was (Author: robvanderboom):
True many components involved.
I will deliver more details next week, heap dump etc.
Its more tracing down what it can and can not be, ofcourse i am not sure, only things i no so far:
- there are no know keycloak related issues so far and it looks like relating to infinispan cache replication between the nodes. The issue increases/ show up only when having many cached sessions (>300.000) but it is NOT related to activity since it keeps increasing with same amount even at night when almost no traffic on the site.
- Its is only the TQ bundler taken up more and more cpu not other tasks (unless heap grows towards max. but since we doubled heap GC doesnt grow above 1% cpu when tq bundler is already into the 30% and quickly after higher. Network data transfer (also when almost no trafic on site is growing as fast as the cpu to levels we can not explain since). 
-0 errors occure anywhere that can explain issues.

So will try to hand over more details,  

> increasing networkdata, cpu and heap
> ------------------------------------
>
>                 Key: JGRP-2396
>                 URL: https://issues.jboss.org/browse/JGRP-2396
>             Project: JGroups
>          Issue Type: Bug
>    Affects Versions: 4.0.19
>            Reporter: Rob van der Boom
>            Assignee: Bela Ban
>            Priority: Major
>
> hey,
> we have an keycloak (sso) setup, version 7.0.1 running in kubernetes - aws.
> Its build on wildfly 17, infinispan 9.4 and jgroups 4.0.19.
> We have 3 pods running in standalone-ha with cache setup on distribution (all 3 nodes - so equivalent to replication)
> ISSUE:
> We see a slowly growing of networkstatistics, heap and cpu, while the number of sessions in keycloak (cached) remain almost stable.
> The cpu growth is caused by the TQbundler process, which explaines the networkdata growth. It looks like this is causing also a memory leakage.. 
> every 5 days we have to restart the pods and then every resets to a very low level including the heap. this while all sessions are still valid and cached.
> The only issue i could find maybe related to this is:
> https://issues.jboss.org/browse/JGRP-2382?jql=project%20%3D%20JGRP%20AND%20text%20~%20leak
> Could this be the same issue and does it also cause increasing network and cpu (since that is why we have to restart, the heap has much space left !).
> And if so how does this issue continue since for us its a major issue.
> We als had this issue already in keycloak 5 (wildfly 15), thats why we upgraded to the latest available version.



--
This message was sent by Atlassian Jira
(v7.13.8#713008)