[
https://issues.jboss.org/browse/JGRP-2396?page=com.atlassian.jira.plugin....
]
Rob van der Boom edited comment on JGRP-2396 at 11/8/19 8:31 AM:
-----------------------------------------------------------------
True many components involved.
I will deliver more details next week, heap dump etc.
Its more tracing down what it can and can not be, ofcourse i am not sure, only things i no
so far:
- there are no know keycloak related issues so far and it looks like relating to
infinispan cache replication between the nodes. The issue increases/ show up only when
having many cached sessions (>300.000) but it is NOT related to activity since it keeps
increasing with same amount even at night when almost no traffic on the site.
- Its is only the TQ bundler taken up more and more cpu not other tasks (unless heap grows
towards max. but since we doubled heap GC doesnt grow above 1% cpu when tq bundler is
already into the 30% and quickly after higher. Network data transfer (also when almost no
trafic on site is growing as fast as the cpu to levels we can not explain since).
- zero errors occure anywhere that can explain issues.
So will try to hand over more details, thanks in advance
was (Author: robvanderboom):
True many components involved.
I will deliver more details next week, heap dump etc.
Its more tracing down what it can and can not be, ofcourse i am not sure, only things i no
so far:
- there are no know keycloak related issues so far and it looks like relating to
infinispan cache replication between the nodes. The issue increases/ show up only when
having many cached sessions (>300.000) but it is NOT related to activity since it keeps
increasing with same amount even at night when almost no traffic on the site.
- Its is only the TQ bundler taken up more and more cpu not other tasks (unless heap grows
towards max. but since we doubled heap GC doesnt grow above 1% cpu when tq bundler is
already into the 30% and quickly after higher. Network data transfer (also when almost no
trafic on site is growing as fast as the cpu to levels we can not explain since).
- zero errors occure anywhere that can explain issues.
So will try to hand over more details,
increasing networkdata, cpu and heap
------------------------------------
Key: JGRP-2396
URL:
https://issues.jboss.org/browse/JGRP-2396
Project: JGroups
Issue Type: Bug
Affects Versions: 4.0.19
Reporter: Rob van der Boom
Assignee: Bela Ban
Priority: Major
hey,
we have an keycloak (sso) setup, version 7.0.1 running in kubernetes - aws.
Its build on wildfly 17, infinispan 9.4 and jgroups 4.0.19.
We have 3 pods running in standalone-ha with cache setup on distribution (all 3 nodes -
so equivalent to replication)
ISSUE:
We see a slowly growing of networkstatistics, heap and cpu, while the number of sessions
in keycloak (cached) remain almost stable.
The cpu growth is caused by the TQbundler process, which explaines the networkdata
growth. It looks like this is causing also a memory leakage..
every 5 days we have to restart the pods and then every resets to a very low level
including the heap. this while all sessions are still valid and cached.
The only issue i could find maybe related to this is:
https://issues.jboss.org/browse/JGRP-2382?jql=project%20%3D%20JGRP%20AND%...
Could this be the same issue and does it also cause increasing network and cpu (since
that is why we have to restart, the heap has much space left !).
And if so how does this issue continue since for us its a major issue.
We als had this issue already in keycloak 5 (wildfly 15), thats why we upgraded to the
latest available version.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)