[jboss-jira] [JBoss JIRA] (WFLY-12718) Clustering: replicated-cache sampling errors
Radoslav Husar (Jira)
issues at jboss.org
Tue Oct 29 13:25:00 EDT 2019
[ https://issues.jboss.org/browse/WFLY-12718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805746#comment-13805746 ]
Radoslav Husar edited comment on WFLY-12718 at 10/29/19 1:24 PM:
-----------------------------------------------------------------
[~tommaso-borgato] There are about 100.000 sessions created on the cluster. It looks as though the cookies are not being sent and every request creates a new session eventually running out of memory.
{noformat}
[rhusar at ribera tmp]$ curl https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-Clustering/view/EAP7-Clustering-HTTP-Session/job/eap-7.x-clustering-http-session-shutdown-repl/26/artifact/report/wildfly/wlf_20192528-182527-wildfly-service-1-server.log | grep "New session created" | wc -l
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 15.0M 100 15.0M 0 0 4866k 0 0:00:03 0:00:03 --:--:-- 4864k
25171
[rhusar at ribera tmp]$ curl https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-Clustering/view/EAP7-Clustering-HTTP-Session/job/eap-7.x-clustering-http-session-shutdown-repl/26/artifact/report/wildfly/wlf_20192528-182527-wildfly-service-2-server.log | grep "New session created" | wc -l
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11.3M 100 11.3M 0 0 2942k 0 0:00:03 0:00:03 --:--:-- 2941k
24741
[rhusar at ribera tmp]$ curl https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-Clustering/view/EAP7-Clustering-HTTP-Session/job/eap-7.x-clustering-http-session-shutdown-repl/26/artifact/report/wildfly/wlf_20192528-182527-wildfly-service-3-server.log | grep "New session created" | wc -l
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 28.6M 100 28.6M 0 0 4714k 0 0:00:06 0:00:06 --:--:-- 6606k
21061
[rhusar at ribera tmp]$ curl https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-Clustering/view/EAP7-Clustering-HTTP-Session/job/eap-7.x-clustering-http-session-shutdown-repl/26/artifact/report/wildfly/wlf_20192528-182527-wildfly-service-4-server.log | grep "New session created" | wc -l
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4596k 100 4596k 0 0 2000k 0 0:00:02 0:00:02 --:--:-- 1999k
23242
{noformat}
was (Author: rhusar):
[~tommaso-borgato] There are about 100.000 sessions created on the server. It looks as though the cookies are not being sent and every request creates a new session eventually running out of memory.
{noformat}
[rhusar at ribera tmp]$ curl https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-Clustering/view/EAP7-Clustering-HTTP-Session/job/eap-7.x-clustering-http-session-shutdown-repl/26/artifact/report/wildfly/wlf_20192528-182527-wildfly-service-1-server.log | grep "New session created" | wc -l
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 15.0M 100 15.0M 0 0 4866k 0 0:00:03 0:00:03 --:--:-- 4864k
25171
[rhusar at ribera tmp]$ curl https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-Clustering/view/EAP7-Clustering-HTTP-Session/job/eap-7.x-clustering-http-session-shutdown-repl/26/artifact/report/wildfly/wlf_20192528-182527-wildfly-service-2-server.log | grep "New session created" | wc -l
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11.3M 100 11.3M 0 0 2942k 0 0:00:03 0:00:03 --:--:-- 2941k
24741
[rhusar at ribera tmp]$ curl https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-Clustering/view/EAP7-Clustering-HTTP-Session/job/eap-7.x-clustering-http-session-shutdown-repl/26/artifact/report/wildfly/wlf_20192528-182527-wildfly-service-3-server.log | grep "New session created" | wc -l
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 28.6M 100 28.6M 0 0 4714k 0 0:00:06 0:00:06 --:--:-- 6606k
21061
[rhusar at ribera tmp]$ curl https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-Clustering/view/EAP7-Clustering-HTTP-Session/job/eap-7.x-clustering-http-session-shutdown-repl/26/artifact/report/wildfly/wlf_20192528-182527-wildfly-service-4-server.log | grep "New session created" | wc -l
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4596k 100 4596k 0 0 2000k 0 0:00:02 0:00:02 --:--:-- 1999k
23242
{noformat}
> Clustering: replicated-cache sampling errors
> --------------------------------------------
>
> Key: WFLY-12718
> URL: https://issues.jboss.org/browse/WFLY-12718
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 18.0.0.Final
> Reporter: Tommasso Borgato
> Assignee: Paul Ferraro
> Priority: Blocker
>
> The issue is about replicated-cache in fail-over tests.
> WildFly is started in clustered mode using a replicated cache for replicating HTTP session data across cluster nodes; all 4 nodes in the cluster are initialized with the following cli script:
> {noformat}
> embed-server --server-config=standalone-ha.xml
> /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp)
> /subsystem=infinispan/cache-container=web/replicated-cache=testRepl:add()
> /subsystem=infinispan/cache-container=web/replicated-cache=testRepl/component=locking:write-attribute(name=isolation, value=REPEATABLE_READ)
> /subsystem=infinispan/cache-container=web/replicated-cache=testRepl/component=transaction:write-attribute(name=mode, value=BATCH)
> /subsystem=infinispan/cache-container=web/replicated-cache=testRepl/store=file:add()
> /subsystem=infinispan/cache-container=web:write-attribute(name=default-cache, value=testRepl)
> {noformat}
> The test is run with wildfly-18.0.0.Final.zip;
> The same tests run with version wildfly-17.0.1.Final.zip do not have any problem;
> hence this looks like a regression;
> As usual, we test that the serial value stored in the replicated cache is incremented at every call: when this is not true, we say we have a sampling error;
> Here are the runs that exhibit this issue:
> - **22.82% Fail Rate with WildFly-18 ** [eap-7.x-clustering-http-session-shutdown-repl#23|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-Clustering/view/EAP7-Clustering-HTTP-Session/job/eap-7.x-clustering-http-session-shutdown-repl/23/artifact/report/jm/html/index.html]
> - **0% Fail Rate with WildFly-17 ** [eap-7.x-clustering-http-session-shutdown-repl#24|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-Clustering/view/EAP7-Clustering-HTTP-Session/job/eap-7.x-clustering-http-session-shutdown-repl/24/artifact/report/jm/html/index.html]
> We also repeated the tests to make sure it can be reproduced:
> - **22.75% Fail rate with WildFly-18 ** [eap-7.x-clustering-http-session-shutdown-repl#26|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-Clustering/view/EAP7-Clustering-HTTP-Session/job/eap-7.x-clustering-http-session-shutdown-repl/26/artifact/report/jm/html/index.html]
> - **0% Fail Rate with WildFly-17 ** [eap-7.x-clustering-http-session-shutdown-repl#25|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/EAP7/view/EAP7-Clustering/view/EAP7-Clustering-HTTP-Session/job/eap-7.x-clustering-http-session-shutdown-repl/25/artifact/report/jm/html/index.html]
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
More information about the jboss-jira
mailing list