[jboss-jira] [JBoss JIRA] (WFLY-6583) Session leak on SmartOS hosts

[Michael Noack (JIRA)]

    [ https://issues.jboss.org/browse/WFLY-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200578#comment-13200578 ] 

Michael Noack commented on WFLY-6583:
-------------------------------------

[~ctomc] I've tried several jdks in the past for unrelated reason. In production we currently use:
[root at 0e231418-37b9-657f-ff45-a3b252c7f2da ~]# java -version
openjdk version "1.7.0-internal"
OpenJDK Runtime Environment (build 1.7.0-internal-pkgsrc_2015_04_23_11_23-b00)
OpenJDK 64-Bit Server VM (build 24.76-b04, mixed mode)
For tests with wildfly-10.0.0.Final I used (jdk-8u77-linux-x64.rpm from Oracle):
[root at d9be4e60-9641-4be9-a552-e1d0d7315167 ~]# java -version
java version "1.8.0_77"
Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)

[~pferraro] Thanks's for your reply. I'm well aware of this lack of a guarantee. Since I only used two servers in tests it was rather easy to take logs from both servers and look for matches in both logs at the same time. Also a session getting destroyed on another node wouldn't lead to an increasing amount of sessions on both nodes.

This is confirmed by using a KVM container, where the amount of sessions on both nodes remains constant, as long as polling the web service is time constant. As soon as polling stops, the amount of sessions goes down again, until it eventually reaches 0 on both nodes.

On a native container the amount of session will go down as well for as long as the timeout is, than stay at that level. I ran a 24h tests again from yesterday till today using a ten minutes timeout for sessions. After stopping the polling and waiting for 10 minutes, both nodes would have roughly 1300 sessions left. This amount hasn't changed since. I just checked the server again. No session has been created for the last 3-4 hours, and yet this is the result when queriing the nodes:
{
    "outcome" => "success",
    "result" => 1348
}
{
    "outcome" => "success",
    "result" => 1448
}
Those are KVM images that don't display the problem:
https://docs.joyent.com/public-cloud/instances/virtual-machines/images/linux/centos#centos-7-20160415
Those are container native images using a BrandZ kernel with the issue:
https://docs.joyent.com/public-cloud/instances/infrastructure/images/centos#centos-7-20160405
I'm using the same rpm to install the jdk, and the wildfly nodes are in fact just copies of each other. The only configuration difference is for the interfaces ip adresses in host.xml

> Session leak on SmartOS hosts
> -----------------------------
>
>                 Key: WFLY-6583
>                 URL: https://issues.jboss.org/browse/WFLY-6583
>             Project: WildFly
>          Issue Type: Bug
>          Components: Clustering
>    Affects Versions: 8.2.0.Final, 9.0.0.Final, 10.0.0.Final
>         Environment: CentOS 7 or SmartOS instance using Joyents Infrastructure/Bare metal container.
> [root at 979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# java -version
> java version "1.8.0_77"
> Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
> [root at 979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# uname -a
> Linux 979638eb-b45c-45b3-9fdb-d7f48276e4ef 3.10.0 BrandZ virtual linux x86_64 x86_64 x86_64 GNU/Linux
> [root at 979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# cat /etc/issue
> \S
> Kernel \r on an \m
>            Reporter: Michael Noack
>            Assignee: Paul Ferraro
>            Priority: Minor
>
> When running Wildfly 8.2.0-Final, 9.0.0-Final or 10.0.0-Final in domain mode using the full-ha profile some sessions never get closed when running on SmartOS or a BrandZ kernel on SmartOS. The amount of unclosed sessions rises slowly. With 1 session per second and server created, roughly 30-50 sessions are left unclosed on each server. I've been keeping track of this issue for almost a year now and handled it by restarting the entire cluster at first. It took me a while to connect the dots here.
> When registering a HttpSessionListener and logging any sessionCreated(HttpSessionEvent se) and sessionDestroyed(HttpSessionEvent se) one can cleary see some sessions never generate the sessionDestroyed event.
> The problem disappears when running the very same setup on a KVM instance of CentOS 6 or 7 (regardless whether the KVM host is SmartOS or Linux).



--
This message was sent by Atlassian JIRA
(v6.4.11#64026)